Scene rendering method, device, electronic device, storage medium and program product

Abstract

The present application discloses a scene rendering method, device, electronic device, storage medium and program product, which belongs to the field of computer vision technology. The method includes: decomposing a scene to be rendered into a first scene area and K second scene areas, the first scene area is used to represent the global scene of the scene to be rendered, and the second scene area is used to represent the local scene in the scene to be rendered; constructing a global sampling search grid based on the first scene area, and constructing a local sampling search grid based on the second scene area; obtaining S first sampling points from the first scene area through the global sampling search grid, and obtaining T second sampling points from the target scene area through the target sampling search grid, wherein the target sampling search grid is one of the K local sampling search grids; training a neural network based on the S first sampling points and the T second sampling points; rendering the scene to be rendered based on the trained neural network.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and specifically relates to a scene rendering method, device, electronic device, storage medium and program product. Background Art

[0002] Neural Radiance Field (NeRF), as a novel view synthesis method for implicit scene representation, has been widely used in robotics, urban mapping, autonomous navigation, virtual reality / augmented reality and other fields. Based on a set of image sequences and camera parameters in the physical world scene, NeRF can learn the density and color of 3D points in space by using volume rendering graphics methods to render new view images. However, when NeRF is applied to scenes with complex lighting conditions, the same object will present different colors under different view angles, making it difficult for NeRF to accurately fit the scene, and the scene rendered image is prone to obvious artifacts. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a scene rendering method, device, electronic device, storage medium and program product, which can solve the problem that when NeRF is applied to scenes with complex lighting conditions, NeRF is difficult to accurately fit the scene and scene images are prone to artifacts.

[0004] In a first aspect, an embodiment of the present application provides a scene rendering method, comprising:

[0005] According to scene data of a scene to be rendered, decompose the scene to be rendered into a first scene area and K second scene areas, wherein the first scene area is used to represent a global scene of the scene to be rendered, the second scene area is used to represent a local scene in the scene to be rendered, and K is a positive integer;

[0006] Constructing a global sampling search grid based on the first scene area, constructing a local sampling search grid based on the second scene area, and the K second scene areas correspond one to one to the K local sampling search grids;

[0007] Acquire S first sampling points from the first scene area through the global sampling search grid based on a preset sampling method, and acquire T second sampling points from the target scene area through a target sampling search grid based on the preset sampling method, wherein the target sampling search grid is one of the K local sampling search grids, and S and T are both positive integers;

[0008] Training a neural network based on the S first sampling points and the T second sampling points;

[0009] The scene to be rendered is rendered based on the trained neural network.

[0010] Optionally, decomposing the scene to be rendered into a first scene area and K second scene areas according to scene data of the scene to be rendered includes:

[0011] Get the scene data of the scene to be rendered;

[0012] Determine the first scene area and the point cloud data of the scene to be rendered according to the scene data;

[0013] The point cloud data is clustered to determine the K second scene areas.

[0014] Optionally, the acquiring S first sampling points from the first scene area through the global sampling search grid based on a preset sampling method, and acquiring T second sampling points from the target scene area through a target sampling search grid based on the preset sampling method, comprises:

[0015] Determine whether there is a target intersection grid intersecting with the rendering ray in the K local sampling search grids, wherein the rendering ray is a ray emitted from the target perspective;

[0016] In the case where the target intersection grid exists in the K local sampling search grids, determining whether the target intersection grid contains a sampling point whose voxel value is greater than a first threshold;

[0017] In a case where the target intersection grid includes sampling points whose voxel values ​​are greater than the first threshold, the S first sampling points are acquired from the first scene area, and the T second sampling points are acquired from the target scene area.

[0018] Optionally, the target intersection grid is determined based on the following method:

[0019] Determine N pre-selected grids, where the pre-selected grids are grids intersecting with the rendering ray among the K local sampling search grids, where N is a positive integer and N is less than or equal to K;

[0020] The pre-selected grid with the shortest distance from a preset point among the N pre-selected grids is determined as the target intersection grid, and the preset point is the initial emission point of the rendering ray.

[0021] Optionally, the S first sampling points include a first preselected sampling point and a second preselected sampling point, and the method further includes:

[0022] When the target intersection grid does not exist in the K local sampling search grids, obtaining at least one first intersection point between the global sampling search grid and the rendering ray;

[0023] Determine a first intersection point whose voxel value is greater than a second threshold value among the at least one first intersection point as the first preselected sampling point;

[0024] When the voxel value of the first target intersection point is greater than a third threshold, sampling points around the first target intersection point are determined as the second pre-selected sampling points, wherein the first target intersection point is one of the at least one first intersection point that is closest to a preset point, the preset point is the initial emission point of the rendering ray, and the third threshold is greater than the second threshold.

[0025] Optionally, the T second sampling points include a third preselected sampling point and a fourth preselected sampling point;

[0026] When the target intersection grid includes a sampling point whose voxel value is greater than the first threshold, acquiring the S first sampling points from the first scene area, and acquiring the T second sampling points from the target scene area, comprises:

[0027] When the target intersection grid includes sampling points whose voxel values ​​are greater than the first threshold, acquiring the S first sampling points from the first scene area, and acquiring at least one second intersection point between the target scene area and the rendering ray;

[0028] Determine a second intersection point whose voxel value is greater than a fourth threshold and less than the first threshold among the at least one second intersection point as the third preselected sampling point;

[0029] When the voxel value of the second target intersection point is greater than the first threshold, sampling points around the second target intersection point are determined as the fourth preselected sampling points, wherein the second target intersection point is any one of the at least one second intersection point.

[0030] Optionally, the neural network includes a first network and a second network, the first network is composed of a first diffuse reflection network and a first high-light network, and the second network is composed of K second diffuse reflection networks and K second high-light networks;

[0031] Among them, the number of fully connected network levels of the first diffuse reflection network is greater than the number of connected network levels of the second diffuse reflection network, and the number of connected network levels of the first high-optical network is greater than the number of connected network levels of the second high-optical network.

[0032] Optionally, rendering the scene to be rendered based on the trained neural network includes the following:

[0033] When the target intersection grid does not exist in the K local sampling search grids, rendering the first scene area based on the first network after training;

[0034] When the target intersection grid includes sampling points whose voxel values ​​are greater than the first threshold, rendering the first scene area based on the trained first network, and rendering the target scene area based on the trained second network;

[0035] The scene to be rendered is jointly rendered based on the first network and the second network.

[0036] Optionally, the method further comprises:

[0037] When the target intersection grid contains sampling points whose voxel values ​​are greater than the first threshold, normal vector estimation is performed on the point cloud data in the target scene area to obtain the normal vector of the point cloud data;

[0038] Determining a normal constraint condition based on the normal vector, and determining a lighting constraint condition based on color features of pixels in the scene to be rendered;

[0039] The first network is trained according to the illumination constraint, and the second network is trained according to the illumination constraint and the normal constraint.

[0040] In a second aspect, an embodiment of the present application provides a scene rendering device, including:

[0041] A scene decomposition module, used for decomposing the scene to be rendered into a first scene area and K second scene areas according to the scene data of the scene to be rendered, wherein the first scene area is used to represent the global scene of the scene to be rendered, the second scene area is used to represent the local scene in the scene to be rendered, and K is a positive integer;

[0042] A grid construction module, used to construct a global sampling search grid based on the first scene area, and to construct a local sampling search grid based on the second scene area, wherein the K second scene areas correspond to the K local sampling search grids one by one;

[0043] A sampling acquisition module, configured to obtain S first sampling points from the first scene area through the global sampling search grid based on a preset sampling method, and to obtain T second sampling points from the target scene area through a target sampling search grid based on the preset sampling method, wherein the target sampling search grid is one of the K local sampling search grids, and S and T are both positive integers;

[0044] A network training module, used for training a neural network based on the S first sampling points and the T second sampling points;

[0045] A scene rendering module is used to render the scene to be rendered based on the trained neural network.

[0046] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the scene rendering method described in the first aspect.

[0047] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the scene rendering method described in the first aspect are implemented.

[0048] In a fifth aspect, an embodiment of the present application provides a computer program product, characterized in that it includes computer instructions, which, when executed by a processor, implement the steps of the scene rendering method described in the first aspect.

[0049] In an embodiment of the present application, a first scene area for representing a global scene and K second scene areas for representing a local scene are obtained by performing a two-level decomposition of the scene to be rendered. Then, each scene area is searched and sampled to obtain a sampling result, namely, S first sampling points in the first scene area, and T second sampling points in at least one target scene area. Then, the neural network is trained according to the sampling results. In this way, the trained neural network can complete the rendering of the scene to be rendered, accurately fit the scene to be rendered, improve the rendering effect of the local scene area, eliminate artifacts appearing in the scene, and can be widely applied to various scenes to be rendered with complex lighting. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is one of the flow diagrams of a scene rendering method provided in an embodiment of the present application;

[0051] Figure 2 This is a second flow chart of a scene rendering method provided in an embodiment of the present application;

[0052] Figure 3 It is a structural schematic diagram of a scene rendering device provided in an embodiment of the present application;

[0053] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0055] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0056] The scene rendering method, apparatus and related equipment provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0057] Please refer to Figure 1 , Figure 1 : is a flow chart of a scene rendering method provided by an embodiment of the present application. Figure 1 As shown, the scene rendering method includes the following steps:

[0058] Step 101: Decompose the scene to be rendered into a first scene area and K second scene areas according to scene data of the scene to be rendered, wherein the first scene area is used to represent the global scene of the scene to be rendered, the second scene area is used to represent the local scene in the scene to be rendered, and K is a positive integer.

[0059] It is understandable that in indoor scenes with complex lighting conditions, due to different lighting conditions at different viewing angles, the colors of the same object at different viewing angles are inconsistent, and the original neural radiation field is difficult to fit the scene, resulting in obvious artifacts. However, the scene to be rendered in the embodiments of the present application can be the aforementioned indoor scene with complex lighting conditions, or various outdoor scenes, and the present application does not specifically limit the actual scene to which the scene to be rendered belongs.

[0060] In a specific implementation, a two-level decomposition can be performed for the scene to be rendered that needs to be rendered, that is, the scene to be rendered is decomposed into two levels: a global scene and a local scene, that is, a first scene area and K second scene areas. For example, the scene to be rendered is a flower bed with multiple potted plants in it, and each potted plant is planted with a kind of flower. Then the entire flower bed can be regarded as a global scene, that is, a first scene area, and a local scene can be determined with a potted plant as a unit, that is, a potted plant and the flowers in the potted plant are a second scene area. Through the above steps, the entire scene to be rendered is divided into two levels, which is convenient for subsequent key rendering of the global scene and the local scene, improving the rendering effect and reducing the appearance of artifacts.

[0061] Step 102: construct a global sampling search grid based on the first scene area, and construct a local sampling search grid based on the second scene area, wherein the K second scene areas correspond one to one with the K local sampling search grids.

[0062] It should be noted that the first sampling point can be obtained in the grid structure corresponding to the first scene area, and the second sampling point can also be obtained in the grid structure corresponding to the target scene area. A global sampling search grid is established in the first scene area, and the resolution of the global sampling search grid can be set to 1024*1024*1024. The voxel G of the global sampling search grid initialized by the depth map can also be set A , you can set the voxel value of the grid with depth in the global sampling search grid to The other meshes have voxel values ​​of 0.

[0063] Similarly, a corresponding grid can be created for each second scene area, and K local sampling search grids can be constructed, whose voxel values ​​are {G B , G C , …}, where the grid resolution can be set adaptively according to the different ranges of the second scene area, for example, it can be set to a cube. The resolution can be set to 512*512*512 or 64*64*64, and this application does not impose any specific restrictions on this. Since the local sampling search grid is part of the global sampling search grid, the resolution of the global sampling search grid needs to be greater than the resolution of any local sampling search grid. In addition, the voxel value of the grid with depth in the local sampling search grid can also be set to Other grid voxel values ​​can be 0. In this way, sampling through the global sampling lookup grid and the local sampling lookup grid can enhance the generalization of the neural radiation field and improve the rendering effect compared to deep sampling of the scene.

[0064] Step 103: Acquire S first sampling points from the first scene area through the global sampling search grid based on a preset sampling method, and acquire T second sampling points from the target scene area through the target sampling search grid based on the preset sampling method, wherein the target sampling search grid is one of the K local sampling search grids, and S and T are both positive integers.

[0065] In the above steps, obtaining S first sampling points in the first scene area can be equivalent to obtaining S first sampling points in the global sampling search grid. Similarly, obtaining T second sampling points in at least one target scene area can be equivalent to obtaining T second sampling points in a local sampling search grid corresponding to the target scene area.

[0066] Specifically, sampling is performed in the global sampling search grid. First, the part of the rendering ray emitted by each perspective intersecting with the global sampling search grid can be determined, and a preset number of first sampling points can be collected in the intersecting part.

[0067] Furthermore, since the global sampling search grid is a whole and the local sampling search grid is a part, when the rendering ray emitted from a certain view or multiple viewpoints is emitted to the global sampling search grid, the rendering ray also passes through one or more local sampling search grids. However, there is also a situation where one or more local sampling search grids are blocked and no rendering ray passes through. In order to improve the rendering effect, sampling points can be collected for the local sampling search grids through which the rendering ray passes. At this time, the second scene area corresponding to the local sampling search grid through which the rendering ray passes can be the target scene area. A preset number of second sampling points can be collected at the intersection of the local sampling search grid corresponding to the target scene area and the rendering ray, and the grid voxel values ​​of the second sampling points can be collected. For example, a rendering ray l passes through the target scene area, that is, passes through the local sampling search grid corresponding to the target scene area, and the line segment where the rendering ray l intersects with the local sampling search grid is taken, and M points are determined on the line segment as the second sampling points to collect grid voxel values. The voxel values ​​of the M points here can all be greater than the fourth threshold (the fourth threshold is a pre-set minimum threshold).

[0068] It can be seen that by decomposing the scene to be rendered into two levels of scenes, global and local, it is convenient to accurately sample the scene to be rendered and determine the key area to be rendered (i.e., the target scene area), so as to perform rendering processing on different areas and achieve accurate fitting of the scene to be rendered.

[0069] Step 104: training a neural network based on the S first sampling points and the T second sampling points.

[0070] It is worth mentioning that the neural network in the embodiment of the present application can be a neural radiation field (or an illumination decomposition network), which can specifically include a first network and a second network. The first network and the second network can both be composed of a fully connected network, and the fully connected network is composed of a diffuse reflection network MLP. d For and Gaoguang Network MLP s Among them, the diffuse reflection network learns the diffuse reflection parts in the first scene area and the target scene area respectively through S first sampling points and T second sampling points, and the highlight network learns the highlight parts in the first scene area and the target scene area respectively through S first sampling points and T second sampling points, which can enhance the generalization of the first network and the second network and improve the rendering ability of the scene.

[0071] Specifically, since the first network F1 and the second network F2 can be pre-constructed neural radiation fields (or illumination decomposition networks), they can be respectively expressed as:

[0072] F1:(x1,d1)→(c1,ρ1);

[0073] F2:(x2,d2)→(c2,ρ2);

[0074] Wherein, x1 and x2 represent the position information of a sampling point in the first scene area and the target scene area respectively, wherein the position information x=(x, y, z) can be three-dimensional coordinate data; d1 and d2 represent the direction information of a sampling point in the first scene area and the target scene area respectively, wherein the direction information It can be a direction vector; c1 and c2 represent the color of a certain sampling point output by the first network and the second network respectively, where the color can include c=(r, g, b); ρ1 and ρ2 represent the density of a certain sampling point output by the first network and the second network respectively.

[0075] In addition, through the color and density information output by the first network and the second network respectively, the opacity (Opacity) of one or more sampling points in the first scene area and the opacity of one or more sampling points in the target scene area can be determined, and the rendering fitting effect can be adjusted by the opacity to achieve accurate fitting and improve the artifact elimination effect.

[0076] Step 105: Render the scene to be rendered based on the trained neural network.

[0077] In a specific embodiment of the present application, the first opacity corresponding to the S first sampling points can be determined by the color and density output by the first network, and the opacity corresponding to the T second sampling points can be determined by the color and density output by the second network. Since the opacity can show the degree of light transmission on the current object surface, when the opacity (Opacity) is 0, the surface allows all light to pass through, and the opacity value between 0 and 1 will produce partially transparent pixels. Therefore, the first opacity corresponding to the S first sampling points can determine the degree of light transmission from a certain target line of sight (such as rendering rays) to the surface of the object in the first scene area.

[0078] Specifically, rendering of the scene to be rendered can be achieved through the opacity obtained by the neural network. In order to improve the rendering effect of the object surface, the grid voxel values ​​of the global sampling search grid and the K local sampling search grids can be updated by the opacity of multiple sampling points. In this way, by adjusting the grid voxel values ​​in the global sampling search grid and the K local sampling search grids, the scene rendering effect is enhanced.

[0079] Optionally, decomposing the scene to be rendered into a first scene area and K second scene areas according to scene data of the scene to be rendered includes:

[0080] Get the scene data of the scene to be rendered;

[0081] Determine the first scene area and the point cloud data of the scene to be rendered according to the scene data;

[0082] The point cloud data is clustered to determine the K second scene areas.

[0083] It should be noted that to decompose the scene area of ​​two levels, it is first necessary to obtain the scene data of the scene to be rendered, and the scene data may specifically include image data, depth image data, internal parameter data, and external parameter data, etc. Among them, the image data can be obtained by shooting around the scene to be rendered through a camera device, a photographic device, etc., or the image data of the existing scene can be directly input into the execution body of the embodiment of the present application. After the image data is obtained, the depth map data can be obtained through a depth device or a depth map generation network, or the pre-stored depth map data can be directly used without the image data acquisition step. In addition, the acquired image data can also be processed by an open source method (Colmap) for sparse and dense reconstruction in the motion recovery structure (Structure from Motion, SFM) to complete the acquisition of the position and posture of the camera in the scene and the position of the three-dimensional point in the scene, that is, to obtain the internal parameter data and external parameter data of the camera or other data acquisition equipment.

[0084] In one embodiment of the present application, after the scene data is acquired, the point cloud data of the scene to be rendered can be calculated by projection based on the depth map data, internal parameter data, and external parameter data in the scene data. At the same time, the global scene in the acquired image data can be used as the first scene area, which can be represented by A. Subsequently, the point cloud data is clustered and the entire scene to be rendered is divided into K second scene areas, which can be represented by {B, C, ...}. Specifically, the method used for clustering the point cloud data can be the K-means algorithm, which is not specifically limited in this application.

[0085] It can be seen that, through the embodiment of the present application, the scene to be rendered is divided into a first scene area and K second scene areas, which facilitates the rendering process of key areas in the scene and improves the scene rendering effect.

[0086] Optionally, the acquiring S first sampling points from the first scene area through the global sampling search grid based on a preset sampling method, and acquiring T second sampling points from the target scene area through a target sampling search grid based on the preset sampling method, comprises:

[0087] Determine whether there is a target intersection grid intersecting with the rendering ray in the K local sampling search grids, wherein the rendering ray is a ray emitted from the target perspective;

[0088] In the case where the target intersection grid exists in the K local sampling search grids, determining whether the target intersection grid contains a sampling point whose voxel value is greater than a first threshold;

[0089] In a case where the target intersection grid includes sampling points whose voxel values ​​are greater than the first threshold, the S first sampling points are acquired from the first scene area, and the T second sampling points are acquired from the target scene area.

[0090] It is understandable that the target intersection grid can be a grid where the rendering ray is emitted from the initial point (or preset point) set by the target perspective, and is directed to K local sampling search grids and intersects with one of the local sampling search grids. Since the rendering scene usually has a more complex lighting environment, there may be multiple rendering rays emitted from different perspectives in practice, and each rendering ray may correspond to the rendering of a second scene area, or may not intersect with any second scene area. For example, a rendering ray is shot into the entire scene of the flower bed and intersects with a pot of flowers in the flower bed. Since the occlusion rendering ray of this flower cannot penetrate, the rendering ray usually only intersects with the second scene area corresponding to a flower pot. Furthermore, the voxel value corresponding to the target intersection grid can be obtained and compared with the first threshold. If there is a sampling point greater than the first threshold, it can be considered to be a focus rendering of the target scene area.

[0091] Therefore, in the embodiment of the present application, it is possible to determine whether there is a local scene area that needs to be rendered in the K second scene areas by setting two conditions: one is whether the local sampling search grid corresponding to the second scene area intersecting with the rendering ray is a target intersection grid; the other is whether the target intersection grid contains sampling points with voxel values ​​greater than a first threshold. When the above two conditions are met at the same time, it can be determined that there is a local area that needs to be rendered in the scene to be rendered.

[0092] In some embodiments of the present application, the target intersection grid that intersects with the rendering ray in the K local sampling search grids can be determined first. It can be that a rendering ray has intersection points with multiple local sampling search grids, so there can be multiple pre-selected grids, and each pre-selected grid can be any local sampling search grid in the K local sampling search grids. Subsequently, if there is a pre-selected grid whose voxel value is greater than the first threshold value, and the pre-selected grid closest to the initial point can be determined as the target intersection grid, the target intersection grid is the grid corresponding to the local scene area that needs to be rendered. In this way, in the case of a target intersection grid that needs to be rendered, the target scene area is sampled, and T second sampling points in the target scene area are collected. At the same time, S first sampling points can also be collected in the intersection of the first scene area and the rendering ray. In this way, by sampling the first scene area and the target scene area respectively, the rendering effect of the part that needs to be rendered in the scene to be rendered can be enhanced, and accurate fitting can be achieved.

[0093] Optionally, the target intersection grid is determined based on the following method:

[0094] Determine N pre-selected grids, where the pre-selected grids are grids intersecting with the rendering ray among the K local sampling search grids, where N is a positive integer and N is less than or equal to K;

[0095] The pre-selected grid with the shortest distance from a preset point among the N pre-selected grids is determined as the target intersection grid, and the preset point is the initial emission point of the rendering ray.

[0096] It should be understood that when a rendering ray enters K local sampling search grids, it is possible that it intersects with two or more partially overlapping local sampling search grids. For example, in a scene of a flower bed and flower pots, a rendering ray may pass through the area between two flower pots or the overlapping area, and it may intersect with the second scene area where the two flower pots are located. Therefore, the local sampling search grids intersecting with the rendering ray can be determined as pre-selected grids.

[0097] In a specific embodiment of the present application, the preselected grid corresponding to the rendering ray can be determined by comparing the distance between each preselected grid and the point (or initial point) where the rendering ray is emitted. Further, the target intersection grid can be determined by comparing the distance between each preselected grid and the preset point (i.e., the initial point where the rendering ray is emitted from the target perspective). In this way, the grid corresponding to the preselected area close to the preset point can be set as the target intersection grid. The closer to the initial point where the rendering ray is emitted, the better the light transmission effect and the better the rendering effect.

[0098] Optionally, the S first sampling points include a first preselected sampling point and a second preselected sampling point, and the method further includes:

[0099] When the target intersection grid does not exist in the K local sampling search grids, obtaining at least one first intersection point between the global sampling search grid and the rendering ray;

[0100] Determine a first intersection point whose voxel value is greater than a second threshold value among the at least one first intersection point as the first preselected sampling point;

[0101] When the voxel value of the first target intersection point is greater than a third threshold, sampling points around the first target intersection point are determined as the second pre-selected sampling points, wherein the first target intersection point is one of the at least one first intersection point that is closest to a preset point, the preset point is the initial emission point of the rendering ray, and the third threshold is greater than the second threshold.

[0102] In some embodiments of the present application, when the rendering ray does not pass through any second scene area, it indicates that there is no scene area that needs to be rendered, and only the first scene area can be collected. Specifically, the first sampling point collected can be a first pre-selected sampling point or a second pre-selected sampling point. Among them, the first pre-selected sampling point can be a point whose voxel value is greater than the second threshold value among multiple first intersection points, and the second threshold value can be a pre-set minimum standard value, which is used to determine whether the point can more accurately reflect the value of the illumination level of the viewing angle where the current rendering ray is located. The second pre-selected sampling point can be determined based on the first target intersection point, and the first target intersection point is the point closest to the preset point and whose voxel value is greater than the third threshold value. A plurality of second pre-selected sampling points can also be sampled around the first target intersection point through a probability distribution function, such as a normal distribution function or a uniform distribution function. In this way, by collecting the first pre-selected sampling point and the second pre-selected sampling point, the fitting degree of the neural radiation field can be improved, the rendering effect can be improved, and the appearance of artifacts can be reduced.

[0103] Optionally, the T second sampling points include a third preselected sampling point and a fourth preselected sampling point;

[0104] When the target intersection grid includes a sampling point whose voxel value is greater than the first threshold, acquiring the S first sampling points from the first scene area, and acquiring the T second sampling points from the target scene area, comprises:

[0105] When the target intersection grid includes sampling points whose voxel values ​​are greater than the first threshold, acquiring the S first sampling points from the first scene area, and acquiring at least one second intersection point between the target scene area and the rendering ray;

[0106] Determine a second intersection point whose voxel value is greater than a fourth threshold and less than the first threshold among the at least one second intersection point as the third preselected sampling point;

[0107] When the voxel value of the second target intersection point is greater than the first threshold, sampling points around the second target intersection point are determined as the fourth preselected sampling points, wherein the second target intersection point is any one of the at least one second intersection point.

[0108] In some embodiments of the present application, when there is a second scene area (i.e., a target scene area) that needs to be rendered emphatically, sampling can be performed within the target scene area. Specifically, a plurality of second intersection points that intersect with the rendering ray in the target scene area can be first determined, thereby determining a second sampling point among the second intersection points, and the collected second sampling point can be a third preselected sampling point or a fourth preselected sampling point. Among them, the third preselected sampling point can be a point whose voxel value collected at the intersection of the rendering ray and the target scene area is greater than the fourth threshold but less than the first threshold, the fourth preselected sampling point can be obtained by collecting around the second target intersection point, and the second target intersection point can be a point whose voxel value collected at the intersection is greater than the first threshold.

[0109] It is understandable that the voxel value of the second target intersection point is greater than the first threshold, and the first threshold is greater than the fourth threshold, indicating that the second target intersection point can more accurately reflect the lighting conditions under the perspective of this rendering ray. Therefore, the sampling points around the second target intersection point can also be sampled through a probability distribution function, such as a normal distribution function or a uniform distribution function, to obtain a fourth pre-selected sampling point. In this way, when there is a target scene area that needs to be rendered, the third pre-selected sampling point and the fourth pre-selected sampling point are collected to accurately reflect the lighting effect of the perspective of the current rendering ray, which is convenient for accurate fitting and improves the rendering effect.

[0110] Optionally, the neural network includes a first network and a second network, the first network is composed of a first diffuse reflection network and a first high-light network, and the second network is composed of K second diffuse reflection networks and K second high-light networks;

[0111] Among them, the number of fully connected network levels of the first diffuse reflection network is greater than the number of connected network levels of the second diffuse reflection network, and the number of connected network levels of the first high-optical network is greater than the number of connected network levels of the second high-optical network.

[0112] In some embodiments of the present application, the first network for performing illumination decomposition on the first scene area may be a fully connected network with 10 levels. The second network for performing illumination decomposition on K second scene areas may include K second diffuse reflection networks and K second highlight networks, and a second diffuse reflection network and a second highlight network may perform illumination decomposition on a second scene area. The network for performing illumination decomposition on the second scene area may be a fully connected network with 5 levels. Among them, the first diffuse reflection network in the first network may have 6 layers of 64 dimensions, and the first highlight network may have 4 layers of 32 dimensions. A second diffuse reflection network in the second network may have 3 layers of 64 dimensions, and a second highlight network may have 2 layers of 32 dimensions. The present application does not limit the number of levels of the first network and the second network in the actual application process. By dividing the neural radiation field into two levels, namely global and local, and using a fully connected network with different numbers of levels, the generalization of the overall neural radiation field can be enhanced, and the rendering effect of the local scene can be improved through the second network, so as to achieve accurate fitting of the scene rendering.

[0113] Optionally, rendering the scene to be rendered based on the trained neural network includes the following:

[0114] When the target intersection grid does not exist in the K local sampling search grids, rendering the first scene area based on the first network after training;

[0115] When the target intersection grid includes sampling points whose voxel values ​​are greater than the first threshold, rendering the first scene area based on the trained first network, and rendering the target scene area based on the trained second network;

[0116] The scene to be rendered is jointly rendered based on the first network and the second network.

[0117] In some other specific embodiments of the present application, different situations can be distinguished when rendering the scene to be rendered. For example, when there is an area that needs to be rendered in the scene to be rendered, that is, when the target intersection grid contains sampling points whose voxel values ​​are greater than the first threshold, the first scene area can be rendered through the first network, and the target scene area can be rendered through the second network. When there is no area that needs to be rendered in the scene to be rendered, only the first scene area can be rendered. In addition, the initial image of the entire scene to be rendered can be rendered by jointly rendering through the neural network, that is, the first network and the second network, to achieve accurate fitting of the scene to be rendered, which is widely applicable to various scenes to be rendered with complex lighting.

[0118] Optionally, the method further comprises:

[0119] When the target intersection grid contains sampling points whose voxel values ​​are greater than the first threshold, normal vector estimation is performed on the point cloud data in the target scene area to obtain the normal vector of the point cloud data;

[0120] Determining a normal constraint condition based on the normal vector, and determining a lighting constraint condition based on color features of pixels in the scene to be rendered;

[0121] The first network is trained according to the illumination constraint, and the second network is trained according to the illumination constraint and the normal constraint.

[0122] In some embodiments, when part or all of a scene in a third scene area overlaps with the target scene area, that is, part of the scene in the third scene area through which the rendering ray passes also belongs to the target scene area, the normal vector of the point cloud data in the target scene area can be estimated to obtain the normal vector The normal vector estimation can be performed using the principal component analysis (PCA) method. The specific method used in the normal vector estimation is not limited in this application. Subsequently, the normal constraint of the second training network can be constructed using the normal vector. Specifically, the ray emission direction and the normal vector can be calculated and rendered. The cosine of the angle between them is the normal constraint.

[0123] In this way, by training the first network through the illumination constraint conditions in this embodiment, and then training the second network through the normal constraint conditions and the illumination constraint conditions, the first network and the second network can be made to fit the rendering scene requirements, adapt to different viewing angles and different illumination conditions, and improve the rendering effect of the local scene area.

[0124] like Figure 2As shown, the embodiment of the present application also provides a specific implementation process of a scene rendering method, including the following steps:

[0125] Step 201, obtaining scene data of a scene to be rendered;

[0126] Step 202: determining the first scene area and the point cloud data of the scene to be rendered according to the scene data;

[0127] Step 203: clustering the point cloud data to determine the K second scene areas;

[0128] Step 204: construct a global sampling search grid and K local sampling search grids according to the first scene area and the K second scene areas, respectively, and the K second scene areas correspond to the K second sampling search grids one by one;

[0129] Step 205: Determine whether there is a target intersection grid intersecting with the rendering ray in the K local sampling search grids, wherein the rendering ray is a ray emitted from the target perspective;

[0130] Step 206: When the target intersection grid exists in the K local sampling search grids, determine whether the target intersection grid contains a sampling point whose voxel value is greater than a first threshold;

[0131] Step 207: when the target intersection grid includes sampling points whose voxel values ​​are greater than the first threshold, obtain the S first sampling points from the first scene area, and obtain the T second sampling points from the target scene area;

[0132] Step 208: When the target intersection grid does not exist in the K local sampling search grids, obtain S first sampling points from the first scene area;

[0133] Step 209: training a neural network based on the S first sampling points, or the S first sampling points and the T second sampling points; the neural network is also trained based on normal constraints and illumination constraints;

[0134] Step 210: Render the scene to be rendered based on the trained neural network.

[0135] It should be noted that the specific implementation process of the above embodiment can be specifically referred to Figure 1 The relevant descriptions of the specific embodiments in the above can achieve the same beneficial effects, and to avoid repetition, they will not be repeated here.

[0136] Please refer to Figure 3 In another embodiment of the present application, a scene rendering device 300 is provided, comprising:

[0137] A scene decomposition module 301 is used to decompose the scene to be rendered into a first scene area and K second scene areas according to the scene data of the scene to be rendered, wherein the first scene area is used to represent the global scene of the scene to be rendered, the second scene area is used to represent the local scene in the scene to be rendered, and K is a positive integer;

[0138] A grid construction module 302, configured to construct a global sampling search grid based on the first scene area, and to construct a local sampling search grid based on the second scene area, wherein the K second scene areas correspond to the K local sampling search grids one by one;

[0139] A sampling acquisition module 303 is used to obtain S first sampling points from the first scene area through the global sampling search grid based on a preset sampling method, and to obtain T second sampling points from the target scene area through a target sampling search grid based on the preset sampling method, wherein the target sampling search grid is one of the K local sampling search grids, and S and T are both positive integers;

[0140] A network training module 304, configured to train a neural network based on the S first sampling points and the T second sampling points;

[0141] The scene rendering module 305 is used to render the scene to be rendered based on the trained neural network.

[0142] Optionally, the scene decomposition module 301 includes:

[0143] A first acquisition submodule is used to acquire scene data of a scene to be rendered;

[0144] A first determining submodule, used to determine the first scene area and the point cloud data of the scene to be rendered according to the scene data;

[0145] The second determination submodule is used to cluster the point cloud data to determine the K second scene areas.

[0146] Optionally, the sampling and acquisition module 303 includes:

[0147] A third determination submodule is used to determine whether there is a target intersection grid intersecting with the rendering ray in the K local sampling search grids, wherein the rendering ray is a ray emitted from a target perspective;

[0148] A fourth determination submodule, configured to determine, when the target intersection grid exists in the K local sampling search grids, whether the target intersection grid contains a sampling point whose voxel value is greater than a first threshold;

[0149] The second acquisition submodule is used to acquire the S first sampling points from the first scene area and acquire the T second sampling points from the target scene area when the target intersection grid contains sampling points whose voxel values ​​are greater than the first threshold.

[0150] Optionally, the sampling and acquisition module 303 further includes a grid determination submodule, and the grid determination submodule includes:

[0151] A first determining unit is used to determine N pre-selected grids, where the pre-selected grids are grids intersecting with the rendering ray among the K local sampling search grids, wherein N is a positive integer and N is less than or equal to K;

[0152] The second determining unit is used to determine the pre-selected grid with the shortest distance from a preset point among the N pre-selected grids as the target intersection grid, and the preset point is the initial emission point of the rendering ray.

[0153] Optionally, the S first sampling points include a first preselected sampling point and a second preselected sampling point, and the scene rendering device 300 further includes:

[0154] A third acquisition module, configured to acquire at least one first intersection point between the global sampling search grid and the rendering ray when the target intersection grid does not exist in the K local sampling search grids;

[0155] A first determining module, configured to determine a first intersection point whose voxel value is greater than a second threshold value among the at least one first intersection point as the first pre-selected sampling point;

[0156] The second determination module is used to determine the sampling points around the first target intersection point as the second pre-selected sampling points when the voxel value of the first target intersection point is greater than a third threshold, wherein the first target intersection point is the one of the at least one first intersection point that is closest to a preset point, the preset point is the initial emission point of the rendering ray, and the third threshold is greater than the second threshold.

[0157] Optionally, the T second sampling points include a third preselected sampling point and a fourth preselected sampling point;

[0158] The second acquisition submodule includes:

[0159] a first acquisition unit, configured to acquire the S first sampling points from the first scene area and acquire at least one second intersection point between the target scene area and the rendering ray when the target intersection grid includes sampling points whose voxel values ​​are greater than the first threshold;

[0160] A third determining unit, configured to determine, among the at least one second intersection point, a second intersection point whose voxel value is greater than a fourth threshold and less than the first threshold as the third pre-selected sampling point;

[0161] A fourth determination unit is used to determine sampling points around the second target intersection point as the fourth pre-selected sampling points when the voxel value of the second target intersection point is greater than the first threshold, wherein the second target intersection point is any one of the at least one second intersection point.

[0162] Optionally, the neural network includes a first network and a second network, the first network is composed of a first diffuse reflection network and a first high-light network, and the second network is composed of K second diffuse reflection networks and K second high-light networks;

[0163] Among them, the number of fully connected network levels of the first diffuse reflection network is greater than the number of connected network levels of the second diffuse reflection network, and the number of connected network levels of the first high-optical network is greater than the number of connected network levels of the second high-optical network.

[0164] Optionally, the scene rendering module 305 includes any one of the following:

[0165] A first rendering submodule, configured to render the first scene area based on the first network after training when the target intersection grid does not exist in the K local sampling search grids;

[0166] A second rendering submodule is configured to render the first scene area based on the trained first network and render the target scene area based on the trained second network when the target intersection grid contains sampling points whose voxel values ​​are greater than the first threshold;

[0167] The third rendering submodule is used to jointly render the scene to be rendered based on the first network and the second network.

[0168] Optionally, the scene rendering device 300 further includes:

[0169] A normal vector estimation module, configured to estimate the normal vector of the point cloud data in the target scene area to obtain the normal vector of the point cloud data when a part of the scene in the at least one third scene area overlaps with the target scene area;

[0170] A condition determination module, used to determine a normal constraint condition based on the normal vector, and to determine an illumination constraint condition based on the color characteristics of the pixel points in the scene to be rendered;

[0171] A conditional training module is used to train the first network according to the illumination constraint condition, and to train the second network according to the illumination constraint condition and the normal constraint condition.

[0172] The scene rendering device 300 provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment will not be described again here.

[0173] like Figure 4 As shown, the embodiment of the present application also provides an electronic device, including a processor 400, a memory 420 and a transceiver 410, a program or instruction stored in the memory 420 and executable on the processor 400, and the program or instruction is executed by the processor 400 to implement the following Figure 1 The various processes of the scene rendering method embodiment shown in the figure can achieve the same technical effect. To avoid repetition, they will not be described here.

[0174] The embodiment of the present application also provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the above Figure 1 The various processes of the scene rendering method embodiment can achieve the same technical effect, and will not be described again here to avoid repetition.

[0175] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0176] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0177] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including a number of instructions for a terminal (which can be a mobile phone, computer, computing network, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0178] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A scene rendering method, characterized in that: include: According to scene data of a scene to be rendered, decompose the scene to be rendered into a first scene area and K second scene areas, wherein the first scene area is used to represent a global scene of the scene to be rendered, the second scene area is used to represent a local scene in the scene to be rendered, and K is a positive integer; Constructing a global sampling search grid based on the first scene area, constructing a local sampling search grid based on the second scene area, and the K second scene areas correspond one to one to the K local sampling search grids; Acquire S first sampling points from the first scene area through the global sampling search grid based on a preset sampling method, and acquire T second sampling points from the target scene area through a target sampling search grid based on the preset sampling method, wherein the target sampling search grid is one of the K local sampling search grids, and S and T are both positive integers; Training a neural network based on the S first sampling points and the T second sampling points; Rendering the scene to be rendered based on the trained neural network; The method of acquiring S first sampling points from the first scene area through the global sampling search grid based on the preset sampling method, and acquiring T second sampling points from the target scene area through the target sampling search grid based on the preset sampling method, comprises: When there is a target intersection grid intersecting with the rendering ray in the K local sampling search grids, determining whether the target intersection grid contains a sampling point whose voxel value is greater than a first threshold; wherein the rendering ray is a ray emitted from a target perspective; In a case where the target intersection grid includes sampling points whose voxel values ​​are greater than the first threshold, the S first sampling points are acquired from the first scene area, and the T second sampling points are acquired from the target scene area.

2. The method according to claim 1, characterized in that The step of decomposing the scene to be rendered into a first scene area and K second scene areas according to the scene data of the scene to be rendered comprises: Get the scene data of the scene to be rendered; Determine the first scene area and the point cloud data of the scene to be rendered according to the scene data; The point cloud data is clustered to determine the K second scene areas.

3. The method according to claim 1, characterized in that: The target intersection grid is determined based on the following method: Determine N pre-selected grids, where the pre-selected grids are grids intersecting with the rendering ray among the K local sampling search grids, where N is a positive integer and N is less than or equal to K; The pre-selected grid with the shortest distance from a preset point among the N pre-selected grids is determined as the target intersection grid, and the preset point is the initial emission point of the rendering ray.

4. The method according to claim 1, characterized in that The S first sampling points include a first preselected sampling point and a second preselected sampling point, and the method further includes: When the target intersection grid does not exist in the K local sampling search grids, obtaining at least one first intersection point between the global sampling search grid and the rendering ray; Determine a first intersection point whose voxel value is greater than a second threshold value among the at least one first intersection point as the first preselected sampling point; When the voxel value of the first target intersection point is greater than a third threshold, sampling points around the first target intersection point are determined as the second pre-selected sampling points, wherein the first target intersection point is one of the at least one first intersection point that is closest to a preset point, the preset point is the initial emission point of the rendering ray, and the third threshold is greater than the second threshold.

5. The method according to claim 1, characterized in that The T second sampling points include a third preselected sampling point and a fourth preselected sampling point; When the target intersection grid includes a sampling point whose voxel value is greater than the first threshold, acquiring the S first sampling points from the first scene area, and acquiring the T second sampling points from the target scene area, comprises: When the target intersection grid includes sampling points whose voxel values ​​are greater than the first threshold, acquiring the S first sampling points from the first scene area, and acquiring at least one second intersection point between the target scene area and the rendering ray; Determine a second intersection point whose voxel value is greater than a fourth threshold and less than the first threshold among the at least one second intersection point as the third preselected sampling point; When the voxel value of the second target intersection point is greater than the first threshold, sampling points around the second target intersection point are determined as the fourth preselected sampling points, wherein the second target intersection point is any one of the at least one second intersection point.

6. The method according to claim 1, characterized in that The neural network includes a first network and a second network, the first network is composed of a first diffuse reflection network and a first high-light network, and the second network is composed of K second diffuse reflection networks and K second high-light networks; Among them, the number of fully connected network levels of the first diffuse reflection network is greater than the number of connected network levels of the second diffuse reflection network, and the number of connected network levels of the first high-optical network is greater than the number of connected network levels of the second high-optical network.

7. The method according to claim 6, characterized in that The rendering of the scene to be rendered based on the trained neural network includes any one of the following: When the target intersection grid does not exist in the K local sampling search grids, rendering the first scene area based on the first network after training; When the target intersection grid includes sampling points whose voxel values ​​are greater than the first threshold, rendering the first scene area based on the trained first network, and rendering the target scene area based on the trained second network; The scene to be rendered is jointly rendered based on the first network and the second network.

8. The method according to claim 7, characterized in that The method further comprises: When the target intersection grid contains sampling points whose voxel values ​​are greater than the first threshold, normal vector estimation is performed on the point cloud data in the target scene area to obtain the normal vector of the point cloud data; Determining a normal constraint condition based on the normal vector, and determining a lighting constraint condition based on color features of pixels in the scene to be rendered; The first network is trained according to the illumination constraint, and the second network is trained according to the illumination constraint and the normal constraint.

9. A scene rendering device, characterized in that: include: A scene decomposition module, used for decomposing the scene to be rendered into a first scene area and K second scene areas according to the scene data of the scene to be rendered, wherein the first scene area is used to represent the global scene of the scene to be rendered, the second scene area is used to represent the local scene in the scene to be rendered, and K is a positive integer; A grid construction module, used to construct a global sampling search grid based on the first scene area, and to construct a local sampling search grid based on the second scene area, wherein the K second scene areas correspond to the K local sampling search grids one by one; A sampling acquisition module, configured to obtain S first sampling points from the first scene area through the global sampling search grid based on a preset sampling method, and to obtain T second sampling points from the target scene area through a target sampling search grid based on the preset sampling method, wherein the target sampling search grid is one of the K local sampling search grids, and S and T are both positive integers; A network training module, used for training a neural network based on the S first sampling points and the T second sampling points; A scene rendering module, used for rendering the scene to be rendered based on the trained neural network; The sampling and acquisition module is used for: When there is a target intersection grid intersecting with the rendering ray in the K local sampling search grids, determining whether the target intersection grid contains a sampling point whose voxel value is greater than a first threshold; wherein the rendering ray is a ray emitted from a target perspective; In a case where the target intersection grid includes sampling points whose voxel values ​​are greater than the first threshold, the S first sampling points are acquired from the first scene area, and the T second sampling points are acquired from the target scene area.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the scene rendering method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the scene rendering method according to any one of claims 1 to 8 are implemented.

12. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the scene rendering method as claimed in any one of claims 1 to 9.

Images