Three-dimensional radiation field modeling method and system based on contrast voxel module and medium

Abstract

The invention discloses a three-dimensional radiation field modeling method and system based on a contrast voxel module and a medium. Relates to the technical field of three-dimensional modeling. According to the method, improvement is carried out on the basis of a traditional neural radiation field model, in the process of constructing a three-dimensional radiation field model of a target area, surrounding points and ray points of all rays are obtained, and the mutual relation between the surrounding points and the ray points is learned based on Transform so as to predict the color and density of the ray points; performing global explicit constraint on the scene; acquiring positive and negative sample pairs of each voxel block; positive and negative sample pairs are constrained by a contrast loss function based on a contrast learning mechanism so as to ensure that features in the same voxel are similar and cross-voxel features are different; according to the scheme, based on the ray sampling strategy of the voxel and the local implicit constraint of the transformer in the voxel, the radiation consistency between points in the voxel is improved, and the problem of inaccurate scene representation caused by sparse input is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional modeling, and in particular to a three-dimensional radiation field modeling method, system and medium based on a contrast voxel module. Background Art

[0002] Currently, the research on 3D reconstruction and radiance field representation plays a vital role in applications such as computer vision, augmented reality (AR) and virtual reality (VR). In recent years, the neural radiance field (NeRF)-based method has become a research hotspot in this field due to its outstanding performance in inferring 3D geometry and radiance information from 2D images. The neural radiance field method reconstructs the 3D structure and radiance information of the scene from sparse 2D images taken from different perspectives through a neural network. Its core method is to map each position coordinate in space to color and density through a multi-layer perceptron (MLP), thereby generating a continuous 3D scene representation.

[0003] Although the neural radiance field method performs well in 3D scene reconstruction, it still has some objective shortcomings in certain application scenarios, especially in the case of sparse input. The following are some of the main shortcomings of the existing technology: 1. Highly dependent on the number of input images: Neural radiation fields require a large number of two-dimensional images taken from multiple different angles to obtain high-quality three-dimensional scene reconstruction results. Sparse input images (i.e., images with fewer viewing angles) will lead to a lack of consistency in the generated three-dimensional scene and unclear details, especially at a small number of viewing angles, where the reconstruction effect is significantly reduced; 2. Consistency issues of reconstruction results: Since the neural radiation field relies on the independent prediction of each spatial point and lacks global consistency constraints on the entire 3D scene, the generated 3D scene may have inconsistent local distortion problems in the case of sparse input. This means that the geometric and radiation information between different regions may not remain consistent. Summary of the invention

[0004] The technical problem to be solved by the present invention is that the traditional neural radiation field (NeRF) model performs poorly under sparse input conditions, which is mainly reflected in the difficulty in accurately predicting the density and color of the scene, resulting in inconsistent scene representation, and the generated image has problems such as blur or artifacts; the present invention provides a three-dimensional radiation field modeling method, system and medium based on a contrast voxel module, which is improved on the basis of the traditional neural radiation field model, and effectively improves the radiation consistency between points in the voxel through a voxel-based ray sampling strategy and local implicit constraints of the transformer within the voxel, thereby avoiding the problem of inaccurate scene representation caused by sparse input; this scheme ensures the consistency of characteristics between points across voxel blocks through global explicit contrast regularization, thereby reducing the ambiguity and artifact problems of the radiation field in the scene; and utilizes a contrast learning module to enhance the generalization ability of the model so that it can generate consistent three-dimensional representations under different sparse conditions.

[0005] The present invention is achieved through the following technical solutions: This solution provides a three-dimensional radiation field modeling method based on a contrast voxel module, including: Step 1, collecting image data of the target area; Step 2: construct a scene of the target area based on the image data, divide the scene evenly into multiple voxel blocks of equal volume according to the voxel ray sampling method, and sample rays from the multiple voxel blocks; Step 3: Construct a 3D radiation field model of the target area. During the process: Local implicit constraints on the scene: obtain the surrounding points and ray points of each ray, and learn the relationship between the surrounding points and ray points based on Transformer to predict the color and density of the ray points; Global explicit constraints on the scene: obtain positive and negative sample pairs for each voxel block; constrain the positive and negative sample pairs with a contrastive loss function based on a contrastive learning mechanism to ensure that the features within the same voxel are similar, while the features across voxels are different; Step 4: Generate a three-dimensional radiation field model of the target area.

[0006] A further optimization scheme is that the scene of the target area is constructed based on the image data, the scene is evenly divided into a plurality of voxel blocks of equal volume according to the voxel ray sampling method, and rays are sampled from the plurality of voxel blocks; including the method: S21, configuring scene parameters and voxel block parameters based on image data; S22, dividing the scene evenly into a plurality of cubes of equal size according to the scene parameter and the voxel block parameter, each cube being a voxel block, and each voxel block having a plurality of intersecting rays intersecting with it; S23, randomly select V voxels from all voxel blocks for ray sampling: for voxel block i, randomly select R rays, each of which intersects with multiple points inside voxel block i.

[0007] A further optimization scheme is that the scene parameters include scene range and scene size; and the voxel block parameters include voxel block quantity and voxel block size.

[0008] A further optimization scheme is to perform local implicit constraints on the scene, including the following method: For the current ray, randomly sample S points around the line segment where the current ray intersects with the inside of the voxel block as surrounding points, and uniformly sample P points on the line segment where the current ray intersects with the inside of the voxel block as ray points; The initial features are extracted from the surrounding points based on the multi-layer perceptron MLP, and then the encoder Encoder based on the transformer network extracts the encoded features from the initial features; The decoder based on the Transformer network converts the encoded features into the radiation attribute color and density of the ray point.

[0009] A further optimization scheme is that S points are randomly sampled around the line segment where the current ray intersects with the inside of the voxel block as surrounding points, and P points are uniformly sampled on the line segment where the current ray intersects with the inside of the voxel block as ray points, including the method: Randomly sample S points according to the following formula to get the surrounding points :

[0010] in, represents the kth point in the surrounding point set, Represents the operation of sampling inside the sphere Q; represents the entry intersection point of the ray with the voxel block; represents the intersection point of the ray with the exit of the voxel block; represents the midpoint between the entry intersection point and the exit intersection point, which is used as the center of the sphere Q. radius represents the radius of the sphere Q, which is used to indicate the size of the sampling area. According to the following formula, P points are uniformly sampled to obtain the ray points :

[0011] in, represents the operation of sampling on a ray segment, Represents the tth point in the ray point set.

[0012] A further optimization scheme is to perform global explicit constraints on the scene, including the following methods: Construct positive and negative sample pairs for each voxel block; Randomly sample multiple voxel blocks, calculate the feature distances of positive and negative pairs respectively, and constrain the feature distances between positive pairs to be as small as possible and the feature distances between negative pairs to be as large as possible based on the contrast loss function; The contrastive loss function is back-propagated to the neural radiance field model to update the parameters of the neural radiance field model.

[0013] A further optimization scheme is to construct positive and negative sample pairs of each voxel block; including the method: For each voxel, a certain regional feature is randomly selected and defined as the anchor point; For each anchor point, a regional feature is selected from the same voxel to form a positive sample pair, and multiple regional features are randomly selected from voxel blocks with different anchor points to form a negative sample pair.

[0014] A further optimization scheme is that the contrast loss function includes:

[0015] in, L cont represents the contrast loss function, M represents the total number of voxel blocks, N Indicates the number of rays within each voxel block; represents the characteristics of the nth ray in voxel block m, represents the positive sample features within the voxel block m, represents the rth ray feature in the voxel block q, that is, the negative sample feature, α Represents a coefficient that controls the scale of the contrast loss.

[0016] The present solution also provides a three-dimensional radiation field modeling system based on a contrast voxel module, which is used to implement the above-mentioned three-dimensional radiation field modeling method based on a contrast voxel module. The system includes: An acquisition module, used for acquiring image data of a target area; A first construction module is used to construct a scene of a target area based on the image data, evenly divide the scene into a plurality of voxel blocks of equal volume according to a voxel ray sampling method, and sample rays from the plurality of voxel blocks; The second building module is used to build a three-dimensional radiation field model of the target area. During the process: Local implicit constraints on the scene: obtain the surrounding points and ray points of each ray, and learn the relationship between the surrounding points and ray points based on Transformer to predict the color and density of the ray points; Global explicit constraints on the scene: obtain positive and negative sample pairs for each voxel block; constrain the positive and negative sample pairs with a contrastive loss function based on a contrastive learning mechanism to ensure that the features within the same voxel are similar, while the features across voxels are different; The generation module is used to generate a three-dimensional radiation field model of the target area.

[0017] The present solution also provides a computer-readable medium having a computer program stored thereon, and the computer program is executed by a processor to implement the three-dimensional radiation field modeling method based on the contrast voxel module as described above.

[0018] Compared with the prior art, the present invention has the following advantages and beneficial effects: 1. The three-dimensional radiation field modeling method, system and medium based on the contrast voxel module provided by the present invention effectively improve the radiation consistency between 3D points in the voxel block through the voxel-based ray sampling strategy and the local implicit constraints of the transformer network in the voxel block, avoiding the problem of inaccurate scene representation caused by sparse input; 2. The three-dimensional radiation field modeling method, system and medium based on the contrast voxel module provided by the present invention; through global explicit contrast regularization, the characteristic consistency between 3D points across voxel blocks is ensured, and the ambiguity and artifact problems of the radiation field in the 3D scene are alleviated; 3. The present invention provides a three-dimensional radiation field modeling method, system and medium based on a contrast voxel module; utilizing a contrast learning module, the generalization ability of the model is enhanced, so that it can generate consistent three-dimensional representations under different sparse conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings: Figure 1 Schematic diagram of the process of three-dimensional radiation field modeling method based on contrast voxel module. DETAILED DESCRIPTION

[0020] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0021] The traditional Neural Radiance Field (NeRF) model performs poorly in the case of sparse input, which is mainly reflected in the difficulty in accurately predicting the density and color of the scene, resulting in inconsistent 3D scene representation, blur or artifacts in the generated image, etc. Most existing technologies rely on dense multi-view input data to construct the radiation field, but lack an effective constraint mechanism under sparse data conditions, resulting in insufficient scene coverage and the inability to correctly describe the radiation field characteristics. In view of this, this solution provides the following embodiments to solve the above technical problems.

[0022] Embodiment 1: This embodiment provides a three-dimensional radiation field modeling method based on a contrast voxel module, such as Figure 1 As shown, including: Step 1, collecting image data of the target area; Step 2: construct a scene of the target area based on the image data, divide the scene evenly into multiple voxel blocks of equal volume according to the voxel ray sampling method, and sample rays from the multiple voxel blocks; Step 2 specifically includes the following methods: S21, configuring scene parameters and voxel block parameters based on image data; S22, dividing the scene evenly into a plurality of cubes of equal size according to the scene parameter and the voxel block parameter, each cube being a voxel block, and each voxel block having a plurality of intersecting rays intersecting with it; S23, randomly select V voxels from all voxel blocks for ray sampling: for voxel block i, randomly select R rays, each of which intersects with multiple points inside voxel block i.

[0023] Specifically, in this solution, the scene is evenly divided into multiple voxel blocks, and the size of each voxel block is ; For each voxel block, R rays are randomly selected, where each ray intersects with multiple 3D points inside the voxel block. The process can be described by the following formula:

[0024] in, r j represents the jth ray, V i represents the i-th voxel block, R i represents the set of rays sampled in voxel block i.

[0025] Step 3: Construct a 3D radiation field model of the target area. During the process: First, perform local implicit constraints on the scene: obtain the surrounding points and ray points of each ray, and learn the relationship between the surrounding points and ray points based on Transformer to predict the color and density of the ray points; this step specifically includes the following methods: S31, for the current ray, randomly sample S points around the line segment where the current ray intersects with the inside of the voxel block as surrounding points, and uniformly sample P points on the line segment where the current ray intersects with the inside of the voxel block as ray points; the ray points are recorded as ; The surrounding points are denoted as ; The formula is as follows: ; ; in, Represents the operation of sampling inside the sphere Q; represents the entry intersection point of the ray with the voxel block; represents the intersection point of the ray with the exit of the voxel block; represents the midpoint between the entry intersection point and the exit intersection point, which is used as the center of the sphere Q. radius represents the radius of the sphere Q, which is used to indicate the size of the sampling area. represents the tth point in the ray point set, Represents the kth point in the surrounding point set.

[0026] S32, extracts initial features from surrounding points based on multi-layer perceptron MLP, and then extracts coded features from the initial features based on the encoder of transformer network; the coded features obtained are: ; ; g s represents the initial features; S33, the decoder based on the Transformer network converts the encoded features into the radiation attribute color and density of the ray point: ; Among them, c p Represents ray point Color, Represents ray point density.

[0027] Second, globally and explicitly constrain the scene: obtain positive and negative samples of each voxel block; learn positive and negative samples based on the contrastive learning mechanism feature contrast to ensure that the similarity of points in the same voxel block in the feature space is higher than that of points between different voxel blocks; the specific method for obtaining positive and negative samples of each voxel block includes: S301, constructing positive and negative sample pairs for each voxel block; this step specifically includes the following method: For each voxel, a certain regional feature is randomly selected and defined as an anchor point; For each anchor point, a regional feature is selected from the same voxel to form a positive sample pair, and multiple regional features are randomly selected from voxel blocks with different anchor points to form a negative sample pair.

[0028] S302, randomly sampling multiple voxel blocks, calculating the feature distance between the positive sample of the same voxel block and the negative sample of different voxel blocks, and constraining the feature distance between the positive samples to be as small as possible and the feature distance between the negative samples to be as large as possible based on the contrast loss function; the contrast loss function includes:

[0029] in, L cont represents the contrast loss function, M represents the total number of voxel blocks, and N represents the number of rays in each voxel block; represents the characteristics of the nth ray in voxel block m, represents the characteristics of the positive samples in the voxel block m, It represents the rth ray feature in the voxel block q, that is, the negative sample feature, and α represents the coefficient, which is used to control the scale of the contrast loss.

[0030] S303, back-propagating the contrast loss function to the neural radiation field model to update the parameters of the neural radiation field model, thereby making the regional features within the voxels more similar and further improving the consistency of the 3D field. The global contrast learning method explicitly ensures the radiation differences between different voxels, thereby enhancing the consistency in the three-dimensional space.

[0031] Step 4: Generate a three-dimensional radiation field model of the target area.

[0032] The scene is implicitly constrained locally. Embodiment 2: This embodiment provides a three-dimensional radiation field modeling system based on a contrast voxel module, which is used to implement the three-dimensional radiation field modeling method based on a contrast voxel module described in Embodiment 1. The system includes: An acquisition module, used for acquiring image data of a target area; A first construction module is used to construct a scene of a target area based on the image data, evenly divide the scene into a plurality of voxel blocks of equal volume according to a voxel ray sampling method, and sample rays from the plurality of voxel blocks; The second building module is used to build a three-dimensional radiation field model of the target area. During the process: Local implicit constraints on the scene: obtain the surrounding points and ray points of each ray, and learn the relationship between the surrounding points and ray points based on Transformer to predict the color and density of the ray points; Global explicit constraints on the scene: obtain positive and negative sample pairs for each voxel block; constrain the positive and negative sample pairs with a contrastive loss function based on a contrastive learning mechanism to ensure that the features within the same voxel are similar, while the features across voxels are different; The generation module is used to generate a three-dimensional radiation field model of the target area.

[0033] Embodiment 3: This embodiment provides a computer-readable medium on which a computer program is stored. The computer program is executed by a processor to implement the three-dimensional radiation field modeling method based on the contrast voxel module as described in Embodiment 1; specifically, the following steps are included: Step 1, collecting image data of the target area; Step 2: construct a scene of the target area based on the image data, divide the scene evenly into multiple voxel blocks of equal volume according to the voxel ray sampling method, and sample rays from the multiple voxel blocks; Step 3: Construct a 3D radiation field model of the target area. During the process: Local implicit constraints on the scene: obtain the surrounding points and ray points of each ray, and learn the relationship between the surrounding points and ray points based on Transformer to predict the color and density of the ray points; Global explicit constraints on the scene: obtain positive and negative sample pairs for each voxel block; constrain the positive and negative sample pairs with a contrastive loss function based on a contrastive learning mechanism to ensure that the features within the same voxel are similar, while the features across voxels are different; Step 4: Generate a three-dimensional radiation field model of the target area.

[0034] This scheme enhances the radiation consistency under sparse input conditions through a contrastive learning strategy. This module can maintain similar radiation characteristics of neighboring points in three-dimensional space, thereby improving the rendering quality of the overall image. The voxel-based light sampling method ensures that the light and multiple points in the voxel share similar radiation characteristics, which is the basis for enhancing the consistency of the three-dimensional scene. This method is applicable to other three-dimensional scene representation systems and has strong versatility. By introducing global contrast constraints, the present invention can effectively distinguish the feature differences between and within voxels, ensuring significant differences in the radiation field between different regions. This design can be applied to other models that need to enhance three-dimensional consistency.

[0035] The method of this solution can be extended to other 3D scene representation or modeling technologies, such as 3D environment perception in autonomous driving, 3D map construction in robot navigation, and scene rendering in virtual reality. These application scenarios also have the problem of sparse input. The contrast voxel module of this solution can significantly improve the performance of the model; the implementation method of contrast loss can be adjusted to a certain extent, such as replacing it with a constraint method based on generative adversarial network (GAN) or information entropy to achieve a similar effect of improving radiation consistency. The light sampling strategy within the voxel can also be replaced by other space partitioning methods (such as octree or KD tree) to adapt to different scene structures and requirements.

[0036] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A three-dimensional radiation field modeling method based on a contrast voxel module, characterized in that: include: Step 1, collecting image data of the target area; Step 2: construct a scene of the target area based on the image data, divide the scene evenly into multiple voxel blocks of equal volume according to the voxel ray sampling method, and sample rays from the multiple voxel blocks; Step 3: Construct a 3D radiation field model of the target area. During the process: Local implicit constraints on the scene: obtain the surrounding points and ray points of each ray, and learn the relationship between the surrounding points and ray points based on Transformer to predict the color and density of the ray points; Global explicit constraints on the scene: obtain positive and negative sample pairs for each voxel block; constrain the positive and negative sample pairs with a contrastive loss function based on a contrastive learning mechanism to ensure that the features within the same voxel are similar, while the features across voxels are different; Step 4: Generate a three-dimensional radiation field model of the target area.

2. The three-dimensional radiation field modeling method based on the contrast voxel module according to claim 1 is characterized in that: The scene of the target area is constructed based on the image data, the scene is evenly divided into a plurality of voxel blocks of equal volume according to a voxel ray sampling method, and rays are sampled from the plurality of voxel blocks; Included methods: S21, configuring scene parameters and voxel block parameters based on image data; S22, dividing the scene evenly into a plurality of cubes of equal size according to the scene parameter and the voxel block parameter, each cube being a voxel block, and each voxel block having a plurality of intersecting rays intersecting with it; S23, randomly select V voxels from all voxel blocks for ray sampling: for voxel block i, randomly select R rays, each of which intersects with multiple points inside voxel block i.

3. The three-dimensional radiation field modeling method based on the contrast voxel module according to claim 2 is characterized in that: The scene parameters include scene range and scene size; the voxel block parameters include voxel block quantity and voxel block size.

4. The three-dimensional radiation field modeling method based on the contrast voxel module according to claim 2 is characterized in that: The method of performing local implicit constraints on the scene includes: For the current ray, randomly sample S points around the line segment where the current ray intersects with the inside of the voxel block as surrounding points, and uniformly sample P points on the line segment where the current ray intersects with the inside of the voxel block as ray points; The initial features are extracted from the surrounding points based on the multi-layer perceptron MLP, and then the encoder Encoder based on the transformer network extracts the encoded features from the initial features; The decoder based on the Transformer network converts the encoded features into the radiation attribute color and density of the ray point.

5. The three-dimensional radiation field modeling method based on the contrast voxel module according to claim 4 is characterized in that: The method of randomly sampling S points around the line segment where the current ray intersects with the inside of the voxel block as surrounding points and uniformly sampling P points on the line segment where the current ray intersects with the inside of the voxel block as ray points includes the following methods: Randomly sample S points according to the following formula to get the surrounding points : ； in, represents the kth point in the surrounding point set, Represents the operation of sampling inside the sphere Q; represents the entry intersection point of the ray with the voxel block; represents the intersection point of the ray with the exit of the voxel block; represents the midpoint between the entry intersection point and the exit intersection point, which is used as the center of the sphere Q. radius represents the radius of the sphere Q, which is used to indicate the size of the sampling area. According to the following formula, P points are uniformly sampled to obtain the ray points : ； in, represents the operation of sampling on a ray segment, Represents the tth point in the ray point set.

6. The three-dimensional radiation field modeling method based on the contrast voxel module according to claim 2, characterized in that: The method of globally and explicitly constraining the scene includes: Construct positive and negative sample pairs for each voxel block; Randomly sample multiple voxel blocks, calculate the feature distances of positive and negative pairs respectively, and constrain the feature distances between positive pairs to be as small as possible and the feature distances between negative pairs to be as large as possible based on the contrast loss function; The contrastive loss function is back-propagated to the neural radiance field model to update the parameters of the neural radiance field model.

7. The three-dimensional radiation field modeling method based on the contrast voxel module according to claim 6, characterized in that: The positive and negative sample pairs of each voxel block are constructed; including method: For each voxel, a certain regional feature is randomly selected and defined as the anchor point; For each anchor point, a regional feature is selected from the same voxel to form a positive sample pair, and multiple regional features are randomly selected from voxel blocks with different anchor points to form a negative sample pair.

8. The three-dimensional radiation field modeling method based on the contrast voxel module according to claim 6, characterized in that: The contrast loss function includes: ； in, L cont represents the contrast loss function, M represents the total number of voxel blocks, and N represents the number of rays in each voxel block; represents the characteristics of the nth ray in voxel block m, represents the characteristics of the positive samples in the voxel block m, It represents the rth ray feature in the voxel block q, that is, the negative sample feature, and α represents the coefficient, which is used to control the scale of the contrast loss.

9. A three-dimensional radiation field modeling system based on a contrast voxel module, characterized in that: A system for implementing the three-dimensional radiation field modeling method based on a contrast voxel module according to any one of claims 1 to 8, the system comprising: An acquisition module, used for acquiring image data of a target area; A first construction module is used to construct a scene of a target area based on the image data, evenly divide the scene into a plurality of voxel blocks of equal volume according to a voxel ray sampling method, and sample rays from the plurality of voxel blocks; The second building module is used to build a three-dimensional radiation field model of the target area. During the process: Local implicit constraints on the scene: obtain the surrounding points and ray points of each ray, and learn the relationship between the surrounding points and ray points based on Transformer to predict the color and density of the ray points; Global explicit constraints on the scene: obtain positive and negative sample pairs for each voxel block; constrain the positive and negative sample pairs with a contrastive loss function based on a contrastive learning mechanism to ensure that the features within the same voxel are similar, while the features across voxels are different; The generation module is used to generate a three-dimensional radiation field model of the target area.

10. A computer readable medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the three-dimensional radiation field modeling method based on the contrast voxel module as described in any one of claims 1 to 8.

Images