Hashgrid-based modeling method for neural radiation field and two-dimensional volume density segmentation

Abstract

The present invention discloses a modeling method of neural radiation field and two-dimensional volume density segmentation based on Hashgrid, which includes the following modeling steps: step 1, firstly, collecting human body images from multiple angles through an ordinary RGB camera; step 2, using colmap to perform camera intrinsic parameter calibration and extrinsic parameter calibration on the collected images; step 3, performing pixel segmentation and image preprocessing on the collected images, and removing the background area outside the image to be reconstructed. The present invention collects images of the human body to be reconstructed from various angles, sets the overlapping area between adjacent images, removes the background, and reduces interference. Through multiple iterative training, the rendering is changed from continuous to discrete form, which can quickly perform three-dimensional human body modeling. By substituting sampling points into the calculation, the rendering effect is improved, and the volume density slice map is segmented using u-net, and the noise is removed while the segmentation is performed, the human body is accurately reconstructed, and the modeling efficiency is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of human body modeling, and in particular to a modeling method of neural radiation field and two-dimensional body density segmentation based on Hashgrid. Background Art

[0002] With the development of metaverse and other related technologies, the human body, as one of the important components of the metaverse, the efficiency and effectiveness of human body model reconstruction will directly affect the ecological environment of the metaverse community. In addition, in some medical diagnosis environments, a model with a high degree of detailed restoration will also directly affect the doctor's diagnosis and treatment judgment. Therefore, a human body reconstruction solution with high restoration and high accuracy has high practical value in all of the above fields.

[0003] However, when reconstructing the human body model, the background of the captured image will cause interference and burden in the processing, and the continuous rendering effect is poor, resulting in a low degree of restoration of the human body. Summary of the Invention

[0004] The present invention provides a Hashgrid-based neural radiation field and two-dimensional volume density segmentation modeling method, which can effectively solve the problem raised in the above background technology that when reconstructing the human body model, the background of the collected image will cause interference and burden in the processing, the continuous rendering effect is poor, and the restoration degree of the human body is not high.

[0005] To achieve the above-mentioned object, the present invention provides the following technical solution: a Hashgrid-based neural radiation field and two-dimensional volume density segmentation modeling method, comprising the following modeling steps:

[0006] Step 1: First, use an ordinary RGB camera to collect human body images from multiple angles;

[0007] Step 2: Use colmap to calibrate the camera's intrinsic and extrinsic parameters on the acquired images.

[0008] Step 3: Perform pixel segmentation and image preprocessing on the collected images to remove the background area outside the image to be reconstructed, set the alpha channel of the removed background area to 0, and only retain the main area to be rendered in each image;

[0009] Step 4: Use the pre-processed data to perform perspective synthesis of the area to be reconstructed using the Hashgrid-based neural radiation field to render the entire human body to be reconstructed;

[0010] Step 5: Slice the volume density obtained from the Hashgrid-based neural radiation field;

[0011] Step 6: Segment the volume density after slicing by pixels and calculate the outer contour coordinates of the segmented mask;

[0012] Step 7: Normalize all outer contour pixel coordinates in sequence and combine all pixels to reconstruct a three-dimensional model of the human body to be reconstructed;

[0013] In step 5, after multiple iterations of training, the volume density in the neural radiation field is extracted, and the volume density is sliced according to the quality requirements of the reconstructed model. The volume density is rendered in the 5-dimensional neural radiation field using the classical volume rendering method. The volume density is represented by σ(x), and the preset camera light is represented by r(t)=o+td. Then, the target color C(r) in the rendering is at the nearest boundary t n and the farthest boundary t f The expression is:

[0014] and

[0015] The continuous volume density rendering equation is changed to a discrete form for calculation, which is expressed as:

[0016] and

[0017] where δ i =t i+1 -t i Expressed as the distance between adjacent samples;

[0018] The continuous rendering equation can be rewritten after uniform segmentation as follows:

[0019]

[0020] in

[0021] Then perform uniform sampling within each segment:

[0022] Given the voxel color and volume density in a specified area, the approximate value of the sampling point in the area can be expressed as:

[0023]

[0024] In the entire working area, all voxels below the fixed line of sight are piecewise functions, and their approximate functions can be further expressed as:

[0025]

[0026] A two-stage network is used for training. First, the general shape is roughly trained, and then fine-tuned. In the coarse training stage, the layers are sampled to N cpoints, its rendering equation can be expressed as:

[0027] where w i =T i (1-exp(-σ i δ i );

[0028] Then w i Normalize Get its piecewise function, and then generate N through its inverse transform f points, and render them together with the points obtained by the layered sampling above. After these two samplings, the loss function is finally expressed as follows:

[0029]

[0030] In addition, the position vector x and the direction vector d need to be converted into high-frequency variables, and their position encoding can be expressed as:

[0031] γ(p)=(sin(2 0 πp),cos(2 0 πp),...,sin(2 L-1 πp),cos(2 L-1 πp));

[0032] Finally, the position (x, y, z) and direction (θ, φ) are used as input to generate high-precision rendering effects.

[0033] According to the above technical solution, in step 1, a common visible light camera is used to collect human body data, that is, the image data set should be P = {n|p1, p2, p3...pn} and n> = 30;

[0034] During the data collection process, data collection is required for each angle of the human body to be reconstructed, and the overlapping area between adjacent images is not less than 20%, that is, P1∩P2>=20%, P2∩P3>=20%...Pn-1∩Pn>=20%.

[0035] According to the above technical solution, in step 2, the collected data is input into colmap, colmap first performs pairwise matching between images, and then completes the camera internal and external parameter calibration and camera distortion parameter work;

[0036] Extract the feature points from each image, and then perform feature point matching between two images. Use F(I) to represent the feature points around pixel I.

[0037] For the same image I, J, consider the feature point f∈F(I), and then find the adjacent feature vector fnn∈F(J), which can be expressed as

[0038] According to the above technical solution, in step 3, the u-net human body segmentation network is used to segment the collected multi-angle human body data to obtain a segmented human body mask. The background irrelevant to the reconstructed content in the original image is removed based on the obtained mask, where l:Ω→{1,2,...,K} is the positive label of each pixel and the weight is: The final boundary calculation formula is:

[0039]

[0040] According to the above technical solution, in step 4, the pre-processed image is input into the Hashgrid-based neural radiance field together with the pre-calibrated camera internal and external parameters and distortion parameters, and then multi-view synthesis of the target to be reconstructed is started;

[0041] The hashgrid used is a spatial hash function, which is expressed as:

[0042]

[0043] In the above formula, Represents bitwise XOR operation;

[0044] is the only large prime number;

[0045] T is the dimension of the feature vector.

[0046] According to the above technical solution, in step 6, the sliced volume density image is segmented twice using u-net to obtain the outer contour pixel points of the object under the density layer.

[0047] According to the above technical solution, in step 7, after all the sliced density maps are segmented, each outer contour pixel point is spatially normalized according to its original layer order in space to reconstruct a three-dimensional point cloud model of the human body to be reconstructed.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. After multiple iterative training sessions, multi-view synthesis achieves ideal results. The volume density in the neural radiation field is extracted and sliced, and the volume density rendering equation is changed from continuous to discrete. Each segment is then uniformly sampled. A two-stage network is used for training to accelerate rendering efficiency. The normalized function is then used to obtain a piecewise function. The resulting points are then rendered, allowing the sampling points to contribute more to color calculations. The position and direction vectors are then converted into high-frequency variables to obtain clearer detail features, resulting in better rendering effects.

[0050] 2. By collecting various angles of the human body to be reconstructed and setting the overlapping area between adjacent images, the collected data is input into colmap to complete the camera internal and external parameter calibration and camera distortion parameter work, and the u-net human body segmentation network is used to segment the collected multi-angle human body data to obtain a human body mask. According to the mask, the background irrelevant to the reconstructed content in the original image is removed, and the preprocessed image is input together with the calibrated camera internal and external parameters and distortion parameters into the Hashgrid-based neural radiation field. The reconstructed target is synthesized from multiple perspectives. By setting parameters, interference is eliminated to achieve better synthesis effect of the reconstructed target.

[0051] 3. By using U-Net to segment the body density slice map, the noise points can be effectively removed while correctly segmenting and reconstructing the subject. After obtaining the outer contour coordinate pixel points of all sections, the spatial coordinates are normalized according to the section order, and finally the spatial points are combined into the human body model to be reconstructed.

[0052] In summary, by collecting images of the human body to be reconstructed from various angles and setting the overlapping areas between adjacent images, the background is removed and interference is reduced. Through multiple iterative training, the rendering is changed from continuous to discrete form, which can quickly perform three-dimensional human body modeling. By substituting the sampling points into the calculation, the rendering effect is improved, and the u-net segmentation body density slice map is used to segment and remove noise at the same time, accurately reconstruct the human body, and effectively improve the modeling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0054] Figure 1 It is a schematic diagram of the human body modeling process of the present invention;

[0055] Figure 2 Schematic diagram of the original data overlap of the present invention;

[0056] Figure 3 It is a schematic diagram of colmap calibration of the shooting device of the present invention;

[0057] Figure 4 It is a schematic diagram of the data preprocessing results of the present invention;

[0058] Figure 5 It is a schematic diagram of the rendering result of the present invention;

[0059] Figure 6 is a schematic diagram of a body density slice of the present invention;

[0060] Figure 7 is a schematic diagram of volume density segmentation of the present invention;

[0061] Figure 8 It is a schematic diagram of the modeling results of the present invention. DETAILED DESCRIPTION

[0062] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0063] Example: Figure 1 As shown, the present invention provides a technical solution, a modeling method of neural radiation field and two-dimensional volume density segmentation based on Hashgrid, including the following modeling steps:

[0064] Step 1: Use an ordinary visible light camera to collect human body data, that is, the image dataset should be P = {n|p1,p2,p3...pn} and n>=30. This step does not require any processing of the collected data. However, it should be noted that during the data collection process, data must be collected from all angles of the reconstructed human body and the overlapping area between adjacent images must be no less than 20%, that is, P1∩P2>=20%, P2∩P3>=20%...Pn-1∩Pn>=20%;

[0065] Step 2: Input the collected data into colmap and let colmap complete the camera internal and external parameter calibration and camera distortion parameter work of this set of data. The main principle of this calibration step is to extract the feature points in each picture, and then match the feature points between the pictures. Use F(I) to represent the feature points around pixel I. For I and J in the same image, consider the feature point f∈F(I) and then find the adjacent feature vector fnn∈F(J), which can be expressed as In this step, the original image is used without any preprocessing;

[0066] Step 3: Use the u-net human body segmentation network to segment the collected multi-angle human body data and obtain the segmented human body mask. According to the obtained mask, remove the background irrelevant to the reconstructed content in the original image, where l:Ω→{1,2,...,K} is the positive label of each pixel and the weight is: The final boundary calculation formula is:

[0067]

[0068] Step 4: The pre-processed image is input into the Hashgrid-based neural radiance field along with the pre-calibrated camera internal and external parameters and distortion parameters, and then multi-view synthesis of the target to be reconstructed is started;

[0069] The hashgrid used here is a spatial hash function, which is expressed as:

[0070]

[0071] In the above formula, Represents a bitwise XOR operation;

[0072] π i is the only large prime number;

[0073] T is the dimension of the feature vector;

[0074] Step 5. After multiple iterations of training, the effect of multi-view synthesis reaches a relatively ideal level. Start to extract the volume density in the neural radiation field and slice the volume density. In this step, this method mainly uses the classical volume rendering method to render the volume density in the 5-dimensional neural radiation field. The volume density is expressed as σ(x), and the preset camera light is expressed as r(t) = o + td. Then the target color C(r) in the rendering is at the nearest boundary t n and the farthest boundary t f The expression is:

[0075] and

[0076] However, the volume density rendering equation is continuous, so it needs to be converted into discrete form for calculation, which is expressed as:

[0077] and

[0078] where δ i =t i+1 -t i Expressed as the distance between neighboring samples.

[0079] The continuous rendering equation can be rewritten after uniform segmentation as follows:

[0080]

[0081] in

[0082] Then perform uniform sampling within each segment:

[0083] However, if the voxel color and volume density are given in a specified area, the approximate value of the sampling point in the area can be expressed as:

[0084]

[0085] If it is assumed that all voxels below the fixed line of sight in the entire working area are piecewise functions, then the approximate function can be further expressed as:

[0086]

[0087]

[0088] In addition, in order to speed up the rendering efficiency, this scheme adopts a two-level network for training. First, the general shape is roughly trained, and then fine-tuned. In the rough training stage, the layers are sampled to N c points, its rendering equation can be expressed as:

[0089] where w i =T i (1-exp(-σ i δ i );

[0090] Then w i Normalize Get its piecewise function, and then generate N through its inverse transform f points, and perform fine rendering together with the points obtained by the layered sampling above. Through these two samplings, the included sampling points will have more contributions to the color calculation. Finally, the loss function is expressed as follows:

[0091]

[0092] In addition, the position vector x and the direction vector d need to be converted into high-frequency variables to obtain clearer detail features. The position encoding can be expressed as:

[0093] γ(p)=(sin(2 0 πp),cos(2 0 πp),...,sin(2 L-1 πp),cos(2 L-1 πp));

[0094] Finally, the position (x, y, z) and direction (θ, φ) are used as input to generate high-precision rendering effects.

[0095] Step 6: Use u-net to perform secondary segmentation on the sliced volume density image to obtain the outer contour pixel points of the object under the density layer;

[0096] Step 7: After segmenting all the sliced density maps, each outer contour pixel is spatially normalized according to its original layer order in space to reconstruct a 3D point cloud model of the human body to be reconstructed.

[0097] like Figure 2 As shown in the figure, in step 1, all angles need to be covered during data collection and the overlapping area between adjacent images must be guaranteed to be more than 20%. The reason is that only when all surfaces in the training data are covered can the subsequent perspective synthesis work be guaranteed to accurately restore every detail of the human body in space. In addition, when using colmap to calibrate internal and external parameters in the second step, colmap first needs to match the images. Therefore, if the overlapping area between images is too small, it will affect the accuracy of the second step operation.

[0098] like Figure 3 As shown in Figure 2, the reason why no preprocessing is done on the image in step 2 is that first, the focal length, basic and other intrinsic parameters of the data acquisition device need to be determined based on the original image. At the same time, the external parameters between the devices at two time points when the pictures are taken need to be determined based on the relative positions between the images. If de-distortion, scaling and other preprocessing are performed in this step, it will cause deviations or even errors in colmap when performing image matching and calculating internal and external parameters, which will further affect the accuracy of subsequent operations.

[0099] like Figure 4 As shown, in step 3, u-net is used to perform pixel segmentation on the image collected in the first step, and the original image is preprocessed according to the mask obtained after segmentation. First, the RGB image is converted into an RGBA image, and then the areas outside the mask are cropped and the alpha channels of these areas are set to 0, so that the non-main part becomes transparent. The reason for using this method for preprocessing is to eliminate the interfering environment to avoid negative impact on the results of the fourth step perspective synthesis.

[0100] like Figure 5 As shown, in step 4, the data pre-processed in the previous steps are mainly input into the neural radiation field, and the neural radiation field is used to perform iterative training of perspective synthesis.

[0101] like Figure 6 As shown, in step 5, the main task is to extract the volume density after the view synthesis training is completed and slice the obtained volume density according to the quality requirements of the reconstructed model. The model quality is proportional to the number of slices.

[0102] like Figure 7As shown in Figure 6, in step 6, the volume density slice map obtained in step 5 is segmented using U-Net. Because some noise is generated during the rendering process, there will also be some volume density noise after extracting the volume density. The key to this step is to effectively remove the noise while correctly segmenting and reconstructing the subject. Figure 7 The red outer contour is the correctly segmented area, and the other two areas without red outer contours are noise points.

[0103] like Figure 8 As shown, in step 7, after obtaining the outer contour coordinate pixel points of all sections, the spatial coordinates are normalized according to the section order, and finally the spatial points are combined into the human body model to be reconstructed.

[0104] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A Hashgrid-based modeling method for neural radiation field and two-dimensional volume density segmentation, characterized by: The modeling steps include the following: Step 1: First, use an ordinary RGB camera to collect human body images from multiple angles; Step 2: Use colmap to calibrate the camera's intrinsic and extrinsic parameters on the acquired images. Step 3: Perform pixel segmentation and image preprocessing on the collected images to remove the background area outside the image to be reconstructed, set the alpha channel of the removed background area to 0, and only retain the main area to be rendered in each image; Step 4: Use the pre-processed data to perform perspective synthesis of the area to be reconstructed using the Hashgrid-based neural radiation field to render the entire human body to be reconstructed; Step 5: Slice the volume density obtained from the Hashgrid-based neural radiation field; Step 6: Segment the volume density after slicing by pixels and calculate the outer contour coordinates of the segmented mask; Step 7: Normalize all outer contour pixel coordinates in sequence and combine all pixels to reconstruct a three-dimensional model of the human body to be reconstructed; In step 5, after multiple iterations of training, the volume density in the neural radiation field is extracted, and the volume density is sliced according to the quality requirements of the reconstructed model. The volume density is rendered in the 5-dimensional neural radiation field using the classical volume rendering method. The volume density is represented by σ(x), and the preset camera light is represented by r(t)=o+td. Then, the target color C(r) in the rendering is at the nearest boundary t n and the farthest boundary t f The expression is: and The continuous volume density rendering equation is changed to a discrete form for calculation, which is expressed as: and where δ i =t i+1 -t i Expressed as the distance between adjacent samples; The continuous rendering equation is rewritten after uniform segmentation as follows: in Then perform uniform sampling within each segment: Given the voxel color and volume density in a specified area, the approximate value of the sampling point in the area is expressed as: In the entire working area, all voxels below the fixed line of sight are piecewise functions, and their approximate functions are further expressed as: A two-stage network is used for training. First, the general shape is roughly trained, and then fine-tuned. In the coarse training stage, the layers are sampled to N c points, and its rendering equation is expressed as: among themw i =T i (1-exp(-σ i d i ); Then w i Normalize Get its piecewise function, and then generate N through its inverse transform f points, and render them together with the points obtained by the layered sampling above. After these two samplings, the loss function is finally expressed as follows: In addition, the position vector x and the direction vector d need to be converted into high-frequency variables, and their position encoding is expressed as: γ(p)=(sin(2 0 πp),cos(2 0 πp),...,sin(2 L-1 πp),cos(2 L-1 πp)); Finally, the position (x, y, z) and direction (θ, φ) are used as input to generate a high-precision rendering effect.

2. The Hashgrid-based neural radiation field and two-dimensional volume density segmentation modeling method according to claim 1 is characterized in that: In step 1, a common visible light camera is used to collect human body data, that is, the image dataset should be P = {n|p1, p2, p3...pn} and n>=30; During the data collection process, data collection is required for each angle of the human body to be reconstructed, and the overlapping area between adjacent images is not less than 20%, that is, P1∩P2>=20%, P2∩P3>=20%...Pn-1∩Pn>=20%.

3. The Hashgrid-based neural radiation field and two-dimensional volume density segmentation modeling method according to claim 1 is characterized in that: In step 2, the collected data is input into colmap, which first performs pairwise matching between images and then completes the camera internal and external parameter calibration and camera distortion parameter work; Extract the feature points from each image, and then perform feature point matching between two images. Use F(I) to represent the feature points around pixel I. For the same image I, J, consider the feature point f∈F(I), and then find the adjacent feature vector fnn∈F(J), which is expressed as 4. The Hashgrid-based neural radiation field and two-dimensional volume density segmentation modeling method according to claim 1 is characterized in that: In step 3, the u-net human body segmentation network is used to segment the collected multi-angle human body data to obtain a segmented human body mask. The background irrelevant to the reconstructed content in the original image is removed based on the obtained mask, where l:Ω→{1,2,...,K} is the positive label of each pixel and the weight is: The final boundary calculation formula is:

5. The Hashgrid-based neural radiation field and two-dimensional volume density segmentation modeling method according to claim 1 is characterized in that: In step 4, the pre-processed image is input into the Hashgrid-based neural radiance field together with the pre-calibrated camera internal and external parameters and distortion parameters, and then multi-view synthesis of the target to be reconstructed is started; The hashgrid used is a spatial hash function, which is expressed as: In the above formula, Represents bitwise XOR operation; is the only large prime number; T is the dimension of the feature vector.

6. The Hashgrid-based neural radiation field and two-dimensional volume density segmentation modeling method according to claim 1 is characterized in that: In step 6, the sliced volume density image is segmented twice using U-net to obtain the outer contour pixel points of the object under the density layer.

7. The Hashgrid-based neural radiation field and two-dimensional volume density segmentation modeling method according to claim 1 is characterized in that: In step 7, after all the sliced density maps are segmented, each outer contour pixel point is spatially normalized according to its original layer number order in space to reconstruct and generate a three-dimensional point cloud model of the human body to be reconstructed.

Images