Volume video generation method for real-time rendering of mobile terminal

Through neural surface modeling and semantic label registration technology, combined with sparse reasoning engine, the problem of large computing power consumption and slow modeling speed in mobile volume video rendering is solved, and efficient real-time rendering on mobile devices is achieved, satisfying mobile The end user needs high frame rate and low latency for volumetric video.

CN118918233APending Publication Date: 2024-11-08SHANGHAI JIAOTONG UNIV +1
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410962149.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing video processing technology based on model keyframes and skeletal animations has problems such as large computing power and slow modeling speed on the mobile terminal, making it difficult to render volume videos in real time.

Method used

Using methods based on neural surface modeling, semantic label registration and sparse reasoning techniques, we take still photos by surrounding distributed RGB camera arrays to generate semantic labels and register them, and generate textures after segmentation processing, combining mobile inference The engine renders it to achieve real-time rendering without cloud computing.

Benefits of technology

60FPS real-time rendering is achieved on mobile devices, with a peak signal-to-noise ratio of between 29-31 decibels and a video fluid volume of between 4-10MB/s, which significantly improves the rendering speed and efficiency of volume videos.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918233A_ABST
    Figure CN118918233A_ABST
Patent Text Reader

Abstract

A volume video generation method for real-time rendering of a mobile terminal comprises the steps that static photos are shot through an RGB camera array distributed in a surrounding mode and then modeled into a triangular mesh model, a model semantic tag is generated in the pre-registration stage, registration errors are measured after all frames are registered to adjacent frames, and after a volume video is segmented according to the registration errors, the volume video is subjected to image segmentation; and performing post-registration and generating textures, generating a computational graph for playing of a mobile terminal inference engine (MNN) according to a post-registration result and segmentation information, and packaging and compressing the computational graph, the model and the textures for playing of the mobile terminal or a desktop through the MNN. Based on neural surface modeling, semantic tag registration and sparse reasoning technologies, the volume video can be rendered on mobile equipment in real time, cloud computing auxiliary rendering is not needed, the playing peak signal-to-noise ratio is between 29 decibels and 31 decibels, the video stream volume is between 4 MB / s and 10 MB / s, rendering can be carried out on iPhone X equipment in a 60FPS mode through actual measurement, and rendering can be carried out on an RTX3090 host in a 1000FPS mode through actual measurement.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a technology in the field of image processing, and specifically to a volumetric video generation method for real-time rendering on a mobile terminal. Background Art

[0002] With the rapid development of virtual reality technology, volumetric video, as the next generation of multimedia technology, has added a rich interactive experience to the traditional video field (game production, clothing display). When mobile users watch volumetric videos, they can not only zoom and pause, but also switch different perspectives and browse the video content from all directions to get an immersive viewing experience. Existing video processing technologies based on model keyframes and skeletal animation have the following problems: skeletal animation takes up too much computing power and has a slow modeling speed. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a volumetric video generation method for real-time rendering on mobile terminals. Based on neural surface modeling, semantic label registration, and sparse reasoning technology, the volumetric video can be rendered in real time on mobile devices without the need for cloud computing to assist in rendering. The peak signal-to-noise ratio of playback is between 29-31 decibels, and the video stream volume is between 4-10MB / s. In actual measurements, it can be rendered at 60FPS on an iPhone X device and at 1000FPS on an RTX3090 host.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a volumetric video generation method for real-time rendering on a mobile terminal. A static photo is taken by a surrounding RGB camera array and then modeled as a triangular mesh model. In a pre-registration stage, a model semantic label is generated and all frames are registered to adjacent frames and then the registration error is measured. After the volumetric video is segmented according to the registration error, post-registration is performed and texture is generated. Then, a calculation graph for mobile terminal reasoning engine (MNN) playback is generated according to the post-registration result and segmentation information, and the calculation graph is packaged with the model and texture for playback on a mobile terminal or desktop through the MNN.

[0006] The RGB camera array uses a camera calibration algorithm to obtain the position and distortion coefficient of each camera before shooting; when shooting, the camera is exposed to obtain photos taken from multiple perspectives within a period of time; after shooting, the photos are corrected by the distortion coefficient, and then the foreground segmentation neural network is used to obtain the segmentation results of the photos.

[0007] The modeling is as follows: before modeling, the subject bounding box is obtained through the foreground segmentation result and the camera parameters; during modeling, the surface signed distance function (SDF) is modeled using the accelerated NeuS algorithm; after modeling, the surface mesh is extracted from the implicit field using the marching cube algorithm, and the number of triangular mesh faces is reduced to a reasonable range using the face reduction algorithm.

[0008] The pre-registration generates semantic labels for mesh vertices of each frame and extracts deformation nodes of the mesh before registration. During registration, the Fast-RNRR algorithm is used to register each frame of the mesh to the adjacent frame; after registration, the registration error is measured and recorded.

[0009] The post-registration is initialized according to the pre-registration result, and a more accurate registration result is obtained by lowering the weight of the regularization term.

[0010] The texture generation described above uses a texture coordinate expansion algorithm to generate texture coordinates of a key frame mesh, and then generates textures based on multi-viewing images and foreground segmentation results.

[0011] The present invention relates to a volumetric video generation system for realizing the above method, comprising: a shooting unit, a static modeling unit, a pre-registration unit, a video segmentation unit, a post-registration unit, a texture generation unit, a calculation graph generation unit, a resource packaging unit and a playback unit adapted to mobile devices and computer hosts, wherein: the shooting unit uses an RGB camera array to shoot model movements; the static modeling unit uses an accelerated NeuS algorithm to model a surface triangular mesh; the pre-registration unit aligns each frame of mesh to an adjacent frame; the video segmentation unit divides the video into several segments according to a pre-registration error, and selects a key frame from each segment; the post-registration unit selects a registration object according to a video segmentation result; the texture generation unit generates texture coordinates and textures of each frame; the calculation graph generation unit generates a calculation graph according to a video segmentation result, a control node and a post-registration result; the resource packaging unit encodes a texture sequence into a video, and packages and compresses the calculation graph, mesh index and texture coordinates; the playback unit downloads the resources required for the volumetric video from the cloud, unpacks them, allocates memory to a user device, performs real-time inference, and renders them on the screen, and can switch the playback perspective in real time according to user interaction. Technical Effects

[0012] The present invention realizes a complete volumetric video generation link through volumetric video segmentation technology based on registration error and storage volume, in conjunction with skeletal animation sparse reasoning based on mobile terminal reasoning engine (MNN), including 24-view RGB shooting, fast neural surface reconstruction, semantic label registration (including pre-registration and post-registration), volumetric video segmentation, volumetric video packaging scheme, and mobile device playback scheme. Compared with the prior art, the present invention can reconstruct volumetric video using only 24 RGB cameras, which greatly reduces the equipment cost compared with the deep camera array capture method; for 50,000 faces and 200 frames of volumetric video, the transmission overhead of the present invention is only about 73.3MB, and the peak signal-to-noise ratio is about 29dB-31dB, which can render volumetric video in real time at 60FPS on a general smart phone. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is an overall flow chart of the embodiment;

[0014] Figure 2 It is a schematic diagram of a shooting scheme in an embodiment;

[0015] Figure 3 It is a schematic diagram of the effect of the embodiment;

[0016] Figure 4 This is the system module block diagram. DETAILED DESCRIPTION

[0017] like Figure 1 As shown, this embodiment relates to a volumetric video generation method for real-time rendering on a mobile terminal, including:

[0018] Step 1: Shooting stage, including:

[0019] 1.1. Open up a 5m x 5m open space and lay a green rubber floor. Build several 3m high pillars around it. 24 RGB cameras are hung on the pillars, facing the center of the stage, and distributed as follows: Figure 2 To ensure bright and uniform lighting, the studio is covered with white cloth, high-power light sources are set up behind each side of the cloth, and several white LED tubes are hung on the top.

[0020] 1.2. Make a calibration plate in AprilGrid format, erect it in front of all cameras in turn, and collect corresponding photos; identify the collective features of the collected photos and perform corresponding calculations, obtain the camera's intrinsic parameters (focal length, principal point offset) and distortion coefficient, and erect the calibration plate in the center of the venue, and calculate the camera's extrinsic parameters (camera position) according to the different relative viewing angles of different cameras.

[0021] 1.3. Shooting and transmitting data: 24 cameras form an RGB camera array, and the central control host generates an exposure trigger signal and transmits it to each camera via a daisy chain; the 24 cameras are divided into three groups and the output data is transmitted to the data receiving host via a network cable and a 10G network card, and written to the solid-state drive in real time. After shooting, the shooting data of the three receiving hosts will be uploaded to the cloud server for further processing.

[0022] In this embodiment, the model performs relatively slow movements during the performance, and the camera frame rate is determined according to the range of the model's pre-rehearsal movements, with a typical shooting frame rate of 12 frames per second. The camera exposure time is between 8 milliseconds and 10 milliseconds, and the aperture size is about 2.8 / f. These two exposure parameters will be appropriately adjusted according to the model's clothes and the light and shadow conditions of the studio.

[0023] 1.4. Preprocess photos: After dedistorting all images according to the camera distortion coefficients obtained by intrinsic calibration, the foreground mask is extracted through the foreground segmentation neural network. The visible shell is extracted according to the foreground mask and camera parameters to estimate the bounding box.

[0024] Step 2: Use the fast NeuS algorithm for static modeling, including:

[0025] 2.1. Construct a geometric feature encoding network: Model the space as a signed distance function (SDF), and use a multi-resolution hash encoder and a fast multi-layer perceptron (MLP) as the backbone of the geometric feature encoding network for geometric encoding. The input of the geometric feature encoding network is 3-channel color information, and the output is 1-channel SDF information and 31-channel geometric features.

[0026] 2.2. Training the geometric feature encoding network, including:

[0027] a) In each round of training, a viewpoint is randomly selected from the 24 viewpoints. Each pixel of the photo at this viewpoint corresponds to a ray. Then, several rays are selected from this pile of rays. Then, for each of the selected rays, several points are sampled and sent to the network.

[0028] Preferably, the bounding box obtained in step 1 is divided into a number of grid points according to an occupied grid of a preset resolution, and when sampling points from the light, unoccupied grid points are skipped to accelerate sampling efficiency and improve overall training speed.

[0029] b) Deriving the input coordinate x through the signed distance function, we get the normal n, outputting the tensor z through the geometric feature encoding network, encoding the light direction r to get the tensor r', and then concatenating the three tensors [n, z, r'] to get the input tensor of the coloring MLP, which is then input into the coloring MLP to get the output color;

[0030] c) Rendering based on NeuS theory: Calculate the logistic probability of the sampling point according to the SDF to obtain the occupancy of each sampling point, integrate the occupancy of each sampling point in the penetration order to obtain the overall opacity of the light; integrate the color of each sampling point to obtain the overall color of the light.

[0031] The penetration order means that the sampling points on each ray are accumulated in order from near to far from the camera, that is, NeuS is accumulated from 0 to ∞ according to the weighted value w.

[0032] d) Calculate the color loss, visibility loss and SDF eikonal loss respectively and then weight them to get the total error back propagation, optimize the network parameters and regularly update the occupied grid status, specifically: L NeuS =γ1L color +γ2L eik +γ3L mask , where: γ1γ2γ3 is the loss weight, color loss L color The difference between the overall color of the light and the real color obtained by shooting, taking the norm and averaging; visibility loss L mask Binary cross entropy for the overall light opacity and foreground mask; Eidinburgh loss L eik The square of the normal magnitude is compared to 1.

[0033] Step 3: After extracting the triangular mesh from the implicit neural network using the marching cube algorithm, use the Fast-Quadric-Mesh-Simplification method to reduce the extracted triangular mesh to about 50,000 faces.

[0034] Step 4: Generate vertex semantic labels, including:

[0035] 4.1. Extract the two-dimensional semantic information of the photo based on the human body semantic network. Each pixel corresponds to a semantic label (background, head, torso, left upper arm, left lower arm, left hand, right upper arm, right lower arm, right hand, left thigh, left calf, left foot, right thigh, right calf, right foot); rasterize the frame grid to all shooting angles and obtain the vertex index of each pixel; align the vertex index and semantic label pixel by pixel, and select the semantic label with the most votes to assign it to the vertex index. If the model is wearing non-close-fitting clothing such as a skirt, set the label of non-close-fitting parts such as thighs to the torso; pick out the small label islands, erase their labels, and then perform a breadth-first search for all vertices from the large label block. If a vertex with an erased label is found, set its label to be consistent with the large label block.

[0036] The human semantic network is implemented using, but not limited to, the detectron2 network (github.com / facebookresearch / detectron2).

[0037] 4.2. Use the deformation node algorithm to generate deformation nodes for the triangular mesh: sample a small number of vertices from the vertex set as deformation nodes, and each deformation node stores an affine transformation matrix X i , describing the translation and rotation of the node. At the same time, each deformation node is connected to nearby vertices by weighted edges. Each vertex is deformed by all deformation nodes connected to it, including:

[0038] a. Initialize the priority queue, add all mesh vertices v into the queue, and remove them from the queue in ascending order of x coordinates;

[0039] b. Take out vertices from the queue one by one. For each vertex n taken out i , add it to the deformation node set n, and add n i The subscript sets in the original vertex set v are merged into a tensor N2V, whose shape is

[0040] c. Traversal and n i The vertex v whose geodesic distance is within R j , v j Remove from the priority queue;

[0041] d. v j The subscripts j and n in the original vertex set v i The subscript i in the deformation node set n forms a tuple (j, i), and all such tuples are merged into a tensor E2VN with a shape of

[0042] e. According to v j and n i The connection weight w is calculated by using the geodesic distance d between ′ =(1-d 2 / R 2 ) 3 , w ′ Divide all the v j The weights of the connected deformation nodes are summed to obtain the normalized weight w, and all w are merged into a tensor E2W with a shape of Represents the weight of the edge connecting the deformation node and the vertex;

[0043] 4.3. Use FastRNRR as the backbone of the registration algorithm for pre-registration: Perform the following operations on each frame. For the kth frame, register it to the k-7, k-6, ..., k-1, k+1, ..., k+7 frames, specifically including: first, form a semantic point pair M, traverse the vertices of the source triangle mesh, search for the vertex of the target triangle mesh that is closest to it and has the same semantic label, and add this pair of points to M; then traverse the target triangle mesh, search for the vertex of the source triangle mesh that is closest to it and has the same semantic label, and add this pair of points to M. Finally, use the L-BFGS algorithm to solve the registration optimization term L = L align +β1L reg +β2L rot , and obtain the aligned triangular mesh and control node affine transformation matrix X.

[0044] 4.4 Measuring pre-registration error: calculating chamfer distance Distance from Hausdorf Among them: the vertex set of the target frame is v, and the vertex set of the registration result is v and They are the vertex sets from the target frame vertex v and the registration result. Select the vertices in .

[0045] Step 5: Segment the volumetric video based on an ordered set: define an ordered set G = {(i1, j1, k1), ..., (i g , j g , k g )}, looking for a set of optimal partitions G * , so that the total target loss OPT(1, N) is minimized, and the optimization goal of key frame division is established Where: i, j, k represent the start frame, end frame and key frame of the video segment respectively, i1=1, j g =N, i≤k≤j, N is the total number of frames, g is the number of segments, and the overhead of dividing the i-th frame to the j-th frame into a whole segment Registration loss overhead Storage volume overhead α1 is a parameter that controls the ratio of CD and MD, E is a dimension adjustment parameter, S0 represents the fixed storage volume of the segment, ΔS represents the marginal storage volume, and M is a dimension adjustment parameter used to control a reasonable segment length to balance the ratio of storage overhead and registration loss within a reasonable range.

[0046] The segmentation is such that adjacent segments are connected end to end.

[0047] In this embodiment, the value range of α1 is [0.75, 0.99]; E=0.01m, S0=2MB, ΔS=0.05MB, and the value range of M is [0.1MB, 0.5MB].

[0048] Step 6, post-registration: After initializing the affine transformation matrix X0 of this registration with the affine transformation matrix X obtained by the pre-registration in step 4, lower the regularization term weight, further align, obtain a more fitted surface and a more accurate affine transformation matrix, lower the regularization term weights β1, β2, and obtain a more fitted registration result, thereby avoiding volumetric video jitter.

[0049] The registration is implemented using, but not limited to, the technology described in "Quasi-Newton Solver for Robust Non-Rigid Registration" by Yuxin Yao et al.

[0050] Step 7: Coordinate expansion and texture reconstruction, including:

[0051] 7.1. After the texture coordinates of the keyframe triangle mesh are expanded, the vertices at the texture seams are divided into multiple parts, that is, the split vertices are in the same position but belong to different texture blocks.

[0052] The coordinate unfolding refers to: unfolding the 3D mesh to the UV plane;

[0053] The unfolding is implemented by using, but not limited to, the technology described in "Feature-based surface parameterization and texture mapping" by Eugene Zhang, Konstantin Mischaikow, etc.

[0054] 7.2. Texture coordinate propagation: Map the V2U in the segmentation result of step 5 to the normal frame to obtain the vertex index and vertex position after the coordinate expansion, and merge them with the texture coordinates of the key frame to obtain the result after the texture coordinate expansion.

[0055] 7.3. Reconstruct the texture of the triangular mesh based on the multi-view images, including:

[0056] a. Rasterize the single-channel depth (the distance from the surface to the camera) to each viewing angle, with the resolution and position consistent with the shooting camera;

[0057] b. Rasterize the dual-channel texture coordinates to each viewing angle, with the resolution and position consistent with the shooting camera;

[0058] c. Rasterize the single-channel first weight to each viewing angle, where the first weight value is equal to the fifth power of the absolute value of the cosine of the normal and the introduced angle;

[0059] d. Calculate the second weight according to the depth information, apply the Laplacian operator to the depth information, remove pixels whose absolute value is less than 0.005, and the second weight of the removed pixels is 0, otherwise it is 1;

[0060] e. The foreground segmentation result is used as the third weight. When any pixel is in the foreground, its third weight is 1, otherwise it is 0;

[0061] f. Calculate the total weight, which is equal to the first weight multiplied by the second weight multiplied by the third weight;

[0062] g. Set a buffer with a length equal to the number of texture pixels, and initially set it to empty;

[0063] h. Compare the texture coordinates, weights, and captured colors pixel by pixel, calculate the buffer index based on the texture coordinates, and if the weight is not 0, put the captured color and weight into the buffer;

[0064] i. Traverse the buffer, calculate the weighted sum of the colors at each position, and store the result in the OpenCV image;

[0065] j. Perform expansion operation on the OpenCV texture image;

[0066] k. Compare the textures before and after dilation pixel by pixel. If the buffer at that position is empty, replace the texture pixel color before dilation with the texture pixel color after dilation.

[0067] 1. Write the pixel-replaced image to disk as the texture image of the triangle mesh of this frame.

[0068] Step 8: Generate a computational graph, including:

[0069] 8.1. Extract the tensors E2VN, E2W, N2V, n, v obtained in steps 4 and 6 and the affine transformation matrix X of the normal frame obtained in step 4.3;

[0070] 8.2. Calculate the intermediate matrices X, Y, and Z, specifically:

[0071] i)v←v k [E2VN[0]], where: refers to the vertex position of the keyframe, Refers to the vertex position. This step changes the shape of v from [#v,3] to [#e,3], that is, v is transferred from the vertex space to the edge space;

[0072] ii)n←v k [N2V[E2VN[1]]], where: Refers to the position of the deformation node. This step changes the shape of n from [#n,3] to [#e,3], that is, n is transferred from the vertex space to the edge space;

[0073] iii) in: Refers to the length of the edge connecting n and v, and the fourth dimension is filled with 1;

[0074] iv)Y←E2W*d, where: is the intermediate variable matrix, representing the weighted d;

[0075] v)Z←E2W*n, where: is the intermediate variable matrix, representing the weighted n;

[0076] vi) Sparse Y from #e dimension to [#v, #n] dimension;

[0077] vii) Sparse Z from #e dimension to [#v, #n] dimension;

[0078] viii)

[0079] ix)

[0080] x)

[0081] 8.3. Generate the MNN calculation graph, specifically: in: X L , X R is the affine transformation matrix of the previous and next frames, X t is the affine transformation matrix at some intermediate moment.

[0082] Step 9: Pack and compress the obtained texture information and set information. Specifically, discard the last frame of each texture image obtained in step 7.3 and encode the remaining texture images in sequence into H.264 format video files. Integer quantize the key frame vertex index obtained in step 3. Quantize the floating point number of the key frame texture coordinates obtained in step 7 and then add them to the calculation result obtained in step 8. Figure 1 And compress to generate a compressed package.

[0083] Step 10: Playback: This includes playback on an iOS mobile device or PC device, specifically including:

[0084] 10.1. Unpack and dequantize the 7z format resources to obtain the calculation graph, vertex index, texture coordinates and video segmentation information;

[0085] 10.2. Pre-allocate memory: import the computation graph into the mobile inference engine (MNN), import the texture video file into iOSAVPlayer or OpenCV VideoCapture, and import the vertex index and texture coordinates into the METAL backend or OpenGL backend;

[0086] 10.3. Play the texture video in a loop. Each time it is updated, obtain the texture image and playback time at that moment.

[0087] 10.4. The current frame number is inferred based on the playback time, and the current segment number and position within the segment are inferred based on the video segment information;

[0088] 10.5. Select the calculation graph according to the segment number and input the position within the segment to obtain the triangular mesh vertex;

[0089] 10.6. Import the triangle mesh vertices into the METAL backend or OpenGL backend. If the frame number is updated, replace the old texture stored in the METAL backend or OpenGL backend with the updated texture.

[0090] 10.7. Get the user's current viewport based on the screen sliding gesture or mouse, and organize it into the observation matrix V and projection matrix P;

[0091] 10.8. According to V, P transforms the triangular mesh from model space to standard space, then rasterizes and colors it into a frame and sends it to the screen buffer.

[0092] After specific practical experiments, the average peak signal-to-noise ratio (PSNR) of the present invention was tested on a "shirt man" dataset with a total of 198 frames and a duration of 16.5 seconds and a "long skirt woman" dataset with a total of 212 frames and a duration of 17.7 seconds. Specifically, the following steps are taken: 1. assigning the modeling camera parameters to the virtual camera; 2. rendering to obtain images of each frame of the model at each perspective; 3. comparing these images with the images taken at the corresponding perspectives to calculate the PSNR; 4. averaging these PSNRs to obtain the final result.

[0093] Compared with the prior art FastRNRR alignment, with a PSNR of 30.53, and the prior art using a greedy method for segmentation, in the case of the same volume, the average PSNR is reduced from 31.14 to 30.85. The average PSNR obtained by the present invention based on the "Men's Shirt" data set is 31.14, the transmission data volume is 73.5MB, and the average transmission bandwidth is 4.5MB / s; the average PSNR obtained by the present invention based on the "Women's Long Skirt" data set is 30.43, the transmission data volume is 104.7MB, and the average transmission bandwidth is 5.9MB / s.

[0094] Compared with the prior art that directly calculates the position of each point, with an FPS of about 8 and a mobile phone temperature stabilized at 44.8°C after one hour of testing, the present invention renders a frame rate of 60FPS on an iPhoneX and a mobile phone temperature stabilized at 41.2°C after one hour of testing.

[0095] In summary, the present invention can achieve a 5-fold increase in the speed of volumetric video production in an environment without a depth camera through the shooting scheme of 4RGB cameras, the fast modeling algorithm, the EDNode-based sparse solution algorithm and its implementation on mobile devices, and shorten the modeling of each frame from 2.5 hours to 0.5 hours.

[0096] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A method for generating volumetric video for real-time rendering on mobile terminals, characterized in that: Still photos are captured by an array of surrounding RGB cameras and then modeled as a triangular mesh. During the pre-registration phase, semantic labels for the model are generated, and all frames are registered to adjacent frames, after which the registration error is measured. The volumetric video is segmented based on the registration error, and then post-registration is performed to generate textures. The post-registration results and segmentation information are then used to generate a computational graph for playback on a mobile inference engine (MNN). This graph, along with the model and textures, is then packaged and compressed for playback on mobile devices or desktops using the MNN. The RGB camera array uses a camera calibration algorithm to obtain the camera position and distortion coefficient of each camera before shooting; during shooting, multi-view photos are obtained by camera exposure; after shooting, the photos are corrected using the distortion coefficients, and then a foreground segmentation neural network is used to obtain the photo segmentation results; The pre-registration generates semantic labels for mesh vertices in each frame and extracts deformation nodes of the mesh before registration. During registration, the Fast-RNRR algorithm is used to register each mesh frame to the adjacent frame. After registration, the registration error is measured and recorded; The post-registration is initialized according to the pre-registration result, and a more accurate registration result is obtained by lowering the weight of the regularization term.

2. The method for generating volumetric video for real-time rendering on a mobile terminal according to claim 1, wherein: The shooting specifically includes: 1.

1. Create a 5m x 5m open space with a green rubber floor. Build several 3m tall pillars around it. Hang 24 RGB cameras on the pillars, facing the center of the stage. To ensure bright and uniform lighting, cover the studio with white cloth. High-power light sources are installed behind each side of the cloth, and several white LED tubes are hung from the ceiling. 1.

2. Prepare an AprilGrid calibration plate and place it in front of all cameras to collect corresponding photos. Identify the collective features of the collected photos and perform corresponding calculations to obtain the camera's intrinsic parameters and distortion coefficients. Set up the calibration plate in the center of the venue and calculate the camera's extrinsic parameters based on the relative viewing angles of different cameras. 1.

3. Capture and transmit data: 24 cameras form an RGB camera array. A central control host generates an exposure trigger signal, which is transmitted to each camera via a daisy chain. The 24 cameras are divided into three groups and the output data is transmitted to the data receiving host via a network cable and a 10G network card. The data is written to the solid-state drive in real time. After the shooting is completed, the shooting data of the three receiving hosts is uploaded to the cloud server for further processing. 1.

4. Preprocessing photos: After dedistorting all images based on the camera distortion coefficients obtained from intrinsic calibration, a foreground mask is extracted using a foreground segmentation neural network. The visual hull is extracted based on the foreground mask and camera parameters to estimate the bounding box.

3. The method for generating volumetric video for real-time rendering on a mobile terminal according to claim 1, wherein: The modeling described above uses the fast NeuS algorithm for static modeling, specifically including: 2.

1. Constructing a geometric feature encoding network: The space is modeled as a signed distance function (SDF). A multi-resolution hash encoder and a fast multi-layer perceptron (MLP) are used as the backbone of the geometric feature encoding network. The input of the geometric feature encoding network is 3-channel color information, and the output is 1-channel SDF information and 31-channel geometric features. 2.

2. Training the geometric feature encoding network, specifically including: In each round of training, a viewpoint is randomly selected from the 24 viewpoints. Each pixel in the photo of this viewpoint corresponds to a ray. Then, several rays are selected from this pile of rays. For each of these rays, several points are sampled and sent to the network. b) Derivative the signed distance function with respect to the input coordinate x to obtain the normal n, output tensor z through the geometric feature encoding network, and tensor r' encoded by the light direction r. The three tensors [n, z, r'] are then concatenated to obtain the input tensor of the shading MLP, which is then fed into the shading MLP to obtain the output color. c) Rendering based on NeuS theory: Calculate the logistic probability of the sampling points based on the SDF to obtain the occupancy of each sampling point. Integrate the occupancy of each sampling point in the penetration order to obtain the overall opacity of the light. Integrate the color of each sampling point to obtain the overall color of the light. d) Calculate the color loss, visibility loss and SDF eikonal loss respectively and then weight them to get the total error back propagation, optimize the network parameters and regularly update the occupied grid status, specifically: L NeuS =γ1L color +γ2L eik +γ3L mask , where: γ1γ2γ3 is the loss weight, color loss L color The difference between the overall color of the light and the actual color obtained by shooting, taking the norm and averaging; visibility loss L mask Binary cross entropy for the overall light opacity and foreground mask; Eikonal loss L eik The square of the normal magnitude is compared to 1.

4. The method for generating volumetric video for real-time rendering on a mobile terminal according to claim 1, wherein: The semantic labels are generated in the following way: 4.

1. Extract the 2D semantic information of the photo based on the human body semantic network. Each pixel corresponds to a semantic label (background, head, torso, left upper arm, left lower arm, left hand, right upper arm, right lower arm, right hand, left thigh, left calf, left foot, right thigh, right calf, right foot). Rasterize the frame grid to all shooting angles and obtain the vertex index of each pixel. Align vertex indices and semantic labels pixel by pixel, and select the semantic label with the most votes and assign it to the vertex index. If the model is wearing non-close-fitting clothing such as a skirt, set the labels of non-close-fitting parts such as thighs to torso. Pick out small label islands and erase their labels. Then, perform a breadth-first search on all vertices starting from the large label block. If a vertex with an erased label is found, set its label to match the large label block. 4.

2. Use the deformation node algorithm to generate deformation nodes for the triangle mesh: sample a small number of vertices from the vertex set as deformation nodes, and each deformation node stores an affine transformation matrix X i , describing the translation and rotation of the node. At the same time, each deformation node is connected to the nearby vertices with weighted edges. Each vertex is affected by all the deformation nodes connected to it and deforms, including: a. Initialize the priority queue, add all mesh vertices v to the queue, and remove them from the queue in ascending order of x coordinates; b. Take out vertices from the queue one by one. For each vertex n taken out i , add it to the deformation node set n, and add n i The subscript sets in the original vertex set v are merged into a tensor N2V, whose shape is c. Traversal and n i Vertex v whose geodesic distance is within R j , v j Remove from the priority queue; d. j Subscripts j and n in the original vertex set v i The subscript i in the deformation node set n forms a tuple (j, i), and all such tuples are merged into a tensor E2VN with the shape e. According to v j and n i The geodesic distance d between them is used to calculate the connection weight w′=(1-d 2 / R 2 ) 3 , divide w′ by all the values ​​that are equal to v j The weights of the connected deformation nodes are summed to obtain the normalized weight w, and all w are merged into a tensor E2W with a shape of Represents the weight of the edge connecting the deformation node and the vertex; 4.

3. Use FastRNRR as the backbone of the registration algorithm for pre-registration: perform the following operations on each frame. For the kth frame, register it to the k-7, k-6, ..., k-1, k+1, ..., k+7 frames, specifically: first, form a semantic point pair M, traverse the vertices of the source triangle mesh, search for the vertex of the target triangle mesh that is closest to it and has the same semantic label, and add this pair of points to M; then traverse the target triangle mesh, search for the vertex of the source triangle mesh that is closest to it and has the same semantic label, and add this pair of points to M. Finally, use the L-BFGS algorithm to solve the registration optimization term L = L align +β1L reg +β2L rot , get the aligned triangular mesh and control node affine transformation matrix X, 4.4 Measuring pre-registration error: calculating chamfer distance Distance to Hausdorff Among them: the vertex set of the target frame is v, and the vertex set of the registration result is v and They are the vertex sets from the target frame vertex v and the registration result .

5. The method for generating volumetric video for real-time rendering on mobile terminals according to claim 1, wherein: The volume video segmentation is to define an ordered set G = {(i1, j1, k1), ..., (i g ,j g , k g )}, seeking a set of optimal partitions G * , so that the total target loss OPT(1, N) is minimized, and the optimization goal of key frame division is established Where: i, j, k represent the start frame, end frame and key frame of the video segment respectively, i1=1, j g =N,i≤k≤j, N is the total number of frames, g is the number of segments, and the overhead of dividing the i-th frame to the j-th frame into a whole segment Registration loss overhead Storage volume overhead α1 is a parameter that controls the ratio of CD and MD, E is a dimension adjustment parameter, S0 represents the fixed storage volume of the segment, ΔS represents the marginal storage volume, and M is a dimension adjustment parameter used to control a reasonable segment length to balance the ratio of storage overhead and registration loss within a reasonable range.

6. The method for generating volumetric video for real-time rendering on a mobile terminal according to claim 1, wherein: The post-registration refers to: after initializing the affine transformation matrix X0 of this registration with the affine transformation matrix X obtained in the pre-registration, lowering the regularization term weight, further registering to obtain a more fitting surface and a more accurate affine transformation matrix, lowering the regularization term weights β1 and β2, and obtaining a more fitting registration result, thereby avoiding volumetric video jitter.

7. The method for generating volumetric video for real-time rendering on mobile terminals according to claim 1, wherein: The generating of texture specifically includes: 7.

1. After unfolding the texture coordinates of the keyframe triangle mesh, split the vertices at the texture seams into multiple ones. That is, the split vertices are in the same position but belong to different texture blocks. 7.

2. Texture Coordinate Propagation: Map the V2U in the segmentation result to the normal frame to obtain the vertex index and vertex position after coordinate expansion, and merge them with the texture coordinates of the key frame to obtain the result of texture coordinate expansion; 7.

3. Reconstruct textures based on multi-view images to obtain a texture image of the triangular mesh, including: a. Rasterize the single-channel depth (the distance from the surface to the camera) to each view angle, with the same resolution and position as the shooting camera; b. Rasterize the dual-channel texture coordinates to each viewing angle, with the resolution and position consistent with the shooting camera; c. Rasterize the single-channel first weight to each viewing angle, where the first weight value is equal to the fifth power of the absolute value of the cosine of the angle between the normal and the introduced angle; d. Calculate the second weight based on the depth information, apply the Laplacian operator to the depth information, and remove pixels whose absolute value is less than 0.

005. The second weight of the removed pixels is 0, otherwise it is 1; e. The foreground segmentation result is used as the third weight. If any pixel is in the foreground, its third weight is 1, otherwise it is 0; f. Calculate the total weight, which is equal to the first weight multiplied by the second weight multiplied by the third weight; g. Set up a buffer with a length equal to the number of texture pixels, initially empty; h. Compare the texture coordinates, weights, and captured colors pixel by pixel, calculate the buffer index based on the texture coordinates, and if the weight is not 0, put the captured color and weight into the buffer; i. Traverse the buffer, calculate the weighted sum of the colors at each position, and store the result in the OpenCV image; j. Perform expansion operation on OpenCV texture image; k. Compare the textures before and after dilation pixel by pixel. If the buffer at that position is empty, replace the texture pixel color before dilation with the texture pixel color after dilation.

1. Write the pixel-replaced image to disk as the texture image of the triangle mesh of this frame.

8. The method for generating volumetric video for real-time rendering on a mobile terminal according to claim 1, wherein the computation graph is obtained by: 8.

1. Extract tensors E2VN, E2W, N2V, n, v and the affine transformation matrix X of the normal frame; 8.

2. Calculate the intermediate matrices X, Y, and Z as follows: i)v←v k [E2VN[0]], where: refers to the vertex position of the keyframe, Refers to the vertex position. This step changes the shape of v from [#v, 3] to [#e, 3], that is, v is transferred from the vertex space to the edge space; ii)n←v k [N2V[E2VN[1]]], where: Refers to the position of the deformation node. This step changes the shape of n from [#n, 3] to [#e, 3], that is, it transfers n from the vertex space to the edge space; iii) in: Refers to the length of the edge connecting n and v, and the fourth dimension is filled with 1; iv)Y ← E2W*d, where: is the intermediate variable matrix, representing the weighted d; v)Z←E2W*n, where: is the intermediate variable matrix, representing the weighted n; vi) Sparse Y, adjusting it from #e dimension to [#v, #n] dimension; vii) Sparse Z, adjusting it from #e dimension to [#v, #n] dimension; viii) ix) x) 8.

3. Generate the MNN calculation graph, specifically: in: X L , X R is the affine transformation matrix of the previous and next frames, X t is the affine transformation matrix at some intermediate moment.

9. The method for generating volumetric video for real-time rendering on a mobile terminal according to claim 1, wherein: The packaging and compression is to package and compress the obtained texture information and set information, specifically: discard the last frame of each frame of the texture image and encode the remaining texture images in sequence into a video file in H.264 format, integer quantize the obtained key frame vertex index, quantize the obtained key frame texture coordinate floating point number, and compress them together with the obtained calculation graph to generate a compressed package.

10. A volumetric video generation system implementing the method according to any one of claims 1 to 9, characterized in that: include: The system comprises a shooting unit, a static modeling unit, a pre-registration unit, a video segmentation unit, a post-registration unit, a texture generation unit, a computational graph generation unit, a resource packaging unit, and a playback unit adapted for mobile devices and computer hosts, wherein: the shooting unit uses an RGB camera array to shoot model movements; the static modeling unit uses an accelerated NeuS algorithm to model a surface triangular mesh; the pre-registration unit aligns each frame of the mesh to the adjacent frame; the video segmentation unit divides the video into several segments according to the pre-registration error, and selects key frames from each segment; the post-registration unit selects the registration object according to the video segmentation result; the texture generation unit generates texture coordinates and textures for each frame; the computational graph generation unit generates a computational graph according to the video segmentation result, the control node, and the post-registration result; the resource packaging unit encodes the texture sequence into a video, and packages and compresses the computational graph, mesh index, and texture coordinates; the playback unit downloads the resources required for the volumetric video from the cloud, unpacks them, allocates memory to the user device, and then performs real-time inference and rendering on the screen, and can switch the playback perspective in real time according to user interaction.

Citation Information

Cited By

  • Spatial-temporal data volume rendering method for unreal engine

    CN120782939A