Three-dimensional model and video fusion method based on 3D Gaussian

Through the 3D Gaussian method, the fusion process of the three-dimensional model and video is automated. The editor only needs to confirm the model position and size, which solves the problems of manual participation and high cost in the traditional method, and achieves efficient automation fusion effect.

CN118887331APending Publication Date: 2024-11-01HANGZHOU ARCVIDEO TECHNOLOGY CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410844682.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-27
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

The fusion process of traditional three-dimensional models and video requires manual participation, the steps are cumbersome and costly, and there is a lack of automated solutions.

Method used

Using a 3D Gaussian method, the 3D Gaussian modeling module, video space restoration module, embedding position confirmation module, mask information generation module and video rendering module are used to realize the automatic integration of three-dimensional models and videos. Only the editor needs to confirm the model Position and size.

Benefits of technology

The automated fusion of three-dimensional models and video is realized, which significantly saves labor costs and simplifies the entire fusion process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118887331A_ABST
    Figure CN118887331A_ABST
Patent Text Reader

Abstract

The invention discloses a 3D Gaussian-based three-dimensional model and video fusion method, which comprises a spatial restoration process and a fusion rendering process, and is characterized in that the spatial restoration process further comprises the following steps: designing a 3D Gaussian modeling module to collect image data of an object to be embedded, and completing 3D Gaussian modeling of the object through the image data; designing a video space restoration module; the fusion rendering process further comprises the following steps: designing an embedding position confirmation module; designing a mask information generation module, performing primary image rendering according to a camera track of a shot video, rendering the 3D Gaussian model generated by the embedding position confirmation module and a simple 3D Gaussian model modeled from the shot video frame by frame according to the camera track to obtain a plurality of groups of image sequences, calculating mask information among the models according to the plurality of groups of image sequences, and determining the mask information according to the mask information. Obtaining a mask image group; and a video rendering module is designed to carry out frame-by-frame rendering according to the multiple groups of image sequences generated by the mask information generation module and the mask image group obtained by calculation, and a final shot video of the embedded model is synthesized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of video processing, and in particular relates to a three-dimensional model and video fusion method based on 3D Gaussian. Background Art

[0002] In the video processing process, the task of the radiation field is to capture a series of scenes at a known viewing angle (including the captured images and the internal and external parameter matrices of the camera corresponding to each image), and train the neural network with the captured data to obtain a 3D static scene representation. The trained radiation field can render images from any angle of the scene (which can be an angle outside the training data). 3D Gaussian modeling is a modeling method similar to the radiation field. First, a sparse scene point cloud is constructed based on the captured images through motion structure matching (Structure From Motion, SFM). This point cloud is used to initialize the 3D Gaussian point cloud set. During the training process, the 3D Gaussian point cloud set is continuously optimized to finally obtain a high-quality 3D static scene representation.

[0003] The fusion of traditional 3D models and filmed videos is usually achieved with the help of 3D modeling software or editing software. It generally includes the following steps: a) Model the real object into a 3D model by scanning or shooting; b) Import the mesh file into the scene in the third-party software to ensure that it is placed in the appropriate position and size; c) In the 3D software, set up a virtual camera to simulate the camera in the video. Adjust the position, direction and parameters of the camera to match the camera in the video; d) In the 3D software, import the video file and use it as the background. Make sure the frame rate and resolution of the video match the actual video; e) Use the rendering function in the 3D software to render the 3D model in the scene and the video background into an image sequence or video.

[0004] The above process of integrating traditional 3D models and shot videos requires the editor to manually work and complete each step step by step. It has high requirements on the editor's proficiency in using software, and each link requires manual participation, so the labor cost of the entire workflow is high. Summary of the invention

[0005] In view of the above problems, the present invention provides a 3D model and video fusion method based on 3D Gaussian, which simplifies the fusion process of 3D model and video. During the entire fusion process, only the position and size of the 3D model need to be manually confirmed, and the rest of the process can be completed automatically, saving a lot of labor costs.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A 3D Gaussian-based three-dimensional model and video fusion method, including a spatial restoration process and a fusion rendering process,

[0008] Among them, the spatial restoration process further includes:

[0009] Design a 3D Gaussian modeling module to collect the image data of the object to be embedded, and complete the 3D Gaussian modeling of the object through the image data for subsequent fusion with the video;

[0010] Design a video spatial restoration module to perform frame extraction on the input captured video, estimate the camera position of the image data obtained by frame extraction to obtain the camera movement trajectory of the captured video, and use the image data after frame extraction by the 3D Gaussian modeling for fast modeling, so as to restore the spatial structure of the captured video and the 3D Gaussian model of the video;

[0011] The fusion rendering process further includes:

[0012] Design an embedding position confirmation module to simultaneously load the 3D Gaussian model of the real object and the simple 3D Gaussian model modeled from the captured video for visualization. The editor adjusts the specific position and size of the 3D Gaussian model of the real object in the visualization interface and exports multiple new 3D Gaussian models;

[0013] Design a mask information generation module. First, perform an image rendering according to the camera trajectory of the captured video, render the 3D Gaussian model generated by the embedding position confirmation module and the simple 3D Gaussian model modeled from the captured video frame by frame according to the camera trajectory to obtain multiple sets of image sequences, and calculate the mask information between the models based on the multiple sets of image sequences to obtain a mask image group;

[0014] Design a video rendering module to calculate and render frame by frame according to the multiple sets of image sequences generated by the mask information generation module and the calculated mask image group to synthesize the captured video of the final embedded model.

[0015] In a possible implementation manner, the design 3D Gaussian modeling module specifically includes:

[0016] Collect an image set Y = y of N real objects in a shooting manner i (0 < i ≤ N) Through structure from motion matching, obtain the corresponding camera parameters ObjectCam. The camera parameters include the focal length, distortion parameters, and internal and external parameter matrices of the camera of the position matrix, and construct an initial scene point cloud P, defined as follows,

[0017] ObjectCam, P = SFM(Y)

[0018] 3D Gaussian modeling (GS_model) models the real object through ObjectCam, the initialized scene point cloud P and N images Y of the real object to generate 3D_GS (3D Gaussian model) A. The 3D Gaussian model A will be used for subsequent video fusion. The modeling process is as follows

[0019] 3D_GS A=GS_model(ObjectCam, P, Y).

[0020] In a possible implementation manner, the design of the video space restoration module specifically includes:

[0021] The input video V is processed by frame extraction to obtain the video image set Y V , process the video image set Y in the same way as the real object image set Y V , get the camera trajectory VideoCam and scene point cloud P of the video v .

[0022] In order to restore the spatial structure of the captured video V, the camera trajectory VideoCam and the scene point cloud P v and video image set Y v Modeling the 3D_GS B of the video, the modeling process is as follows:

[0023] 3D_GS B=GS_model fast_light (VideoCam, P v , Y v ).

[0024] In a possible implementation manner, the design embedding position confirmation module specifically includes:

[0025] Load 3D_GS A and 3D_GS B into the visualization interface of the Gaussian model at the same time, fix the 3D_GS B of the video without spatial transformation, and let the editor move the 3D_GS A of the real object to any position for positioning;

[0026] After confirming the specific fusion position of the real object and the captured video, two new 3D_GS models are exported in the visualization interface, namely 3D_GS AB after the fusion of the real object and the captured video, and 3D_GS A1 of the real object after the spatial position transformation.

[0027] In a possible implementation manner, the design mask information generating module specifically includes:

[0028] According to the camera trajectory VideoCam of the video, the 3D_GS B of the video and the 3D_GS AB after the real object and the video are fused are rendered frame by frame, thereby obtaining two sets of rendered image sequences, namely, the picture sequence B and the image sequence AB;

[0029] The interpolation calculation is performed frame by frame for the image sequence B and the image sequence AB. The calculation process of the mask information of each frame is as follows

[0030]

[0031] The generated Mask image group contains the front-to-back depth relationship of 3D_GS A and 3D_GS B in each frame, which is used for subsequent video rendering.

[0032] In a possible implementation manner, the design video rendering module specifically includes:

[0033] The real object 3D_GS A1 after the spatial position transformation is rendered frame by frame according to the camera trajectory VideoCam of the shooting video to obtain the image sequence A1, and then the mask group Mask, the image sequence A1 and the shooting video are comprehensively rendered to output the final fused video.

[0034] The present invention has the following beneficial effects: the fusion of 3D Gaussian model (three-dimensional model) and shot video is realized through 3D Gaussian modeling; first, the three-dimensional modeling of real objects is completed by 3D Gaussian modeling, and the spatial structure of the shot video is restored with the help of 3D Gaussian modeling, and then the editor places the three-dimensional model in the visualization page at an appropriate position and size, and then the fusion rendering can be automatically realized without adjusting the camera position and parameters. Compared with the traditional method of fusing three-dimensional models and shot videos, the present invention only requires the editor to confirm the placement position and size of the 3D Gaussian model (three-dimensional model), and the fusion of the model and the shot video can be automatically realized, saving a lot of labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of a flow chart of a 3D model and video fusion method based on 3D Gaussian according to an embodiment of the present invention;

[0036] Figure 2 It is a schematic diagram of the fusion effect in a specific application embodiment. DETAILED DESCRIPTION

[0037] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0038] See Figure 1 , which shows a step flowchart of a 3D model and video fusion method based on 3D Gaussian in an embodiment of the present invention, including a space restoration process and a fusion rendering process. The space restoration process further includes: designing a 3D Gaussian modeling module to collect image data of the object to be embedded, and completing the 3D Gaussian modeling of the object through the image data for subsequent fusion with the video; designing a video space restoration module to perform frame extraction on the input captured video, estimate the camera position of the image data obtained by frame extraction to obtain the camera movement trajectory of the captured video, and use the image data after frame extraction for fast modeling by the 3D Gaussian modeling to restore the spatial structure of the captured video and the 3D Gaussian model of the video. The fusion rendering process further includes: designing an embedding position confirmation module to simultaneously load and visualize the 3D Gaussian model of the real object and the simple 3D Gaussian model modeled from the captured video, allowing the editor to adjust the specific position and size of the 3D Gaussian model of the real object in the visualization interface, and export multiple new 3D Gaussian models; designing a mask information generation module to first perform an image rendering according to the camera trajectory of the captured video, and render the 3D Gaussian model generated by the embedding position confirmation module and the simple 3D Gaussian model modeled from the captured video frame by frame according to the camera trajectory to obtain multiple groups of image sequences, and calculate the mask information between the models based on the multiple groups of image sequences to obtain a mask image group; designing a video rendering module to calculate and render frame by frame according to the multiple groups of image sequences generated by the mask information generation module and the calculated mask image group to synthesize the captured video of the final embedded model.

[0039] In a 3D model and video fusion method based on 3D Gaussian in another embodiment of the present invention, the design of the 3D Gaussian modeling module specifically includes:

[0040] An image set Y = y of N real objects collected by shooting i (0 < i ≤ N) Through structure from motion matching, obtain the corresponding camera parameters ObjectCam, where the camera parameters include focal length, distortion parameters, and internal and external parameter matrices of the camera for the position matrix, and construct an initial scene point cloud P, defined as follows,

[0041] ObjectCam, P = SFM(Y)

[0042] 3D Gaussian modeling (GS_model) models the real object through ObjectCam, the initialized scene point cloud P and N images Y of the real object to generate 3D_GS (3D Gaussian model) A. The 3D Gaussian model A will be used for subsequent video fusion. The modeling process is as follows

[0043] 3D_GS A=GS_model(ObjectCam, P, Y).

[0044] In a 3D Gaussian-based three-dimensional model and video fusion method according to another embodiment of the present invention, designing a video space restoration module specifically includes:

[0045] The input video V is processed by frame extraction to obtain the video image set Y V , process the video image set Y in the same way as the real object image set Y V , get the camera trajectory VideoCam and scene point cloud P of the video v ;

[0046] In order to restore the spatial structure of the captured video V, the camera trajectory VideoCam and the scene point cloud P v and video image set Y v Modeling 3D_GS B of the video. The modeling process of 3D_GS B is different from that of the real object 3D_GS A. The modeling of 3D_GS B does not need to restore every detail of the video, but quickly model a rough structure of the space. It is a lightweight and fast modeling. The modeling process is as follows:

[0047] 3D_GS B=GS_model fast _ light (VideoCam, P v , Y v ).

[0048] In a 3D Gaussian-based three-dimensional model and video fusion method according to another embodiment of the present invention, designing an embedding position confirmation module specifically includes:

[0049] Load 3D_GS A and 3D_GS B into the visualization interface of the Gaussian model at the same time. Fix the 3D_GS B of the video without spatial transformation. The editor moves the 3D_GS A of the real object to any position for positioning. During positioning, various operations such as scaling can be performed to confirm the emission situation.

[0050] After confirming the specific fusion position of the real object and the captured video, two new 3D_GS models are exported in the visualization interface, namely 3D_GS AB after the fusion of the real object and the captured video, and 3D_GS A1 of the real object after the spatial position transformation.

[0051] In a 3D Gaussian-based 3D model and video fusion method according to another embodiment of the present invention, in order to generate mask information of a specific fusion of a real object and a captured video under a camera trajectory of the captured video, a mask information generation module is designed to specifically include:

[0052] According to the camera trajectory VideoCam of the video, the 3D_GS B of the video and the 3D_GS AB after the real object and the video are fused are rendered frame by frame, thereby obtaining two sets of rendered image sequences, namely, the picture sequence B and the image sequence AB;

[0053] The interpolation calculation is performed frame by frame for the image sequence B and the image sequence AB. The calculation process of the mask information of each frame is as follows:

[0054]

[0055] The generated Mask image group contains the front-to-back depth relationship of 3D_GS A and 3D_GS B in each frame, which is used for subsequent video rendering.

[0056] In a 3D Gaussian-based three-dimensional model and video fusion method according to another embodiment of the present invention, designing a video rendering module specifically includes:

[0057] The real object 3D_GS A1 after the spatial position transformation is rendered frame by frame according to the camera trajectory VideoCam of the shooting video to obtain the image sequence A1, and then the mask group Mask, the image sequence A1 and the shooting video are comprehensively rendered to output the final fused video.

[0058] Figure 2 It is a schematic diagram of the fusion effect in a specific application example. The left picture is the original video, the right picture is the effect after the model is fused, and the fused model is shown in the box. The three-dimensional model and video fusion method based on 3D Gaussian in the embodiment of the present invention realizes the fusion of 3D Gaussian model (three-dimensional model) and shot video through 3D Gaussian modeling; first use 3D Gaussian modeling to complete the three-dimensional modeling of the real object, and use 3D Gaussian modeling to restore the spatial structure of the shot video, and then the editor places the three-dimensional model in the visualization page at an appropriate position and size, and then the fusion rendering can be automatically realized without adjusting the camera position and parameters. Compared with the traditional method of fusing three-dimensional models and shot videos, the present invention only requires the editor to confirm the placement position and size of the 3D Gaussian model (three-dimensional model), and the fusion of the model and the shot video can be automatically realized, saving a lot of labor costs.

[0059] It should be understood that the exemplary embodiments described herein are illustrative rather than restrictive. Although one or more embodiments of the present invention have been described in conjunction with the accompanying drawings, it should be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A 3D model and video fusion method based on 3D Gaussian, It is characterized in that Including the space restoration process and the fusion rendering process, The space restoration process further includes: Design a 3D Gaussian modeling module to collect image data of the object to be embedded, and use the image data to complete the 3D Gaussian modeling of the object for subsequent fusion with the video; The video space restoration module is designed to extract frames according to the input video, and the camera position is estimated on the image data obtained by the extraction to obtain the camera movement trajectory of the video. The 3D Gaussian modeling uses the image data after the extraction to perform rapid modeling, thereby restoring the spatial structure of the video and the 3D Gaussian model of the video. The fusion rendering process further includes: Design an embedded position confirmation module to simultaneously load the 3D Gaussian model of the real object and the simple 3D Gaussian model visualization modeled from the captured video. The editor can adjust the specific position and size of the 3D Gaussian model of the real object in the visualization interface and export multiple new 3D Gaussian models. Design a mask information generation module, first perform an image rendering according to the camera trajectory of the video, render the 3D Gaussian model generated by the embedding position confirmation module and the simple 3D Gaussian model modeled from the video frame by frame according to the camera trajectory, obtain multiple sets of image sequences, calculate the mask information between the models according to the multiple sets of image sequences, and obtain a mask image group; The video rendering module is designed to calculate and render frame by frame according to the multiple image sequences generated by the mask information generation module and the calculated mask image group, and synthesize the final captured video of the embedded model.

2. The 3D model and video fusion method based on 3D Gaussian as claimed in claim 1, It is characterized in that The design of the 3D Gaussian modeling module specifically includes: The image set Y = y of N real objects collected in a shooting manner i (0 < i ≤ N) Through structure from motion matching, the corresponding camera parameters ObjectCam are obtained. The camera parameters include the focal length, distortion parameters, and internal and external parameter matrices of the position matrix of the camera, and an initial scene point cloud P is constructed, defined as follows ObjectCam, P = SFM (Y) 3D Gaussian modeling (GS_model) models the real object through ObjectCam, the initialized scene point cloud P and N images Y of the real object, and generates 3D_GS (3D Gaussian model) A. The 3D Gaussian model A will be used for subsequent video fusion. The modeling process is as follows 3D_GSA=GS_model(ObjectCam, P, Y).

3. The 3D model and video fusion method based on 3D Gaussian as claimed in claim 2, It is characterized in that The video space restoration module is designed to specifically include: The input video V is processed by frame extraction to obtain the video image set Y " , process the video image set Y in the same way as the real object image set Y " , get the camera trajectory VideoCam and scene point cloud P of the video v . In order to restore the spatial structure of the captured video V, the camera trajectory VideoCam and the scene point cloud P v and video image set Y v Modeling the 3D_GS B of the video, the modeling process is as follows: 3D_GSB=GS_model $%&'_)i*+' (VideoCam, P v ,Y v )。 4. The 3D model and video fusion method based on 3D Gaussian as claimed in claim 3, It is characterized in that The design embedding position confirmation module specifically includes: Load 3D_GS A and 3D_GS B into the visualization interface of the Gaussian model at the same time, fix the 3D_GS B of the video without spatial transformation, and let the editor move the 3D_GS A of the real object to any position for positioning; After confirming the specific fusion position of the real object and the captured video, two new 3D_GS models are exported in the visualization interface, namely 3D_GS AB after the fusion of the real object and the captured video, and 3D_GS A1 of the real object after the spatial position transformation.

5. The 3D model and video fusion method based on 3D Gaussian as claimed in claim 4, It is characterized in that The design mask information generation module specifically includes: According to the camera trajectory VideoCam of the video, the 3D_GS B of the video and the 3D_GS AB after the real object and the video are fused are rendered frame by frame, thereby obtaining two sets of rendered image sequences, namely, the picture sequence B and the image sequence AB; The interpolation calculation is performed frame by frame for the image sequence B and the image sequence AB. The calculation process of the mask information of each frame is as follows The generated Mask image group contains the front-to-back depth relationship of 3D_GS A and 3D_GS B in each frame, which is used for subsequent video rendering.

6. The 3D Gaussian-based three-dimensional model and video fusion method according to claim 5, It is characterized in that The design video rendering module specifically includes: The real object 3D_GS A1 after the spatial position transformation is rendered frame by frame according to the camera trajectory VideoCam of the shooting video to obtain the image sequence A1, and then the mask group Mask, the image sequence A1 and the shooting video are comprehensively rendered to output the final fused video.

Citation Information

Cited By

  • Personnel positioning method and system based on 3D Gaussian splash model and video fusion

    CN120318327A

  • Personnel positioning method and system based on 3D Gaussian splash model and video fusion

    CN120318327B