Fusion method of dynamic video and three-dimensional model based on image generation

Abstract

The invention discloses a fusion method of a dynamic video and a three-dimensional model based on image generation, and relates to the technical field of image generation. A NeRF three-dimensional geometric perception generation network is introduced to directly process three-dimensional geometric data, so that each frame of generated image contains geometric information of a scene, object deformation, distortion or position error is avoided, and the fusion efficiency is improved. The shape and the position of an object are always matched with an actual geometric structure in a scene through the three-dimensional perception capability of NeRF, dynamic understanding of the geometric structure in the generation process is achieved, meanwhile, geometric information of a time sequence is embedded, the dynamic position and a rotation matrix of the object are introduced into the time sequence, and the geometric information is obtained. The geometric state of the object in the scene can be effectively captured and updated, the light path is dynamically adjusted to enable the motion trail of the object to be completely consistent with the change in the scene, large-range motion and complex rotation are effectively dealt with, the geometric capture capability is enhanced, and the video generation precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and in particular to a method for fusing dynamic video and a three-dimensional model based on image generation. Background Art

[0002] With the continuous development of image generation technology, the fusion of dynamic video and three-dimensional models has gradually been applied to multiple fields. By seamlessly combining dynamic video content with three-dimensional virtual models, highly realistic visual effects and immersive experiences are created.

[0003] At present, the mainstream image generation models used in the fusion of dynamic video and 3D models, such as GAN (Generative Adversarial Network) and VAE (Variational Autoencoder), are mainly used to generate high-quality 2D images. However, these models lack the understanding of 3D geometric structure and mostly only process the pixel values ​​in the image, ignoring the 3D shape, position and motion trajectory of objects in the scene. As a result, when generating dynamic videos, the model is difficult to maintain the consistency of the geometric structure of the object, and often problems such as object deformation, distortion or position error occur.

[0004] Due to the lack of dynamic understanding of geometric structures, the continuity between generated video frames is difficult to guarantee. Traditional image generation models cannot effectively capture the geometric characteristics of three-dimensional objects that change over time. This problem is particularly evident in dynamic scenes. Objects in the video may experience discontinuous jumps or changes in shape between different frames, seriously affecting visual fluency.

[0005] To solve the above problems, some solutions have introduced technologies such as keyframe interpolation and optical flow to estimate the motion trajectory of objects and insert transitions between frames. However, these methods rely too much on motion prediction models and have difficulty processing complex three-dimensional geometric information. When objects rotate or move over a large range, the prediction accuracy will drop significantly, affecting the generation effect.

[0006] In addition, some solutions use texture mapping technology to fit the generated two-dimensional image onto the surface of the three-dimensional model. However, in dynamic scenes, texture mapping often suffers from stretching, tearing, or misalignment. Therefore, a method for fusing dynamic video generated by images with three-dimensional models is urgently needed to solve such problems. Summary of the invention

[0007] In view of the above existing problems, the present invention is proposed.

[0008] The present invention provides a method for fusing dynamic video and three-dimensional model based on image generation to solve the problem that traditional dynamic video generation method based on the fusion of image generation and three-dimensional model lacks in-depth understanding of three-dimensional dynamic geometric structure, and the generated results still have visual inconsistency, deformation and lack of real-time performance in complex dynamic scenes.

[0009] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0010] The present invention provides a method for fusing dynamic video and three-dimensional model based on image generation, which comprises:

[0011] Step S1, introduce the 3D geometry perception generation network,

[0012] NeRF (Neural Radiance Fields) is used to generate a network to process 3D geometric data in dynamic videos and generate geometrically perceived images.

[0013] Step S2, embedding geometric information of time series,

[0014] Based on the geometric perception image generated in step S1, the time series geometric change data is introduced to generate a time series geometric perception image.

[0015] Step S3: weighted fusion and geometric consistency optimization.

[0016] Based on the time series geometric perception image generated in step S2, a weighted fusion algorithm based on a three-dimensional model is used to fuse the image of each frame with the texture information of the three-dimensional model pixel by pixel.

[0017] Step S4, final output and real-time rendering,

[0018] After the geometric consistency fusion in step S3, the optimized image sequence is output and rendered in real time.

[0019] Furthermore, in step S1, the 3D geometric data includes 3D geometric coordinates, object shape and perspective changes, and NeRF generates an image matching the 3D model according to the 3D geometric coordinates, object shape and perspective changes in the scene, namely, a geometrically perceived image.

[0020] Furthermore, in step S1, NeRF generates a network and processes the three-dimensional geometric data to generate a geometrically perceived image. Specifically:

[0021] For each frame in the dynamic video, a geometric perception image is generated using the input of three-dimensional geometric data, which includes three-dimensional coordinates, perspective changes, and object shape information. The three-dimensional coordinate is defined as x∈R 3 , the viewing direction is d∈R 3, NeRF generates color c∈R according to the coordinates and viewing angle of the point in the scene 3 And the volume density σ∈R:

[0022] F θ (x, d) = (c, σ), where x represents the three-dimensional geometric coordinates of a sampling point in the scene, d represents the viewing direction from the camera to the sampling point, i.e., a vector, c represents the color value of the point at the viewing angle, and the output is a color vector of RGB three channels, and σ represents the volume density of the point, i.e., the opacity of the point in the scene;

[0023] NeRF samples the light along the light r(t) emitted from the camera to each pixel in the scene, and solves the weighted sum of the color and opacity of the light as it passes through the scene. The parameterized equation of the light is:

[0024] r(t)=o+td,t∈[t n ,t f ], where o represents the three-dimensional position of the camera's optical center, t represents the parameterized variable on the ray from the camera to the scene, and t n and t f They represent the interception range of the light at short distance and long distance respectively, d represents the direction of the light,

[0025] NeRF samples each ray segment by segment to generate a series of sampling points x i = r(t i ), and use the network F for each sampling point θ Calculate color c i and volume density σ i ,

[0026] The sampling points on each ray are weighted and accumulated, and the volume rendering formula is:

[0027] Among them, C(r) represents the final color value on the light ray r, that is, the pixel color in the output image, N represents the number of sampling points along the light ray, and T i represents the cumulative transmittance of the first i sampling points, defined as represents the transmittance of the light that starts from the camera and is not completely absorbed before point i, σ i Represents the sampling point x i The volume density, δ i Represents the distance t between adjacent sampling points i+1 -t i , that is, the step length of the light between sampling points, c i Represents the sampling point x i The color value of the

[0028] Furthermore, during the training process of step S1, NeRF optimizes the network parameter θ based on the mean square error (MSE) loss between the real image and the synthetic image:

[0029] Among them, L represents the total loss function, which measures the difference between the synthetic image and the real image. represents the light set, that is, all the light in the image, C G (r) represents the color value of light r in the real image, C(r) represents the color value corresponding to the light generated by the NeRF network,

[0030] Each ray r outputs its corresponding color C(r), thereby synthesizing the entire geometrically perceived image, where each pixel contains the geometric and lighting information of the scene.

[0031] Furthermore, in step S2, the method of introducing the geometric change data of the time series is:

[0032] Collect the object's motion trajectory, rotation angle and perspective change information, and dynamically embed the geometric changes in the time series.

[0033] Furthermore, in step S2, the geometric information of the time series is embedded as follows:

[0034] Based on the geometric perception image generated in step S1, the geometric change data of the object in the time series is introduced, including the object's motion trajectory, rotation angle and viewing angle change, and the three-dimensional position of the object at time t is set to x(t)∈R 3 , the rotation matrix of the object is R(t)∈SO(3), then the change of the position and direction of the object over time is expressed as:

[0035]

[0036] Among them, x(t) represents the three-dimensional position of the object at time t, x0 represents the initial position of the object, that is, the position at time t=0, v(t) represents the instantaneous velocity vector of the object, that is, the motion trajectory of the object at each time point, R(t) represents the rotation matrix of the object at time t, that is, the rotation state of the object, R0 represents the initial rotation matrix of the object, ω(t) represents the instantaneous angular velocity vector of the object, that is, the rotation change of the object,

[0037] The geometric perception image of each frame is constructed using the position and rotation data x(t) and R(t) of the object in the time series. Specifically, the geometric perception image I(t) at time t is generated based on the geometric information transformation of the previous frame I(t-Δt):

[0038] Among them, I(t) represents the geometric perception image at time t, It represents the geometric transformation function, which is responsible for transforming the previous frame image I(t-Δt) according to the displacement x(t) and rotation matrix R(t) of the object. I(t-Δt) represents the geometric perception image of the previous frame, and Δt represents the time interval between two frames.

[0039] Furthermore, in step S2, the geometric information embedding method of the time series also includes:

[0040] The light in the scene is recalculated based on the time series motion data of the object, and the path r(t) of the light is dynamically adjusted according to the displacement and rotation of the object:

[0041] r(t)=o(t)+t·R(t)·d(t), where r(t) represents the light path at time t, o(t) represents the dynamic position of the camera's optical center at time t, and d(t) represents the camera's viewing direction at time t, which is updated based on the object's motion and rotation.

[0042] After combining the object's motion trajectory and the change in viewing angle, each frame of geometric perception image I(t) at time t is calculated by the light integral generated by the NeRF network. The light color C(t) generated by NeRF depends on the geometric information of the time series, and the formula is:

[0043] Where C(t) represents the color corresponding to the light ray r(t) at time t, based on the position and rotation state of the objects in the scene, T i (t) represents the cumulative transmittance of the i-th sampling point, defined as Among them, σ j (t) and δ j (t) represent the density and distance of the sampling points, respectively, and change with time, σ i (t) represents the volume density of the object, which is updated over time, δ i (t) represents the distance between adjacent sampling points, defined as t i+1 (t)-t i (t), c i (t) represents the color value of the sampling point at time t. Combined with the geometric change data in the time series, a geometrically perceived image is generated in which each frame is geometrically consistent and continuous with the previous frame.

[0044] Furthermore, in step S3, a geometric consistency optimization mechanism is introduced, and real-time lighting simulation is performed to optimize image authenticity.

[0045] In step S3, the fusion process uses GPU acceleration and parallel computing technology.

[0046] Furthermore, in step S3, the weighted fusion and geometric consistency optimization method is:

[0047] Perform weighted fusion processing on each frame image I(t) and combine it with the texture information of the 3D model. Assuming that the texture information of the 3D model is T(x, y, z), the texture value of each point in the 3D space is given, and the mapping between each pixel in the image and the 3D space point is:

[0048] I(t,u,v)=α·I(t,u,v)+(1-α)·T(x(u,v),y(u,v),z(u,v)), where I(t,u,v) represents the color value of the image at the pixel coordinate (u,v) at time t, T(x(u,v),y(u,v),z(u,v)) represents the texture value corresponding to the coordinate (x(u,v),y(u,v),z(u,v)) in the three-dimensional model, which corresponds to the mapping of the pixel position (u,v) in the image in the three-dimensional space, α represents the fusion weight coefficient, which controls the weighted proportion of the original image and texture information, and its value range is 0≤α≤1.

[0049] A geometric consistency optimization mechanism is introduced to compare the geometric information of the same position in the previous and next frames for optimization. Suppose the same pixel position in the t-th frame and the t-1 frame is (u, v):

[0050] L geo =∑ (u,v) |I(t,u,v)-I(t-1,u,v)| 2 , where L geo represents the geometric consistency loss function, that is, the color difference of the same pixel at time t and t-1, I(t,u,v) represents the color value of the pixel (u,v) in the t-th frame, and I(t-1,u,v) represents the color value of the same pixel (u,v) in the t-1th frame.

[0051] Furthermore, in step S3, the weighted fusion and geometric consistency optimization method further includes:

[0052] Introduce lighting simulation to calculate the final lighting intensity of the pixel:

[0053] I final (t,u,v)=I(t,u,v)·(k a ·L a +k d ·(n·l)+k s ·(r·v) n ), where I final (t,u,v) represents the final light intensity at pixel (u,v) at time t, k a Indicates the ambient light coefficient, L a Indicates the ambient light intensity, kd represents the diffuse reflection coefficient, n represents the normal vector of the object surface, l represents the incident light direction vector, k s represents the specular reflection coefficient, r represents the direction vector of the reflected light, r=2(n·l)nl, v represents the direction vector of the camera viewing angle, and n represents the highlight index, which controls the highlight sharpness. The final light intensity of each pixel is calculated through real-time lighting simulation combined with ambient light, diffuse reflection and specular reflection, thereby improving the authenticity and visual effect of the image.

[0054] The beneficial effects of the present invention are:

[0055] The present invention introduces the NeRF three-dimensional geometric perception generation network to directly process three-dimensional geometric data, so that the image generated in each frame contains the geometric information of the scene, avoiding object deformation, distortion or position error. The three-dimensional perception capability of NeRF ensures that the shape and position of the object always match the actual geometric structure in the scene, thereby realizing dynamic understanding of the geometric structure during the generation process. At the same time, by embedding the geometric information of the time series, it can effectively capture the motion trajectory, rotation angle and viewing angle change of the object, ensure the geometric consistency between frames, and avoid frame skipping or sudden change of object morphology.

[0056] The present invention introduces the dynamic position and rotation matrix of objects in the time series, which can effectively capture and update the geometric state of objects in the scene, dynamically adjust the light path to make the movement trajectory of the object completely consistent with the changes in the scene, effectively cope with large-scale movements and complex rotations, enhance the geometric capture capability, and improve the accuracy of video generation.

[0057] The present invention, based on a weighted fusion algorithm of a three-dimensional model, fuses each frame of image with the texture information in the three-dimensional model pixel by pixel, calculates the mapping position of the three-dimensional model pixel by pixel, and makes the texture information of each frame closely integrated with the surface of the three-dimensional geometric model, thereby reducing the misalignment and stretching of the texture in dynamic scenes.

[0058] The present invention introduces real-time lighting simulation and incorporates ambient light, diffuse reflection and specular reflection into the rendering process, so that the generated image not only has consistent geometric structure but also has more realistic lighting effects, so that objects in the video scene can show natural light and shadow changes under different lighting conditions, thereby increasing the visual realism and stereoscopic perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0060] Figure 1 The figure is a flow chart of the method for fusing dynamic video and three-dimensional model based on image generation according to the present invention. DETAILED DESCRIPTION

[0061] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0062] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0063] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0064] Example 1, reference Figure 1 This embodiment provides a method for fusing a dynamic video generated based on an image with a three-dimensional model, comprising the following steps:

[0065] Step S1, introduce the 3D geometry perception generation network,

[0066] NeRF (Neural Radiance Fields) is used to generate a network to process 3D geometric data in dynamic videos and generate geometrically perceived images.

[0067] In step S1, the 3D geometric data includes 3D geometric coordinates, object shape and view angle change. NeRF generates an image matching the 3D model according to the 3D geometric coordinates, object shape and view angle change in the scene, i.e., a geometrically perceived image.

[0068] In step S1, NeRF generates a network and processes 3D geometric data to generate a geometrically perceived image. Specifically:

[0069] For each frame in the dynamic video, a geometric perception image is generated using the input of three-dimensional geometric data, which includes three-dimensional coordinates, perspective changes, and object shape information. The three-dimensional coordinate is defined as x∈R 3 , the viewing direction is d∈R 3 , NeRF generates color c∈R according to the coordinates and viewing angle of the point in the scene 3 And the volume density σ∈R:

[0070] F θ(x, d) = (c, σ), where x represents the three-dimensional geometric coordinates of a sampling point in the scene, d represents the viewing direction from the camera to the sampling point, i.e., a vector, c represents the color value of the point at the viewing angle, and the output is a color vector of RGB three channels, and σ represents the volume density of the point, i.e., the opacity of the point in the scene;

[0071] NeRF samples the light along the light r(t) emitted from the camera to each pixel in the scene, and solves the weighted sum of the color and opacity of the light as it passes through the scene. The parameterized equation of the light is:

[0072] r(t)=o+td,t∈[t n ,t f ], where o represents the three-dimensional position of the camera's optical center, t represents the parameterized variable on the ray from the camera to the scene, and t n and t f They represent the interception range of the light at short distance and long distance respectively, d represents the direction of the light,

[0073] NeRF samples each ray segment by segment to generate a series of sampling points x i = r(t i ), and use the network F for each sampling point θ Calculate color c i and volume density σ i ,

[0074] The sampling points on each ray are weighted and accumulated, and the volume rendering formula is:

[0075] Among them, C(r) represents the final color value on the light ray r, that is, the pixel color in the output image, N represents the number of sampling points along the light ray, and T i represents the cumulative transmittance of the first i sampling points, defined as represents the transmittance of the light that starts from the camera and is not completely absorbed before point i, σ i Represents the sampling point x i The volume density, δ i Represents the distance t between adjacent sampling points i+1 -t i , that is, the step length of the light between sampling points, c i Represents the sampling point x i The color value of

[0076] During the training process of step S1, NeRF optimizes the network parameters θ based on the mean square error MSE loss between real images and synthetic images:

[0077] Among them, L represents the total loss function, which measures the difference between the synthetic image and the real image. represents the light set, that is, all the light in the image, C G (r) represents the color value of light r in the real image, C(r) represents the color value corresponding to the light generated by the NeRF network,

[0078] Each ray r outputs its corresponding color C(r), thus synthesizing the entire geometric perception image. Each pixel in the image contains the geometric information and lighting information of the scene.

[0079] Specifically, the NeRF 3D geometry perception generation network is able to process relatively complex 3D geometric data, including the 3D coordinates, shapes, and perspective changes of objects, so that the geometric information contained in each generated frame of the image effectively reflects the actual object form in the scene, and combines density and color to make each pixel correspond one-to-one to a point in 3D space.

[0080] Step S2, embedding geometric information of time series,

[0081] Based on the geometric perception image generated in step S1, the time series geometric change data is introduced to generate a time series geometric perception image.

[0082] In step S2, the method of introducing the geometric change data of the time series is:

[0083] Collect the object's motion trajectory, rotation angle and perspective change information, and dynamically embed the geometric changes in the time series.

[0084] In step S2, the geometric information of the time series is embedded as follows:

[0085] Based on the geometric perception image generated in step S1, the geometric change data of the object in the time series is introduced, including the object's motion trajectory, rotation angle and viewing angle change, and the three-dimensional position of the object at time t is set to x(t)∈R 3 , the rotation matrix of the object is R(t)∈SO(3), then the change of the position and direction of the object over time is expressed as:

[0086]

[0087] Among them, x(t) represents the three-dimensional position of the object at time t, x0 represents the initial position of the object, that is, the position at time t=0, v(t) represents the instantaneous velocity vector of the object, that is, the motion trajectory of the object at each time point, R(t) represents the rotation matrix of the object at time t, that is, the rotation state of the object, R0 represents the initial rotation matrix of the object, ω(t) represents the instantaneous angular velocity vector of the object, that is, the rotation change of the object,

[0088] The geometric perception image of each frame is constructed using the position and rotation data x(t) and R(t) of the object in the time series. Specifically, the geometric perception image I(t) at time t is generated based on the geometric information transformation of the previous frame I(t-Δt):

[0089] Among them, I(t) represents the geometric perception image at time t, represents the geometric transformation function, which is responsible for transforming the previous frame image I(t-Δt) according to the displacement x(t) and rotation matrix R(t) of the object. I(t-Δt) represents the geometric perception image of the previous frame, and Δt represents the time interval between two frames.

[0090] In step S2, the geometric information embedding method of the time series also includes:

[0091] The light in the scene is recalculated based on the time series motion data of the object, and the path r(t) of the light is dynamically adjusted according to the displacement and rotation of the object:

[0092] r(t)=o(t)+t·R(t)·d(t), where r(t) represents the light path at time t, o(t) represents the dynamic position of the camera's optical center at time t, and d(t) represents the camera's viewing direction at time t, which is updated based on the object's motion and rotation.

[0093] After combining the object's motion trajectory and the change in viewing angle, each frame of geometric perception image I(t) at time t is calculated by the light integral generated by the NeRF network. The light color C(t) generated by NeRF depends on the geometric information of the time series, and the formula is:

[0094] Where C(t) represents the color corresponding to the light ray r(t) at time t, based on the position and rotation state of the objects in the scene, T i (t) represents the cumulative transmittance of the i-th sampling point, defined as Among them, σ j (t) and δ j (t) represent the density and distance of the sampling points, respectively, and change with time, σ i (t) represents the volume density of the object, which is updated over time, δ i (t) represents the distance between adjacent sampling points, defined as t i+1 (t)-t i (t), c i (t) represents the color value of the sampling point at time t. Combined with the geometric change data in the time series, a geometric perception image is generated in which each frame is geometrically consistent and continuous with the previous frame.

[0095] Specifically, embedding time series geometric information greatly improves the geometric continuity and consistency between frames, introduces the object's motion trajectory, rotation angle and viewing angle changes, dynamically adjusts the geometric information in each frame, ensures the continuity and smoothness of the image in the time series, and the generated video will not jump or distort in the geometric structure, so that objects in dynamic scenes maintain morphological consistency at different time points.

[0096] Step S3: weighted fusion and geometric consistency optimization.

[0097] Based on the time series geometric perception image generated in step S2, a weighted fusion algorithm based on a three-dimensional model is used to fuse the image of each frame with the texture information of the three-dimensional model pixel by pixel.

[0098] In step S3, a geometric consistency optimization mechanism is introduced, and real-time lighting simulation is performed to optimize image authenticity.

[0099] In step S3, the fusion process uses GPU acceleration and parallel computing technology.

[0100] In step S3, the weighted fusion and geometric consistency optimization method is:

[0101] Perform weighted fusion processing on each frame image I(t) and combine it with the texture information of the 3D model. Assuming that the texture information of the 3D model is T(x, y, z), the texture value of each point in the 3D space is given, and the mapping between each pixel in the image and the 3D space point is:

[0102] I(t,u,v)=α·I(t,u,v)+(1-α)·T(x(u,v),y(u,v),z(u,v)), where I(t,u,v) represents the color value of the image at the pixel coordinate (u,v) at time t, T(x(u,v),y(u,v),z(u,v)) represents the texture value corresponding to the coordinate (x(u,v),y(u,v),z(u,v)) in the three-dimensional model, which corresponds to the mapping of the pixel position (u,v) in the image in the three-dimensional space, α represents the fusion weight coefficient, which controls the weighted proportion of the original image and texture information, and its value range is 0≤α≤1.

[0103] A geometric consistency optimization mechanism is introduced to compare the geometric information of the same position in the previous and next frames for optimization. Suppose the same pixel position in the t-th frame and the t-1 frame is (u, v):

[0104] L geo =∑ (u,v) |I(t,u,v)-I(t-1,u,v)| 2 , where L georepresents the geometric consistency loss function, that is, the color difference of the same pixel at time t and t-1, l(t,u,v) represents the color value of the pixel (u,v) in the t-th frame, I(t-1,u,v) represents the color value of the same pixel (u,v) in the t-1th frame,

[0105] Specifically, weighted fusion of the generated image and the texture information of the three-dimensional model is performed pixel by pixel, which can effectively combine the generated geometrically perceived image with the existing three-dimensional texture data, so that the texture information of the generated image in three-dimensional space matches the actual scene. The geometric consistency optimization mechanism compares the geometric information at the same position in adjacent frames, maximizes the geometric consistency between frames, and reduces the probability of geometric distortion or frame skipping.

[0106] In step S3, the weighted fusion and geometric consistency optimization method further includes:

[0107] Introduce lighting simulation to calculate the final lighting intensity of the pixel:

[0108] I final (t,u,v)=I(t,u,v)·(k a ·L a +k d ·(n·l)+k s ·(r·v) n ), where I final (t,u,v) represents the final light intensity at pixel (u,v) at time t, k a Indicates the ambient light coefficient, L a Indicates the ambient light intensity, k d represents the diffuse reflection coefficient, n represents the normal vector of the object surface, l represents the incident light direction vector, k s represents the mirror reflection coefficient, r represents the direction vector of the reflected light, r=2(n·l)nl, v represents the direction vector of the camera viewing angle, and n represents the highlight index, which controls the highlight sharpness. The final light intensity of each pixel is calculated through real-time lighting simulation combined with ambient light, diffuse reflection and mirror reflection, thereby improving the authenticity and visual effect of the image.

[0109] Specifically, ambient light, diffuse reflection and specular reflection are introduced for simulation, which greatly improves the lighting effect of the generated image and enhances the realism of the final rendered image. By combining the changes in the surface normal vector of the object and the reflected light, a more realistic lighting effect is generated, which adds more visual layers to the image and makes the changes in light and shadow in dynamic scenes more natural and realistic.

[0110] Step S4, final output and real-time rendering,

[0111] After the geometric consistency fusion in step S3, the optimized image sequence is output and rendered in real time.

[0112] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for fusing dynamic video and three-dimensional model based on image generation, characterized in that: include, Step S1, introduce a 3D geometry perception generation network, use NeRF to generate a network, process the 3D geometry data in the dynamic video, and generate a geometry perception image. Step S2, embedding the geometric information of the time series, based on the geometric perception image generated in step S1, introducing the geometric change data of the time series, and generating a time series geometric perception image, Step S3, weighted fusion and geometric consistency optimization, based on the time series geometric perception image generated in step S2, a weighted fusion algorithm based on a three-dimensional model is used to fuse the image of each frame with the texture information of the three-dimensional model pixel by pixel. Step S4, final output and real-time rendering, after the geometric consistency fusion in step S3, output the optimized image sequence and perform real-time rendering.

2. The method for fusing dynamic video and three-dimensional model based on image generation according to claim 1, characterized in that: In step S1, the 3D geometric data includes 3D geometric coordinates, object shape and viewing angle changes. NeRF generates an image matching the 3D model, namely, a geometrically perceived image, based on the 3D geometric coordinates, object shape and viewing angle changes in the scene.

3. The method for fusing dynamic video and three-dimensional model based on image generation according to claim 2, characterized in that: In step S1, NeRF generates a network and processes 3D geometric data to generate a geometrically perceived image. Specifically: For each frame in the dynamic video, a geometric perception image is generated using the input of three-dimensional geometric data, which includes three-dimensional coordinates, perspective changes, and object shape information. The three-dimensional coordinate is defined as x∈R 3 , the viewing direction is d∈R 3 , NeRF generates color c∈R according to the coordinates and viewing angle of the point in the scene 3 And the volume density σ∈R: F θ (x, d) = (c, σ), where x represents the three-dimensional geometric coordinates of a sampling point in the scene, d represents the viewing direction from the camera to the sampling point, i.e., a vector, c represents the color value of the point at the viewing angle, and the output is a color vector of RGB three channels, and σ represents the volume density of the point, i.e., the opacity of the point in the scene; NeRF samples the light along the light r(t) emitted from the camera to each pixel in the scene, and solves the weighted sum of the color and opacity of the light as it passes through the scene. The parameterized equation of the light is: r(t)=o+td,t∈[t n ,t f ], where o represents the three-dimensional position of the camera's optical center, t represents the parameterized variable on the ray from the camera to the scene, and t n and t f They represent the interception range of the light at short distance and long distance respectively, d represents the direction of the light, NeRF samples each ray segment by segment to generate a series of sampling points x i = r(t i ), and use the network F for each sampling point θ Calculate color c i and volume density σ i , The sampling points on each ray are weighted and accumulated, and the volume rendering formula is: Among them, C(r) represents the final color value on the light ray r, that is, the pixel color in the output image, N represents the number of sampling points along the light ray, and T i represents the cumulative transmittance of the first i sampling points, defined as represents the transmittance of the light that starts from the camera and is not completely absorbed before point i, σ i Represents the sampling point x i The volume density, δ i Represents the distance t between adjacent sampling points i+1 -t i , that is, the step length of the light between sampling points, c i Represents the sampling point x i The color value of the 4. The method for fusing dynamic video and three-dimensional model based on image generation according to claim 3, characterized in that: During the training process of step S1, NeRF optimizes the network parameters θ based on the mean square error MSE loss between real images and synthetic images: Among them, L represents the total loss function, which measures the difference between the synthetic image and the real image. represents the light set, that is, all the light in the image, C G (r) represents the color value of light r in the real image, C(r) represents the color value corresponding to the light generated by the NeRF network, Each ray r outputs its corresponding color C(r).

5. The method for fusing dynamic video and three-dimensional model based on image generation according to claim 4, characterized in that: In step S2, the method of introducing the geometric change data of the time series is: Collect the object's motion trajectory, rotation angle and perspective change information, and dynamically embed the geometric changes in the time series.

6. The method for fusing dynamic video and three-dimensional model based on image generation according to claim 5, characterized in that: In step S2, the geometric information of the time series is embedded as follows: Based on the geometric perception image generated in step S1, the geometric change data of the object in the time series is introduced, including the object's motion trajectory, rotation angle and viewing angle change, and the three-dimensional position of the object at time t is set to x(t)∈R 3 , the rotation matrix of the object is R(t)∈SO(3), then the change of the position and direction of the object over time is expressed as: Among them, x(t) represents the three-dimensional position of the object at time t, x0 represents the initial position of the object, that is, the position at time t=0, v(t) represents the instantaneous velocity vector of the object, that is, the motion trajectory of the object at each time point, R(t) represents the rotation matrix of the object at time t, that is, the rotation state of the object, R0 represents the initial rotation matrix of the object, ω(t) represents the instantaneous angular velocity vector of the object, that is, the rotation change of the object, The geometric perception image of each frame is constructed using the position and rotation data x(t) and R(t) of the object in the time series. Specifically, the geometric perception image I(t) at time t is generated based on the geometric information transformation of the previous frame I(t-Δt): Among them, I(t) represents the geometric perception image at time t, It represents the geometric transformation function, which is responsible for transforming the previous frame image I(t-Δt) according to the displacement x(t) and rotation matrix R(t) of the object. I(t-Δt) represents the geometric perception image of the previous frame, and Δt represents the time interval between two frames.

7. The method for fusing dynamic video and three-dimensional model based on image generation according to claim 6, characterized in that: In step S2, the geometric information embedding method of the time series also includes: The light in the scene is recalculated based on the time series motion data of the object, and the path r(t) of the light is dynamically adjusted according to the displacement and rotation of the object: r(t)=o(t)+t·R(t)·d(t), where r(t) represents the light path at time t, o(t) represents the dynamic position of the camera's optical center at time t, and d(t) represents the camera's viewing direction at time t, which is updated based on the object's motion and rotation. After combining the object's motion trajectory and the change in viewing angle, each frame of geometric perception image I(t) at time t is calculated by the light integral generated by the NeRF network. The light color C(t) generated by NeRF depends on the geometric information of the time series, and the formula is: Where C(t) represents the color corresponding to the light ray r(t) at time t, based on the position and rotation state of the objects in the scene, T i (t) represents the cumulative transmittance of the i-th sampling point, defined as Among them, σ j (t) and δ j (t) represent the density and distance of the sampling points, respectively, and change with time, σ i (t) represents the volume density of the object, which is updated over time, δ i (t) represents the distance between adjacent sampling points, defined as t i+1 (t)-t i (t), c i (t) represents the color value of the sampling point at time t.

8. The method for fusing dynamic video and three-dimensional model based on image generation according to claim 7, characterized in that: In step S3, a geometric consistency optimization mechanism is introduced and real-time lighting simulation is performed. In step S3, the fusion process uses GPU acceleration and parallel computing technology.

9. The method for fusing dynamic video and three-dimensional model based on image generation according to claim 8, characterized in that: In step S3, the weighted fusion and geometric consistency optimization method is: Perform weighted fusion processing on each frame image I(t) and combine it with the texture information of the 3D model. Assuming that the texture information of the 3D model is T(x, y, z), the texture value of each point in the 3D space is given, and the mapping between each pixel in the image and the 3D space point is: I(t,u,v)=α·I(t,u,v)+(1-α)·T(x(u,v),y(u,v),z(u,v)), where I(t,u,v) represents the color value of the image at the pixel coordinate (u,v) at time t, T(x(u,v),y(u,v),z(u,v)) represents the texture value corresponding to the coordinate (x(u,v),y(u,v),z(u,v)) in the three-dimensional model, which corresponds to the mapping of the pixel position (u,v) in the image in the three-dimensional space, α represents the fusion weight coefficient, which controls the weighted proportion of the original image and texture information, and its value range is 0≤α≤1. A geometric consistency optimization mechanism is introduced to compare the geometric information of the same position in the previous and next frames for optimization. Suppose the same pixel position in the t-th frame and the t-1 frame is (u, v): L geo =∑ (u,v) |I(t,u,v)-I(t-1,u,v)| 2 , where L geo represents the geometric consistency loss function, that is, the color difference of the same pixel at time t and t-1, I(t,u,v) represents the color value of the pixel (u,v) in the t-th frame, and I(t-1,u,v) represents the color value of the same pixel (u,v) in the t-1th frame.

10. The method for fusing dynamic video and three-dimensional model based on image generation according to claim 9, characterized in that: In step S3, the weighted fusion and geometric consistency optimization method further includes: Introduce lighting simulation to calculate the final lighting intensity of the pixel: I final (t,u,v)=I(t,u,v)·(k a ·L a +k d ·(n·l)+k s ·(r·v) n ), where I final (t,u,v) represents the final light intensity at pixel (u,v) at time t, k a Indicates the ambient light coefficient, L a Indicates the ambient light intensity, k d represents the diffuse reflection coefficient, n represents the normal vector of the object surface, l represents the direction vector of the incident light, k s represents the mirror reflection coefficient, r represents the direction vector of the reflected light, r=2(n·l)nl, v represents the direction vector of the camera viewing angle, and n represents the highlight index, which controls the highlight sharpness.

Images