Video editing method, device, equipment and storage medium

Abstract

The present application discloses a video editing method, device, equipment and storage medium, which relates to the field of computer vision. The method includes: obtaining a surround video of an inserted target, wherein the surround video includes video frames of the inserted target at different viewing angles; inputting the surround video into a preset three-dimensional reconstruction model to obtain a target three-dimensional model, wherein the preset three-dimensional reconstruction model is obtained by training each video frame obtained by segmenting the surround video; obtaining video reconstruction information based on the optical flow information of the video to be inserted in combination with a preset optimization algorithm, wherein the video reconstruction information includes camera internal and external parameter information and dense point cloud information; implanting the target three-dimensional model into a user-selected area in the video to be inserted based on the camera internal and external parameter information and the dense point cloud information to obtain an edited video. Since the user in the present application only needs to provide a surround video of the inserted target, the video editing process from three-dimensional model production to video insertion can be realized, which greatly improves the video editing speed compared to the existing method.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a video editing method, apparatus, device and storage medium. Background Art

[0002] With the development of video generation technology, video editing methods that insert three-dimensional models into existing videos have been applied in film and television production, advertising production, virtual reality and other fields. Users can perform three-dimensional reconstruction of the insertion target and then insert the generated three-dimensional model into the video to be inserted to achieve a real fusion effect.

[0003] However, traditional methods of object 3D reconstruction often rely on professional scanning equipment and special shooting environments. The reconstruction cost is high and time-consuming, and the implantation of 3D models into existing videos requires the use of multiple post-production software. Therefore, the video editing process of generating 3D models for real objects and implanting 3D models into existing videos is cumbersome and inefficient, and difficult for general users.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a video editing method, which aims to solve the technical problem that the existing video editing method of inserting a three-dimensional model into an existing video is cumbersome and inefficient, and is relatively difficult for general users.

[0006] To achieve the above objectives, the present application proposes a video editing method, which includes:

[0007] Acquire a surround video of the inserted target, wherein the surround video includes video frames of the inserted target at different viewing angles;

[0008] Inputting the surround video into a preset 3D reconstruction model to obtain a target 3D model, wherein the preset 3D reconstruction model is trained by each video frame obtained by segmenting the surround video;

[0009] Obtaining video reconstruction information based on the optical flow information of the video to be inserted and a preset optimization algorithm, wherein the video reconstruction information includes camera internal and external parameter information and dense point cloud information;

[0010] The target three-dimensional model is implanted into a user-selected area in the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain an edited video.

[0011] In one embodiment, the preset 3D reconstruction model includes a geometric coding perceptron and a texture coding perceptron; the training step of the preset 3D reconstruction model includes:

[0012] Initialize a tetrahedral mesh, input the three-dimensional coordinates of the tetrahedral mesh into the geometric encoding perceptron, obtain corresponding geometric information, and convert the target mesh according to the geometric information;

[0013] Inputting the three-dimensional coordinates of the target grid into the texture encoding perceptron to obtain texture information;

[0014] Inputting the environment lighting map, the camera pose information of the surround video, the target mesh and the texture information into a preset differentiable renderer to obtain a current rendering image;

[0015] Perform loss calculation according to the current rendering image and the foreground image of the inserted target at different viewing angles to obtain a loss gradient, and optimize the geometry encoding perceptron, the texture encoding perceptron and the ambient lighting map through the loss gradient based on a gradient feedback mechanism;

[0016] Based on the optimization goal of minimizing empirical risk, a preset three-dimensional reconstruction model is obtained.

[0017] In one embodiment, before the step of initializing the tetrahedral mesh, inputting the three-dimensional coordinates of the tetrahedral mesh into the geometric encoding perceptron, obtaining corresponding geometric information, and converting the target mesh according to the geometric information, the following steps are included:

[0018] When receiving a click instruction from the user, generating a mask area corresponding to the insertion target in the current video frame of the surround video according to the click instruction;

[0019] Segmentation is performed in other video frames obtained by segmenting the surround video according to the mask area to obtain foreground images of the inserted target at different viewing angles.

[0020] In one embodiment, the step of obtaining video reconstruction information based on the optical flow information of the video to be inserted in combination with a preset optimization algorithm includes:

[0021] Inputting a plurality of video frames corresponding to the video to be inserted into the optical flow prediction model to determine the optical flow displacement value of each of the video frames;

[0022] When the optical flow displacement value is greater than a preset threshold, adding the video frame corresponding to the optical flow displacement value to a key frame set;

[0023] A preset optimization algorithm is used in the key frame set to determine the video reconstruction information to obtain camera internal and external parameter information and dense point cloud information.

[0024] In one embodiment, the step of determining video reconstruction information using a preset optimization algorithm for each video frame in the key frame set to obtain camera internal and external parameter information and dense point cloud information includes:

[0025] Dividing the key frame set into an online optimization set and an offline optimization set;

[0026] In the online optimization set, a preset optimization algorithm is used to determine the relative position and depth map of the camera and the camera intrinsic parameters corresponding to each video frame;

[0027] In the offline optimization set, a preset optimization algorithm is used to update the camera relative pose, the depth map and the camera intrinsic parameters to obtain camera internal and external parameter information and dense point cloud information.

[0028] In one embodiment, the step of implanting the target three-dimensional model into the user-selected area in the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain the edited video includes:

[0029] Upon receiving a selection instruction from a user, generating a user-selected area in the video to be inserted;

[0030] Back-projecting the pixel coordinates of the area selected by the user into the camera coordinate system corresponding to the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain the three-dimensional point cloud coordinates of the area selected by the user;

[0031] Determine the three-dimensional plane normal vector corresponding to the video to be inserted, and pre-process the target three-dimensional model in combination with the three-dimensional point cloud coordinates of the area selected by the user;

[0032] Different viewing angle pictures corresponding to the preprocessed target three-dimensional model are rendered according to the internal and external parameter information of the video, and the edited video is synthesized according to each of the viewing angle pictures and a number of video frames corresponding to the video to be inserted.

[0033] In one embodiment, the step of determining the three-dimensional plane normal vector corresponding to the video to be inserted, and preprocessing the target three-dimensional model in combination with the three-dimensional point cloud coordinates of the user-selected area includes:

[0034] Based on a random sampling consensus algorithm, a three-dimensional plane normal vector is determined according to a camera coordinate system corresponding to the video to be inserted;

[0035] The target three-dimensional model is preprocessed based on the three-dimensional plane normal vector and combined with the three-dimensional point cloud coordinates of the user-selected area.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a video editing device, which includes: a model generation module, a video reconstruction module and a model implantation module;

[0037] The model generation module is used to obtain a surround video of the inserted target, wherein the surround video includes video frames of the inserted target at different viewing angles;

[0038] The model generation module is further used to input the surround video into a preset 3D reconstruction model to obtain a target 3D model, wherein the preset 3D reconstruction model is trained by each video frame obtained by segmenting the surround video;

[0039] The video reconstruction module is used to obtain video reconstruction information according to the optical flow information of the video to be inserted in combination with a preset optimization algorithm, wherein the video reconstruction information includes camera internal and external parameter information and dense point cloud information;

[0040] The model implantation module is used to implant the target three-dimensional model into the user-selected area in the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain the edited video.

[0041] In addition, to achieve the above objectives, the present application also proposes a video editing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video editing method described above.

[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the video editing method described above are implemented.

[0043] The present application provides a video editing method, which includes obtaining a surround video of an inserted target, wherein the surround video includes video frames of the inserted target at different viewing angles; inputting the surround video into a preset 3D reconstruction model to obtain a target 3D model, wherein the preset 3D reconstruction model is obtained by training each video frame obtained by segmenting the surround video; obtaining video reconstruction information based on the optical flow information of the video to be inserted in combination with a preset optimization algorithm, wherein the video reconstruction information includes camera internal and external parameter information and dense point cloud information; and implanting the target 3D model into a user-selected area in the video to be inserted based on the camera internal and external parameter information and the dense point cloud information to obtain an edited video. Since the user in the present application only needs to provide a surround video of the inserted target, the target 3D model can be obtained from the preset 3D reconstruction model, which simplifies the traditional 3D reconstruction process and can reduce the cost of 3D reconstruction. Furthermore, the insertion of the target 3D model is realized in the user-selected area of ​​the video to be inserted based on the video reconstruction information, thereby realizing a video editing process that integrates 3D model production and video insertion, which greatly improves the speed of video editing compared to the existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the video editing method of the present application;

[0047] Figure 2 A flowchart of the second embodiment of the video editing method of the present application is provided;

[0048] Figure 3 A flowchart of the third embodiment of the video editing method of the present application is provided;

[0049] Figure 4 A schematic diagram of the module structure provided in Example 1 of the video editing device of the present application;

[0050] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the video editing method in the embodiment of the present application.

[0051] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0053] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0054] The main solution of the embodiment of the present application is: obtaining a surround video of the inserted target, the surround video containing video frames of the inserted target at different perspectives; inputting the surround video into a preset 3D reconstruction model to obtain a target 3D model, the preset 3D reconstruction model is trained by each video frame obtained by segmenting the surround video; obtaining video reconstruction information based on the optical flow information of the video to be inserted in combination with a preset optimization algorithm, the video reconstruction information including camera internal and external parameter information and dense point cloud information; rendering the target 3D model to the user-selected area in the video to be inserted based on the camera internal and external parameter information and the dense point cloud information to obtain an edited video.

[0055] The video editing process of generating 3D models for real objects and inserting them into existing videos is difficult and costly for ordinary people. Reconstructing the precise geometry and surface texture of real objects often requires professional 3D scanning equipment and special shooting environments. Inserting a 3D model into an existing video usually requires a long time for professional film and television post-production engineers, such as first using Boujou to restore the sparse point cloud and camera internal and external parameters of the video, and then manually aligning the object placement plane with the target plane in the video in 3D rendering software such as 3ds Max and Blender, and using the camera parameters restored by Boujou to render 3D pictures from different perspectives frame by frame. In addition to video post-production software, current AR augmented reality technology SDKs such as Android's ARcore and Google's ARKit need to first use the phone's precise camera internal parameters and auxiliary sensors such as IMU and radar to complete the camera pose recovery and scene geometry reconstruction before rendering the 3D object into the real-time video stream. The overall process is cumbersome and inefficient.

[0056] In this application, the user only needs to provide a surround video of the insertion target, and the target 3D model can be obtained from the preset 3D reconstruction model, which simplifies the traditional 3D reconstruction process and can reduce the 3D reconstruction cost. Then, the target 3D model can be inserted into the user-selected area of ​​the video to be inserted according to the video reconstruction information, thereby realizing an integrated video editing process from 3D model production to video insertion, which greatly improves the speed of video editing compared to the existing method.

[0057] It should be noted that the execution subject of this embodiment can be a computing service device with interactive display, data processing, network communication and program running, such as a tablet computer, a personal computer, a mobile phone, etc., or other electronic devices that can achieve the same or similar functions. This embodiment is not limited to this. The following uses a video editing device (hereinafter referred to as "editing device") as an example to illustrate this embodiment and the following embodiments.

[0058] Based on this, the embodiment of the present application provides a video editing method, referring to Figure 1 , Figure 1 This is a flow chart of the video editing method embodiment 1 of the present application. The video editing method comprises steps S10 to S40:

[0059] Step S10: Acquire a surround video of the inserted target, wherein the surround video includes video frames of the inserted target at different viewing angles.

[0060] It should be noted that the inserted target is the target object to be reconstructed into a three-dimensional model. The user can first shoot a video around the inserted target so that the captured surround video includes different perspectives of the inserted target, and then upload the surround video to the editing device.

[0061] Step S20: inputting the surround video into a preset 3D reconstruction model to obtain a target 3D model, wherein the preset 3D reconstruction model is trained by each video frame obtained by segmenting the surround video.

[0062] It should be understood that the preset 3D reconstruction model can reconstruct the 3D mesh model and surface material of the inserted target through the surround video shot by the user. The preset 3D model can be a de-rendering model based on a neural network, which can include a geometry encoding perceptron for extracting the 3D mesh model corresponding to the inserted target, and a texture encoding perceptron for obtaining the surface material of the inserted target.

[0063] It should also be noted that the video frames containing the inserted target at different viewing angles can be used as supervision data for training the preset 3D reconstruction model. The model training process is specifically described below. The model training process includes steps A01 to A05:

[0064] Step A01: Initialize a tetrahedral mesh, input the three-dimensional coordinates of the tetrahedral mesh into the geometric encoding perceptron, obtain corresponding geometric information, and convert the geometric information to obtain a target mesh.

[0065] It should be noted that the Signed Distance Function (SDF) is a mathematical representation method for representing the shape of a geometric object. It defines a function that returns a signed distance value from the surface of a body for each point in space. This value can be positive, negative or zero, and is used to represent the positional relationship of the points. The geometric encoding perceptron can be a multi-layer perceptron (MLP). In the geometric encoding perceptron, the MLP network structure can be used to learn and approximate the SDF, thereby realizing the representation and generation of the geometric shape of a three-dimensional object.

[0066] It should be understood that the geometric information may be the SDF value corresponding to each vertex of the tetrahedral mesh, and the target mesh is a textureless mesh obtained by converting the SDF.

[0067] In the specific implementation, we can first initialize a tetrahedral mesh with a fixed resolution, then input the vertex coordinates of the tetrahedral mesh into the geometric encoding perceptron, convert them into corresponding SDF values, and then use Marching tetrahedrons (an isosurface extraction method that can extract a mesh model based on the SDF values ​​of coordinates in space) to obtain a texture-free mesh.

[0068] Step A02: Input the three-dimensional coordinates of the target grid into the texture coding perceptron to obtain texture information.

[0069] It should be noted that the texture coding perceptron can use the MLP grid structure to encode all material parameters, learn the automatic texture parameterization process of the grid surface, and obtain the pixel value corresponding to each grid vertex.

[0070] In the specific implementation, the texture encoding perceptron first takes the mesh vertices of the mesh as input, outputs the basic color, reflection coefficient and normal coefficient, and determines the pixel values ​​corresponding to each mesh vertex of the target mesh; then, it converts the color map, reflection map and normal map corresponding to the mesh through UV mapping to obtain the texture information.

[0071] Step A03: Input the environment lighting map, the camera pose information of the surround video, the target mesh and the texture information into a preset differentiable renderer to obtain a current rendering image.

[0072] It should be noted that, while initializing the tetrahedral mesh, a fixed-size ambient lighting (High Dynamic Range, HDR) map may also be initialized for subsequent learning of ambient lighting from image observations.

[0073] It should be understood that a common camera pose estimation algorithm, such as the colmap algorithm, may be used to perform Structure From Motion (SFM) recovery on each of the surround videos to obtain the camera pose information of the surround video.

[0074] It should also be noted that the preset differentiable renderer can adopt the diffrast differentiable renderer, which can render a four-channel RGBA image corresponding to the camera pose from the input mesh model with material, environment HDR and camera pose information.

[0075] Step A04: Calculate the loss based on the current rendering and the foreground image of the inserted target at different viewing angles to obtain a loss gradient, and optimize the geometry encoding perceptron, the texture encoding perceptron and the ambient lighting map through the loss gradient based on a gradient feedback mechanism.

[0076] It should be noted that, in order to obtain the foreground image of the inserted target at different viewing angles, before step A04, a background removal process of the surrounding video may be included, and the steps of the process include steps A001 to A002:

[0077] Step A001: upon receiving a click instruction from the user, generating a mask area corresponding to the insertion target in the current video frame of the surround video according to the click instruction.

[0078] Step A002: Segment the other video frames obtained by segmenting the surround video according to the mask area to obtain foreground images of the inserted target at different viewing angles.

[0079] It should be understood that a video marker segmentation model such as Track Anything can be used to implement video background removal in surround video. Specifically, the user can click on the uploaded surround video in the interactive interface of the editing device, select the insertion target to be 3D modeled in the current video frame, segment the insertion target by the video marker segmentation model and generate the corresponding mask area, and then automatically track the insertion target in other video frames using the frame tracking target technology to obtain the foreground four-channel image segmented from each frame of the surround video, that is, the foreground image under different viewing angles.

[0080] It should be noted that the foreground image under different viewing angles can be used as the ground truth and the four-channel RGBA image corresponding to the camera pose rendered by the differentiable renderer is used for loss calculation. The obtained loss gradient is back-propagated to the geometry encoding perceptron, texture encoding perceptron and initialized environment HDR map for parameter update optimization.

[0081] Step A05: Based on the optimization goal of minimizing the empirical risk, a preset three-dimensional reconstruction model is obtained.

[0082] It should be noted that the model training process can be regarded as an optimization task for the target shape representation of a triangular mesh with geometric shape, texture material and lighting. The above loss calculation formula can be:

[0083]

[0084]

[0085] L total =l1(S,R)+L light +L mat

[0086] Where l1(S,R) is the l1 loss of the current rendering and the foreground image R. light The regularization term constrains the brightness value c of each channel of the ambient HDR image i The difference is not too big, and it is closer to the white light environment in the real world. mat The regularization term constrains the base colors of adjacent mesh vertices (k d (x surf ) and k d (x surf +ε)) will not differ too much, making the texture smoother.

[0087] In the specific implementation, based on the optimization goal of minimizing empirical risk, the preset differentiable renderer diffrast is used to perform the above optimization tasks, and the relevant parameters of the geometric shape, texture material and lighting are updated through the back propagation of the loss gradient to realize the optimization process of the geometric coding perceptron, texture coding perceptron and the initialized environment HDR map, and finally a preset 3D reconstruction model with better effect is obtained, and then the target 3D model corresponding to the inserted target is constructed through the preset 3D reconstruction model.

[0088] Step S30: Obtain video reconstruction information according to the optical flow information of the video to be inserted in combination with a preset optimization algorithm, wherein the video reconstruction information includes camera internal and external parameter information and dense point cloud information.

[0089] It should be noted that the optical flow information can be used to describe the pixel motion in the video to be inserted. According to the optical flow information, the depth information corresponding to each video frame in the video to be inserted can be obtained, and then a dense point cloud of the video scene to be inserted can be reconstructed.

[0090] It should be understood that the preset optimization algorithm can adopt a self-calibration bundle adjustment method, and its optimization task is to minimize the reprojection error. It can be solved by, for example, the gradient descent method, the Newton method, or the Gauss-Newton method to achieve the solution of the camera internal and external parameter information to be inserted into the video.

[0091] Step S40: implanting the target three-dimensional model into the user-selected area in the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain an edited video.

[0092] It should be understood that the user can select the video to be inserted in the interactive interface of the editing device to determine the insertion position of the target three-dimensional model.

[0093] In a specific implementation, after obtaining the camera's internal and external parameter information and dense point cloud information, the coordinate point information selected by the user in the current frame of the video to be inserted can be received to generate a user-selected area, and then based on the user-selected area, the corresponding perspective images of the target three-dimensional model in each video frame of the video to be inserted are rendered, and multi-frame image rendering is completed and the final result video is synthesized to obtain the edited video.

[0094] In this embodiment, the user only needs to provide a surround video of the insertion target, and click to select the target to be reconstructed in the surround video, and the target three-dimensional model can be obtained from the preset three-dimensional reconstruction model, which simplifies the traditional three-dimensional reconstruction process and can reduce the three-dimensional reconstruction cost. Then, the target three-dimensional model can be inserted into the user-selected area of ​​the video to be inserted according to the video reconstruction information, realizing a video editing process from three-dimensional model production to video insertion, which greatly improves the video editing speed compared to the existing method.

[0095] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 2 , Figure 2 The flowchart of the second embodiment of the video editing method of the present application is provided, step S30, which specifically includes steps S301 to S303:

[0096] Step S301: inputting a plurality of video frames corresponding to the video to be inserted into the optical flow prediction model to determine the optical flow displacement value of each of the video frames.

[0097] It should be noted that the optical flow prediction model can use RAFT (Recurrent All-Pairs Field Transforms, a deep neural network for optical flow estimation). RAFT uses a neural network structure and deep learning technology to learn pixel-level motion information from image sequences. Compared with traditional optical flow algorithms, RAFT can better handle complex motion situations. It uses a recurrent neural network (RNN) to process image sequence information, and introduces technologies such as local block decoupling attention mechanism and deformable convolution, which is conducive to improving the accuracy and stability of optical flow estimation.

[0098] In a specific implementation, the video to be inserted is input into RAFT, and the optical flow displacement value of each video frame relative to the previous video frame can be obtained, thereby realizing the calculation of the optical flow information of each video frame of the video to be inserted.

[0099] Step S302: when the optical flow displacement value is greater than a preset threshold, adding the video frame corresponding to the optical flow displacement value to a key frame set.

[0100] In a specific implementation, an optical flow threshold can be set, and video frames whose optical flow displacement values ​​are higher than the optical flow threshold are added to the key frame set, and video frames whose optical flow displacement values ​​are not higher than the optical flow threshold are skipped. After traversing each video frame, the key frame set corresponding to the video to be inserted is finally obtained.

[0101] Step S303: using a preset optimization algorithm in the key frame set to determine video reconstruction information, and obtaining camera internal and external parameter information and dense point cloud information.

[0102] It should be noted that, in order to improve the optimization efficiency, the optimization task of obtaining the above video reconstruction information can be divided into an online optimization part and an offline optimization part. Therefore, step S303 also includes steps S3031 to S3033:

[0103] Step S3031: Divide the key frame set into an online optimization set and an offline optimization set.

[0104] It should be noted that the online optimization set may be a set to be optimized that only selects the last 25 frames in the key frame set as the online local estimation task; the offline optimization set corresponds to the offline global optimization task and may include all the key frames in the key frame set.

[0105] Step S3032: using a preset optimization algorithm in the online optimization set to determine the relative camera pose, depth map, and camera intrinsic parameters corresponding to each video frame.

[0106] Step S3033: using a preset optimization algorithm in the offline optimization set to update the camera relative pose, the depth map and the camera intrinsic parameters to obtain camera intrinsic and extrinsic parameter information and dense point cloud information.

[0107] It should be noted that the online local estimation task can be performed synchronously when each video frame to be inserted into the video is input. After reading the entire video frame sequence and completing the online local estimation, the camera relative pose, depth map and camera intrinsic parameters corresponding to each video frame can be obtained, and then the offline global optimization task can be performed to achieve parameter fine-tuning and updating.

[0108] It should be understood that the set to be optimized includes the online optimization set and the offline optimization set, and the corresponding optimization tasks are the online local estimation task and the offline global optimization task. When performing the optimization task, for any two frames i and j in the set to be optimized whose optical flow displacement is within the specified range, the camera internal and external parameters and the dense point cloud can be estimated by minimizing the reprojection error through the self-calibration bundle adjustment method.

[0109] The task function of minimizing the reprojection error is:

[0110]

[0111] Where P is the set to be optimized. ij is the relative camera pose between the i-th frame and the j-th frame. is the coordinate of the pixel in the i-th frame estimated by optical flow in the j-th frame. i is the depth map corresponding to the i-th frame. θ is the camera internal parameter. ∑ij is the estimated confidence weight. The larger the ∑ij, the better. The more accurate the estimation is, the more accurate the 3D coordinates under the camera coordinates can be projected to the pixel coordinate system according to the intrinsic parameters, π -1 (.,θ) back-projects the pixel coordinates to the camera coordinate system based on the camera intrinsic parameters and their corresponding depth values. Since minimizing the reprojection error is a nonlinear optimization problem, the Gauss-Newton method can be used to iteratively optimize the relative pose G ij , key frame depth z i And the camera intrinsic parameter θ.

[0112] In the specific implementation, after completing the above optimization tasks, the relative camera pose corresponding to each video frame, that is, the camera extrinsic parameters, the depth value of each video frame (dense point cloud information) and the camera intrinsic parameters corresponding to the video frame sequence can be restored to complete the video reconstruction of the video to be inserted.

[0113] This embodiment adopts the self-calibration bundle adjustment method combined with the optical flow algorithm RAFT in the video reconstruction process of the video to be inserted, which can restore the camera internal and external parameters of any continuous video. It is suitable for any video to be inserted with unknown camera parameters, and reconstructs a dense point cloud of the corresponding video scene, thereby enhancing the ability to locate weak texture areas and repeated texture areas in the video scene, and solving the problem that the sparse point cloud of the scene restored by traditional camera tracking software cannot locate weak texture areas and repeated texture areas in the video.

[0114] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those of the first embodiment and the second embodiment can refer to the above introduction, and will not be repeated later. Figure 3 , Figure 3 The flowchart of the third embodiment of the video editing method of the present application is provided, step S40, including steps S401 to S404:

[0115] Step S401: upon receiving a selection instruction from the user, generating a user-selected area in the video to be inserted.

[0116] It should be noted that the user can click on the interactive interface of the editing device to determine the insertion position of the target 3D model in the video to be inserted. Specifically, the user can click on the first frame of the video to be inserted to select four coordinate points to generate a user-selected area.

[0117] Step S402: back-projecting the pixel coordinates of the area selected by the user into the camera coordinate system corresponding to the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain the three-dimensional point cloud coordinates of the area selected by the user.

[0118] It should be noted that the back projection formula π -1 (u r ,z r ,θ) The pixel coordinates u of the area selected by the user r Back-projected to the 3D point cloud coordinates in camera coordinates, where z r The dense point cloud depth value of the video to be inserted is used to determine the three-dimensional point cloud coordinates of the area selected by the user in the camera coordinate system of the video to be inserted.

[0119] Step S403: Determine the three-dimensional plane normal vector corresponding to the video to be inserted, and pre-process the target three-dimensional model in combination with the three-dimensional point cloud coordinates of the area selected by the user.

[0120] It should be understood that the three-dimensional plane normal vector can be determined based on the camera coordinate system corresponding to the video to be inserted based on a random sampling consensus algorithm; based on the three-dimensional plane normal vector, the target three-dimensional model is preprocessed in combination with the three-dimensional point cloud coordinates of the user-selected area.

[0121] It should be noted that the RANdom SAmple Consensus (RANSAC) algorithm is an iterative optimization algorithm based on random sampling and consistency check. The normal vector can be estimated by fitting a plane model. The RANSAC algorithm can also filter out some pixels that are mistakenly selected by the user and noise values ​​of the depth estimation.

[0122] It should also be noted that, considering that the plane normal vector estimated by the RANSAC algorithm may be upward or downward, additional constraints can be added to prevent the subsequent insertion of the target 3D model from being upside down. Specifically, for the estimated plane normal vector The world coordinate system coordinate c of the camera position and the coordinate p of any point on the plane need to satisfy If the condition is not met, the normal vector Do the reverse.

[0123] It should be understood that after obtaining the 3D plane normal vector, when preprocessing the target 3D model, the editing device can rotate the y-axis direction of the target 3D model to be in the same direction as the 3D plane normal vector to achieve a placement visual effect. Furthermore, in order to fit the bottom surface of the target 3D model to the user-selected area as closely as possible, the bottom surface center of the target 3D model can be moved to the plane center of the user-selected area based on the 3D point cloud coordinates of the user-selected area, and the size of the target 3D model can be adjusted according to the ratio of the distance of the user-selected area extending on the x-axis and z-axis to the width of the target 3D model in the x-axis and z-axis directions.

[0124] Step S404: Rendering preprocessed images of different viewing angles corresponding to the target three-dimensional model according to the internal and external parameter information of the video, and synthesizing the edited video according to each of the viewing angle images and a number of video frames corresponding to the video to be inserted.

[0125] It should be noted that after completing the preprocessing of the target three-dimensional model, the perspective pictures corresponding to each video frame to be inserted into the video can be rendered according to the camera intrinsic parameters and camera extrinsic parameters obtained in the above video reconstruction process. The different perspective pictures are RGBA images, and the RGBA images are consistent with the video frame pixels of the inserted video.

[0126] It should also be noted that, since the target 3D model is placed in the user-selected area when the first frame of the video to be inserted is initialized during the preprocessing process, the 2D results obtained by rendering the target 3D model in the RGBA images of different viewing angles are all located in the user-selected area.

[0127] Furthermore, considering that only the camera extrinsic parameters of the key frames are restored in the above video reconstruction process, for the non-key frames to be inserted into the video, their approximate camera extrinsic parameters can be obtained by linear interpolation of adjacent key frames.

[0128] It should also be noted that since the rendering in each video frame to be inserted into the video is relatively independent, the pytorch tensor-based renderer pytorch3d can be used to render multiple frames of images in parallel and synthesize the final result video, where the synthesis process of each rendered RGBA image and the corresponding video frame can also be implemented in parallel.

[0129] In a specific implementation, the camera intrinsic parameters corresponding to the video to be inserted and the camera intrinsic parameters corresponding to each video frame to be inserted can be used to render the RGBA image corresponding to each video frame, and the RGBA image and the video frame can be synthesized to obtain the result video frame, thereby obtaining the edited video.

[0130] Furthermore, the interactive interface of the editing device can also provide the user with a fine-tuning interface for the size and orientation of the target three-dimensional model, so that the user can further adjust the insertion effect of the target three-dimensional model.

[0131] In this embodiment, the two-dimensional pixel coordinates of the user-selected area can be back-projected into the world coordinate system and the three-dimensional plane parameters can be estimated, and the initial size of the target three-dimensional model can be adjusted according to the size of the user-selected area. The user only needs to generate the selected area by clicking on the first frame of the video to be inserted, without setting the camera and plane parameters in the three-dimensional rendering space as in traditional post-production software, and without manually aligning the plane to be video and the three-dimensional plane, providing a user-friendly video editing method that can meet the video editing needs of general users to insert a three-dimensional model into an existing video.

[0132] This embodiment also provides a video editing device, referring to Figure 4 , Figure 4 A schematic diagram of the module structure provided in the first embodiment of the video editing device of the present application; the video editing device comprises: a model generation module 401, a video reconstruction module 402 and a model implantation module 403;

[0133] The model generation module 401 is used to obtain a surround video of the inserted target, wherein the surround video includes video frames of the inserted target at different viewing angles;

[0134] The model generation module 401 is further used to input the surround video into a preset 3D reconstruction model to obtain a target 3D model, wherein the preset 3D reconstruction model is trained by each video frame obtained by segmenting the surround video, and the preset 3D reconstruction model includes a geometric coding perceptron and a texture coding perceptron;

[0135] The video reconstruction module 402 is used to obtain video reconstruction information according to the optical flow information of the video to be inserted in combination with a preset optimization algorithm, wherein the video reconstruction information includes camera internal and external parameter information and dense point cloud information;

[0136] The model implantation module 403 is used to implant the target three-dimensional model into the user-selected area in the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain the edited video.

[0137] Furthermore, the model generation module 401 is also used to initialize a tetrahedral mesh, input the three-dimensional coordinates of the tetrahedral mesh into the geometry coding perceptron to obtain corresponding geometry information, and convert the geometry information to obtain a target mesh; input the three-dimensional coordinates of the target mesh into the texture coding perceptron to obtain texture information; input the ambient lighting map, the camera pose information of the surround video, the target mesh and the texture information into a preset differentiable renderer to obtain a current rendering; perform loss calculation based on the current rendering and the foreground image of the inserted target at different viewing angles to obtain a loss gradient, and optimize the geometry coding perceptron, the texture coding perceptron and the ambient lighting map through the loss gradient based on a gradient feedback mechanism; obtain a preset three-dimensional reconstruction model based on the optimization goal of minimizing empirical risk.

[0138] Furthermore, the model generation module 401 is also used to generate a mask area corresponding to the insertion target in the current video frame of the surround video according to the click instruction when receiving a user's click instruction; and to segment other video frames obtained by segmenting the surround video according to the mask area to obtain foreground images of the insertion target at different perspectives.

[0139] Furthermore, the video reconstruction module 402 is also used to input several video frames corresponding to the video to be inserted into the optical flow prediction model to determine the optical flow displacement value of each of the video frames; when the optical flow displacement value is greater than a preset threshold, the video frame corresponding to the optical flow displacement value is added to a key frame set; a preset optimization algorithm is used in the key frame set to determine the video reconstruction information to obtain the camera internal and external parameter information and dense point cloud information.

[0140] Furthermore, the video reconstruction module 402 is also used to divide the key frame set into an online optimization set and an offline optimization set; in the online optimization set, a preset optimization algorithm is used to determine the camera relative pose, depth map and camera intrinsic parameters corresponding to each video frame; in the offline optimization set, a preset optimization algorithm is used to update the camera relative pose, the depth map and the camera intrinsic parameters to obtain camera internal and external parameter information and dense point cloud information.

[0141] Furthermore, the model implantation module 403 is also used to back-project the pixel coordinates of the user-selected area into the camera coordinate system corresponding to the video to be inserted according to the camera internal and external parameter information and the dense point cloud information, so as to obtain the three-dimensional point cloud coordinates of the user-selected area; determine the three-dimensional plane normal vector corresponding to the video to be inserted, and pre-process the target three-dimensional model in combination with the three-dimensional point cloud coordinates of the user-selected area; render different perspective images corresponding to the pre-processed target three-dimensional model according to the video internal and external parameter information, and synthesize the edited video according to each of the perspective images and several video frames corresponding to the video to be inserted.

[0142] Furthermore, the model implantation module 403 is also used to determine the three-dimensional plane normal vector based on the camera coordinate system corresponding to the video to be inserted based on a random sampling consensus algorithm; based on the three-dimensional plane normal vector, the target three-dimensional model is pre-processed in combination with the three-dimensional point cloud coordinates of the user-selected area.

[0143] In this embodiment, the user only needs to provide a surround video of the inserted target, and the model generation module can obtain the target 3D model corresponding to the inserted target, and the video reconstruction module restores the video reconstruction information including the camera internal and external parameters of the video to be inserted and the dense point cloud information. Finally, the target 3D model is inserted into the user-selected area according to the video reconstruction information in the model implantation module, and the edited video is rendered and synthesized. The video editing process from 3D model production to video insertion can be realized, meeting the video editing needs of general users to insert 3D models into existing videos. Compared with the existing methods, the threshold of video editing is lowered and the speed of video editing is improved.

[0144] The present application provides a video editing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video editing method in the above-mentioned embodiment one.

[0145] Reference below Figure 5 , Figure 5The device structure diagram of the hardware operating environment involved in the video editing method in the embodiment of the present application is shown in FIG. The video editing device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The video editing device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0146] like Figure 5 As shown, the video editing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the video editing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. Communication device 1009 can allow the video editing device to communicate wirelessly or wired with other devices to exchange data. Although the video editing device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0147] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0148] The video editing device provided by the present application adopts the video editing method in the above embodiment to solve the technical problems of video editing. Compared with the prior art, the beneficial effects of the video editing device provided by the present application are the same as the beneficial effects of the video editing method provided by the above embodiment, and the other technical features in the video editing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0149] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0150] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0151] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the video editing method in the above-mentioned embodiment.

[0152] The computer-readable storage medium may be included in the video editing device; or may exist independently without being assembled into the video editing device.

[0153] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a video editing device, the video editing device: obtains a surround video of an inserted target, wherein the surround video contains video frames of the inserted target at different viewing angles; inputs the surround video into a preset 3D reconstruction model to obtain a target 3D model, wherein the preset 3D reconstruction model is trained by video frames obtained by segmenting the surround video; obtains video reconstruction information based on optical flow information of the video to be inserted in combination with a preset optimization algorithm, wherein the video reconstruction information includes camera internal and external parameter information and dense point cloud information; and implants the target 3D model into a user-selected area in the video to be inserted based on the camera internal and external parameter information and the dense point cloud information to obtain an edited video.

[0154] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned video editing method, and can solve the technical problems of video editing. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the video editing method provided in the above-mentioned embodiment, and will not be described in detail here.

[0155] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A video editing method, It is characterized in that The method comprises: Acquire a surround video of the inserted target, wherein the surround video includes video frames of the inserted target at different viewing angles; Inputting the surround video into a preset 3D reconstruction model to obtain a target 3D model, wherein the preset 3D reconstruction model is trained by each video frame obtained by segmenting the surround video; Obtaining video reconstruction information based on the optical flow information of the video to be inserted and a preset optimization algorithm, wherein the video reconstruction information includes camera internal and external parameter information and dense point cloud information; implanting the target three-dimensional model into a user-selected area in the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain an edited video; The step of obtaining video reconstruction information based on the optical flow information of the video to be inserted in combination with a preset optimization algorithm includes: Inputting a plurality of video frames corresponding to the video to be inserted into the optical flow prediction model to determine the optical flow displacement value of each of the video frames; When the optical flow displacement value is greater than a preset threshold, adding the video frame corresponding to the optical flow displacement value to a key frame set; A preset optimization algorithm is used in the key frame set to determine the video reconstruction information to obtain camera internal and external parameter information and dense point cloud information.

2. The method according to claim 1, It is characterized in that The preset 3D reconstruction model includes a geometric coding perceptron and a texture coding perceptron; the training steps of the preset 3D reconstruction model include: Initialize a tetrahedral mesh, input the three-dimensional coordinates of the tetrahedral mesh into the geometric encoding perceptron, obtain corresponding geometric information, and convert the target mesh according to the geometric information; Inputting the three-dimensional coordinates of the target grid into the texture encoding perceptron to obtain texture information; Inputting the environment lighting map, the camera pose information of the surround video, the target mesh and the texture information into a preset differentiable renderer to obtain a current rendering image; Perform loss calculation according to the current rendering image and the foreground image of the inserted target at different viewing angles to obtain a loss gradient, and optimize the geometry encoding perceptron, the texture encoding perceptron and the ambient lighting map through the loss gradient based on a gradient feedback mechanism; Based on the optimization goal of minimizing empirical risk, a preset three-dimensional reconstruction model is obtained.

3. The method according to claim 2, It is characterized in that Before the step of initializing the tetrahedral mesh, inputting the three-dimensional coordinates of the tetrahedral mesh into the geometric coding perceptron, obtaining corresponding geometric information, and converting the target mesh according to the geometric information, the method includes: When receiving a click instruction from the user, generating a mask area corresponding to the insertion target in the current video frame of the surround video according to the click instruction; Segmentation is performed in other video frames obtained by segmenting the surround video according to the mask area to obtain foreground images of the inserted target at different viewing angles.

4. The method according to claim 1, It is characterized in that The step of using a preset optimization algorithm in the key frame set to determine the video reconstruction information and obtain camera internal and external parameter information and dense point cloud information includes: Dividing the key frame set into an online optimization set and an offline optimization set; In the online optimization set, a preset optimization algorithm is used to determine the relative position and depth map of the camera and the camera intrinsic parameters corresponding to each video frame; In the offline optimization set, a preset optimization algorithm is used to update the camera relative pose, the depth map and the camera intrinsic parameters to obtain camera internal and external parameter information and dense point cloud information.

5. The method according to claim 1, It is characterized in that The step of implanting the target three-dimensional model into the user-selected area in the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain the edited video includes: Upon receiving a selection instruction from a user, generating a user-selected area in the video to be inserted; Back-projecting the pixel coordinates of the area selected by the user into the camera coordinate system corresponding to the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain the three-dimensional point cloud coordinates of the area selected by the user; Determine the three-dimensional plane normal vector corresponding to the video to be inserted, and pre-process the target three-dimensional model in combination with the three-dimensional point cloud coordinates of the area selected by the user; Different viewing angle pictures corresponding to the preprocessed target three-dimensional model are rendered according to the camera internal and external parameter information, and the edited video is synthesized according to each of the viewing angle pictures and a plurality of video frames corresponding to the video to be inserted.

6. The method according to claim 5, It is characterized in that The step of determining the three-dimensional plane normal vector corresponding to the video to be inserted, and preprocessing the target three-dimensional model in combination with the three-dimensional point cloud coordinates of the area selected by the user, comprises: Based on a random sampling consensus algorithm, a three-dimensional plane normal vector is determined according to a camera coordinate system corresponding to the video to be inserted; The target three-dimensional model is preprocessed based on the three-dimensional plane normal vector and combined with the three-dimensional point cloud coordinates of the user-selected area.

7. A video editing device, It is characterized in that The device comprises: a model generation module, a video reconstruction module and a model implantation module; The model generation module is used to obtain a surround video of the inserted target, wherein the surround video includes video frames of the inserted target at different viewing angles; The model generation module is further used to input the surround video into a preset 3D reconstruction model to obtain a target 3D model, wherein the preset 3D reconstruction model is trained by each video frame obtained by segmenting the surround video; The video reconstruction module is used to obtain video reconstruction information according to the optical flow information of the video to be inserted in combination with a preset optimization algorithm, wherein the video reconstruction information includes camera internal and external parameter information and dense point cloud information; The model implantation module is used to render the target three-dimensional model to the user-selected area in the video to be inserted according to the camera internal and external parameter information and the dense point cloud information to obtain an edited video; The video reconstruction module is also used to input several video frames corresponding to the video to be inserted into the optical flow prediction model to determine the optical flow displacement value of each video frame; when the optical flow displacement value is greater than a preset threshold, the video frame corresponding to the optical flow displacement value is added to a key frame set; a preset optimization algorithm is used in the key frame set to determine the video reconstruction information to obtain the camera internal and external parameter information and dense point cloud information.

8. A video editing device, It is characterized in that The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video editing method according to any one of claims 1 to 6.

9. A storage medium, It is characterized in that The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the video editing method according to any one of claims 1 to 6 are implemented.

Images