A video compression method based on 3D modeling of key objects

Abstract

The present invention discloses a video compression method based on 3D modeling of key objects, and belongs to the field of video signal processing. The method of the present invention comprises: extracting key frames of key target objects from the compressed video; estimating camera parameters using key frames and obtaining motion trajectories; training the NeRF model of the target object; reconstructing the video based on the NeRF model of the target object, camera parameters, etc., using the NeRF model in the key frame in combination with the camera parameters to render the appearance of the target object, interpolating and rendering the non-key frame, comparing the reconstructed video with the original video, and if the quality meets the requirements, using the current target object NeRF model file and the camera motion trajectory file as the compressed storage file of the target object in the original video. The present invention can still accurately restore the details such as the morphology, light and shadow information of the target object while achieving a high compression rate for the video, and is more efficient and more suitable for quality improvement and enhancement during transmission.

Description

Technical Field

[0001] The invention belongs to the field of video signal processing, and in particular is a video compression method based on 3D modeling of key objects. Background Art

[0002] In today's digital age, the generation and dissemination of video data has exploded, especially in the field of cultural relics protection; as a precious heritage of history and culture, the protection and inheritance of cultural relics is of vital importance. However, the preservation, restoration and display of cultural relics involve a large amount of visual data, such as physical cultural relics, documentary materials and archaeological sites. These video data not only contain rich historical information, but also record the appearance, texture, light and shadow details of cultural relics, which helps researchers, scholars and the public better understand the culture and civilization of the past.

[0003] However, as the video data generated in the field of cultural relics protection is huge and growing, it poses a huge challenge to data storage, transmission and management. Traditional video data compression methods find it difficult to achieve high compression rates while retaining important details, resulting in information loss and quality degradation. The current mainstream video compression standards are based on pixel-level encoding for each frame. Traditional encoding compression storage is not efficient for static object areas. Standards such as H.264 / H.265 are two-dimensional compression methods based on encoding pixels for each frame. This method encodes and transmits the entire frame, with general compression efficiency, making it difficult to fully compress, unable to preserve the fine three-dimensional structural information of the object, and poor reconstruction from different perspectives. Therefore, an innovative method is needed to balance data preservation and transmission efficiency during video data compression.

[0004] The prior art involved in the present invention is described below.

[0005] NeRF (Neural Radiance Fields) is a technology that uses deep learning to model three-dimensional scenes. Its working principle is as follows: Use a neural network to model light field representation, and model each light in the scene into a multi-layer perceptron (MLP). The input of the MLP is the direction and position of the light, and the output is the color of the light at that position and the degree of light scattering. End-to-end training is performed through a large amount of sampled training data to automatically learn the geometric structure and material properties of the scene. The training data comes from a collection of photos taken by multi-view cameras. Use each picture to extract the (direction, depth) pair corresponding to each pixel as input. The trained model can be used to render photos from any new perspective to achieve rendering effects that imitate real scenes. The NeRF model has the ability to continuously represent the light field, and the output can also smoothly transition with slight changes in position and direction. Compared with traditional methods, NeRF can accurately reconstruct complex three-dimensional scenes with fewer parameters.

[0006] The YOLO (You Only Look Once) algorithm is a very popular target detection algorithm. Its main features include: fast speed, which can achieve the effect of real-time target detection; end-to-end detection of targets without generating candidate boxes. The typical process of the YOLO algorithm is: (1) Divide the image into S×S grids; (2) Predict B bounding boxes for each grid, each of which contains position coordinates, width and height dimensions, confidence, and the probability of C categories; (3) Use non-amplified values ​​to expand overlaps; (4) Retain the bounding box with the maximum confidence threshold as the detection result according to the weight.

[0007] COLMAP software / algorithm combines multiple keyframes to estimate parameters and 3D coordinates through the differences between each image. The main steps of the COLMAP algorithm include: (1) import multiple keyframe images into COLMAP; (2) use feature point extraction and matching functions in COLMAP to find the corresponding feature point relationship between images; (3) use COLMAP's incremental SfM (3D reconstruction) module to estimate parameters and sparse point cloud based on feature point matching results; (4) add all images to the reconstructed scene and perform global BA (Bundle Adjustment) optimization to obtain optimized camera parameters; (5) for each image, its external parameter matrix, internal parameter matrix and other parameters can be obtained through the database data provided by COLMAP; (6) from the optimized sparse point cloud, the world coordinates of each 3D point can be derived; combining camera parameters, 3D point coordinates and reprojection projection, the position and viewing angle differences between each image can be analyzed. By analyzing these differences, the measurement results of motion can be estimated and 3D scene analysis can be performed.

[0008] Instant-ngp is an open source deep learning framework for training neural representations of 3D environments from multi-view photos. Based on NeRF, a neural light field modeling algorithm, Instant-ngp uses an end-to-end architecture of a full neural network for 3D modeling. It supports different backbone neural networks such as MLP and matrix neural networks to represent scenes and objects. It can also directly use image data as input to automatically calibrate and extract (view, depth) pairs from images. Instant-ngp can model dynamic and static environments, extract different elements in the scene, such as objects, materials, and light sources, and provides a complete training and rendering layout. After the model is trained, it supports rendering from any perspective. Summary of the invention

[0009] In order to overcome the shortcomings of traditional technologies and efficiently compress video data while retaining key object information, the present invention provides a video compression method based on 3D modeling of key objects, and implements video compression based on NeRF technology. While achieving a high compression rate, it can also ensure that the shape, texture, and light and shadow information of the target object are accurately restored.

[0010] The video compression method based on 3D modeling of key objects provided by the present invention stores a corresponding computer program on a readable storage medium, and the program is executed by a processor to perform the following steps:

[0011] Step 1: for the video to be compressed, identify or select the key target object in the video and extract the key frame of the target object;

[0012] Step 2: Use the key frame of the target object to estimate the camera parameters and obtain the camera motion trajectory file; the camera parameters include internal parameters and external parameters, and the internal parameters include the position and direction of the camera at the corresponding time point; the camera motion trajectory file stores the camera parameters corresponding to each key frame in chronological order;

[0013] Step 3: Obtain multi-angle images of the target object and train the NeRF model of the target object; the NeRF model maps the two-dimensional image information of the target object into a three-dimensional scene and can capture details such as the shape, texture, light and shadow of the object in the scene.

[0014] Step 4: Reconstruct the video using the NeRF model of the target object, the camera motion trajectory file and the compressed key frames, and compare it with the original video. If the quality of the reconstructed video meets the requirements, use the current target object NeRF model file and the camera motion trajectory file as the compressed storage file of the target object in the original video; if the quality of the reconstructed video does not meet the requirements, further execute step 3 to adjust the NeRF model of the target object.

[0015] In the second step, the COLMAP algorithm is used to obtain the corresponding camera parameters for each key frame, and the camera parameters of each image are estimated; the camera posture of the image is sorted according to the shooting time to form the motion trajectory of the camera. For each key frame of the target object, the 3D coordinates of the feature points of the target object are obtained; and the position change information of the feature points of the target object in the video is stored.

[0016] In the step three, after the NeRF model of the target object is obtained through training, the three-dimensional model file of the target object is exported, the object model is repaired in the three-dimensional scene, and then the NeRF model of the corresponding repaired target object is obtained.

[0017] In the step 4, when reconstructing the video, for each key frame, the corresponding camera parameters in the camera motion trajectory file are read, and the appearance of the target object is rendered using the NeRF model combined with the camera parameters; interpolation rendering is performed on non-key frames; and each frame of the rendering is written into the video file in frame order.

[0018] Compared with the traditional video compression method, the method of the present invention has the following advantages:

[0019] (1) The method of the present invention can still accurately restore the shape, light and shadow information and other details of the target object while ensuring a high compression rate. The method of the present invention uses NeRF technology to perform 3D modeling and store the target object. The corresponding NeRF model not only captures the shape structure of the object, but also distinguishes the material, light and shadow and other information; then only the key frames and interpolation parameters need to be transmitted, so the amount of data after video compression is greatly reduced.

[0020] (2) The method of the present invention can better preserve the important details of key objects in the video. Information will not be lost when the video is compressed and stored, and the quality of storage and restoration will not decrease, and serious distortion will not occur. At the same time, the three-dimensional scene representation of the method of the present invention is convenient for post-editing and processing, such as inserting digital objects or modifying scene details. The method of the present invention uses the NeRF model to reconstruct the video with a sense of coherence, and interpolates and renders non-key frames, so that the visual quality is higher than traditional encoding, ensuring that the morphology, texture, and light and shadow information of the target object are accurately restored, with higher efficiency and more suitable for quality improvement and enhancement during transmission.

[0021] (3) Based on the detection of the key frame of the target object, the method of the present invention can further detect key points on the object that are representative and distinguishing, such as edges, corners, etc., and track these key points as feature points, which can effectively improve the tracking stability and accuracy; and during encoding and transmission, only the position change information of the feature points of the target object can be stored, rather than the entire object boundary, which can further reduce the amount of data.

[0022] (4) After encoding three-dimensional information, the method of the present invention can freely switch the viewing angle to view the scene, realize six-degree-of-freedom viewpoint control, and achieve interactivity that cannot be achieved by some traditional video compression methods.

[0023] (5) When compressing a video, the method of the present invention only needs to store information such as the model file and the camera parameters used to restore the trajectory. Under the premise of ensuring the quality of video restoration, the bit rate requirement is greatly reduced, and the required storage space is smaller, which opens up a new situation for Internet video applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of the video compression method based on key object 3D modeling of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0026] The present invention implements a video compression method based on 3D modeling of key objects, firstly identifying an object with a relatively long display time in the video content, then using photos from multiple angles to perform 3D modeling on the object, separating the object's morphological texture and light and shadow, and finally using 3D modeling to store the key object, retaining the object's realistic perspective and light and shadow information at different times. The present invention stores the target object through the NeRF model, and then compresses the video in combination with the camera trajectory and key frames, which greatly reduces the video storage space, and the video restoration quality is higher than that of traditional coding.

[0027] like Figure 1 As shown, the video compression method based on 3D modeling of key objects of the present invention includes the following five steps.

[0028] Step 1: For the video to be compressed, select the key target object that you want to compress in the video and extract the key frame of the target object.

[0029] In the embodiment of the present invention, whether an object is a key target object is determined according to the length of time the object appears in the video, the target object is identified using the target detection algorithm Yolo, and several key frames of the target object are extracted from the video to be compressed.

[0030] Step 2: Use the key frame of the target object to estimate the camera parameters and obtain the camera's shooting trajectory file.

[0031] The COLMAP algorithm is used to obtain the difference of each key frame, calculate the camera parameter coordinates, perform multi-view stereo reconstruction, and restore the camera's shooting trajectory according to the time sequence of image shooting.

[0032] In the embodiment of the present invention, the COLMAP algorithm is used to calculate the camera parameter coordinates based on the difference of each frame in combination with multiple key frames, and multi-view stereo reconstruction is performed to restore the shooting trajectory so as to reconstruct the original video in the future. Finally, the tool or interface provided by COLMAP can be used to export parameter data, including 3D feature point coordinates, camera parameters and camera posture.

[0033] When compressing a video, the present invention may only compress key frames in a traditional manner, while storing information about the target object, including the target object coordinates and the NeRF model, as well as camera parameters.

[0034] In the embodiment of the present invention, Yolo is used to detect key points on the target object that are representative and distinguishable, such as edges and corners, and these key points are tracked as feature points, which can effectively improve tracking stability and accuracy. When encoding and transmitting, only the position change information of the feature points of the target object can be stored, rather than the entire object boundary, which can further reduce the amount of data.

[0035] In an embodiment of the present invention, in COLMAP, the 3D coordinates of the selected feature points of the target object are obtained, and the intrinsic parameter matrix K of each camera is estimated using the feature point coordinates, and the focal length and principal point parameters of the camera are recorded. Then the rotation matrix R and translation vector t of each key frame relative to its adjacent key frame are calculated, and recorded as the camera relative posture [R|t] using a chessboard reconstruction format. Finally, the position and posture constraints of each camera are recorded as a trajectory file in the form of [time, R, t]. The present invention uses a table or parameter file to record the correspondence between each image name and the intrinsic parameter K and extrinsic parameter [R|t] of the corresponding camera; records the correspondence table of each extracted feature point in the image pixel coordinates and the three-dimensional space coordinates; monitors the projection error by bundling adjustment, and when the error is minimized, saves the optimal values ​​of the intrinsic parameter matrix K and extrinsic parameter matrix [R|t] of each camera.

[0036] In the process of reconstructing camera parameters using the COLMAP algorithm, in addition to reconstructing the 3D points in the scene, the camera pose of each image is also estimated. These poses describe the position and direction of the camera at each time point or for each image. Therefore, when there is a sequence of images taken in chronological order, these estimated camera poses can be regarded as the motion trajectory of the camera, and then the original video can be further reconstructed through the motion trajectory and the model.

[0037] Step 3: Use instant-ngp to quickly train the NeRF model of the target object, map the two-dimensional image information into the three-dimensional scene, and capture details such as morphology, texture, and light and shadow.

[0038] First, take photos of the target object from multiple perspectives. The multi-perspective photos can be obtained by using a camera array or taking them separately at different positions. Then use instant-ngp to quickly train the NeRF model using the multi-perspective images. By obtaining multi-angle images of the target object, the NeRF model learns a simple lighting model and separates the object's own color tone and lighting effect.

[0039] NeRF technology is to train a neural network model, i.e., a NeRF model, by taking photos of objects that appear for a long time and aligning and analyzing them, so as to describe the appearance and structure of the object with parameters, thereby 3D modeling and storing the object. The trained NeRF model not only captures the shape and structure of the object, but also distinguishes information such as material, light and shadow.

[0040] Step 4: Export the 3D model file of the target object from the trained NeRF model, repair the object model in the 3D scene, and export the repaired object model file.

[0041] In an embodiment of the present invention, a mesh (triangular patch) file is exported from the NeRF model, and the model is first repaired using meshlab software and then exported to 3Dmax software. In 3Dmax, the model in the mesh file can be subdivided, such as rounding the corners of the model, removing burrs, etc., to make the model more refined.

[0042] In the embodiment of the present invention, model repair is performed in meshlab software, including the following steps:

[0043] Repair bumpy points: Select Mesh>smooth vertex to smooth overly bumpy mesh vertices and repair shape problems.

[0044] Hole repair: Select Mesh>Fill Hole to automatically repair small holes that may exist on the surface of the model.

[0045] Self-intersection repair: Select Edge>Self Collision to detect and repair overlapping intersection problems between mesh elements.

[0046] Gap repair: Select Edge>Close Border to repair gaps that may appear at the edges of the grid.

[0047] Normalization: Select Mesh>Normals>Normalize All to correct the direction of the normals of the entire mesh surface.

[0048] Topology Check: Select Mesh>Topology Check to check the mesh topology for problems such as overlapping vertices.

[0049] Smoothing operation: Select the Smooth tool in the Modify Panel to smooth the area.

[0050] Repair tool: Select Mesh>Repair menu to repair specific problems such as self-intersections, cross sections, etc.

[0051] Reconfiguration: Retriangulation to improve the overall mesh structure or increase the level of detail.

[0052] Step 5: Reconstruct the original video using the repaired object model and camera motion trajectory file, and compare the original video with the reconstructed original video. If the quality meets the requirements, use the target object model file and the camera motion trajectory file exported in step 4 as the compressed storage file of the target object in the original video. If the quality does not meet the requirements, the NeRF model of the target object needs to be re-adjusted, and go to step 3 for execution.

[0053] The method of the present invention is improved based on the existing video compression technology. The existing technology can be used to compress key frames, and then combined with the target object model file, the target object coordinates and the camera motion trajectory file to generate a compressed file of the original video.

[0054] The restored object model is a NeRF model. Using the NeRF model to reconstruct the original video includes the following steps:

[0055] Step 501: Load the NeRF model file and obtain model parameters and structure;

[0056] Step 502: read the camera motion trajectory and obtain the intrinsic and extrinsic parameter matrices corresponding to each frame of the image;

[0057] Step 503: Set the resolution and frame rate of the reconstructed video;

[0058] Step 504, initialize the video file and write header information;

[0059] Step 505, for each frame of the image: 1) update the viewing angle matrix of the current frame according to the external parameters; 2) call the NeRF renderer function to render; 3) update the projection according to the internal parameters, sample and generate the light under the viewing angle; 4) input the light to the NeRF model and predict the color value; 5) accumulate the colors and output the complete frame;

[0060] Step 506, write each frame rendering image into the video file in frame order; obtain the non-key frame by interpolation rendering;

[0061] Step 507: After all frames are completed, video reconstruction is completed.

[0062] The present invention adopts NeRF technology to track the position and posture of key objects in the video in real time, uses the NeRF model combined with camera parameters to perform 3D modeling on the target object in each key frame of the video, and establishes a light field representation. According to the obtained neural light field model, the appearance rendering result of the target object can be quickly synthesized for the corresponding viewing angle, thereby completing the 3D reconstruction of the target object.

[0063] It can be seen from the above that, by adopting the video compression encoding method of the present invention, only the NeRF model file of the target object and the shooting trajectory file of the camera need to be transmitted, which reduces the size of the compressed storage file compared with the prior art. When reconstructing the video, the NeRF model file is used to reconstruct the scene, and the model can be further repaired using software such as meshlab and 3Dmax. Combined with the camera parameters, the video can be reconstructed, so that the morphology, texture, and light and shadow information of the target object can be accurately restored. Compared with the traditional video compression method, the visual quality after reconstruction will not be degraded and will not cause serious distortion.

[0064] Except for the technical features described in the specification, all of them are known technologies to those skilled in the art. The present invention omits the description of known components and known technologies to avoid redundancy and unnecessary limitation of the present invention. The implementation methods described in the above embodiments do not represent all implementation methods consistent with the present application. On the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative labor are still within the protection scope of the present invention.

Claims

1. A video compression method based on 3D modeling of key objects, characterized in that: A computer program corresponding to the method is stored on a readable storage medium, and when the program is executed by a processor, the computer program includes the following steps: (1) for the video to be compressed, identifying or selecting a key target object in the video and extracting a key frame of the target object; judging whether the object is a key target object based on the length of time the object appears in the video; (2) estimating camera parameters using the key frames of the target object to obtain a camera motion trajectory file; wherein the camera parameters include external parameters and internal parameters, and the internal parameters include the position and direction of the camera at a corresponding time point; the camera motion trajectory file stores the camera parameters corresponding to the key frames sorted in chronological order; (3) acquiring multi-angle images of the target object and training the NeRF model of the target object; After the NeRF model of the target object is obtained through training, the 3D model file of the target object is exported, the object model is repaired in the 3D scene, and then the NeRF model of the corresponding repaired target object is obtained; (iv) reconstructing the video using the NeRF model of the target object, the camera motion trajectory file and the compressed key frames, and comparing it with the original video. If the quality of the reconstructed video meets the requirements, the current target object NeRF model file and the camera motion trajectory file are used as the compressed storage file of the target object in the original video; if the quality of the reconstructed video does not meet the requirements, further executing step 3 to adjust the NeRF model of the target object; When reconstructing the video, for each key frame, the corresponding camera parameters in the camera motion trajectory file are read, and the appearance of the target object is rendered using the NeRF model combined with the camera parameters; interpolation rendering is performed on non-key frames; and each frame of the rendering is written into the video file in frame order.

2. The method according to claim 1, characterized in that The step 2 comprises: Use the COLMAP algorithm to obtain the corresponding camera parameters for each key frame of the target object and estimate the camera parameters, including: the camera's intrinsic parameter matrix K, calculate the rotation matrix R and translation vector t of the current key frame relative to the adjacent key frame to obtain the camera's extrinsic parameter matrix [R|t]; According to the shooting time of the key frame image, the position and posture constraints of each camera are recorded into a trajectory file in the form of [time, R, t]; Record the correspondence between each image name and the corresponding camera's intrinsic parameter matrix K and extrinsic parameter matrix [R|t], and save the optimal values ​​of each camera's intrinsic parameter matrix K and extrinsic parameter matrix [R|t].

3. The method according to claim 1, characterized in that In the step 2, for each key frame of the target object, the 3D coordinates of the feature points of the target object are obtained; and the position change information of the feature points of the target object in the video is stored.

Images