3D commodity implantation video algorithm and system based on three-dimensional reconstruction and rendering

Abstract

The invention discloses a 3D commodity implantation video algorithm and system based on three-dimensional reconstruction and rendering, and the method comprises the steps: 1, enabling a 3D commodity model to unfold the surface of the 3D model to a two-dimensional plane through a UV unfolding technology, so as to create a UV coordinate for the 3D model, and generating a seamless UV layout; after UV deployment is completed, surface texture and material information can be mapped and applied through the UV coordinates. And then, extracting the surface color, concave-convex degree and reflection material parameters from the 3D model, and storing the information as a 2D texture image for subsequent mapping. The invention relates to the technical field of computer vision, and has the beneficial effects that the system combines the 3D model of the commodity and a three-dimensional reconstruction technology, and the system has a good application prospect. Users are allowed to customize their 3D assets and seamlessly integrate them into any video without any 3D rendering knowledge. The invention aims to simplify a video editing process and improve efficiency and usability.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and in particular to a 3D commodity implantation video algorithm and system based on three-dimensional reconstruction and rendering. Background Art

[0002] In recent years, the video production industry for product placement has been facing multiple pressures, including a sharp decline in production costs and employee benefits, as well as the loss of talent caused by the rise of foreign video platforms. Due to the strong entry of foreign platforms, local video production companies must consider how to more effectively recover their imaging investments to reduce risks and seek innovative monetization models. However, due to tight cash flow, their cooperation with production companies has become more cautious and funds are limited.

[0003] At the same time, traditional TV stations and advertisers are also in trouble. As TV ratings continue to decline, the old advertising model is becoming increasingly ineffective. Young audiences have long mastered the skills of skipping ads through video recording devices, making it impossible to effectively convey ads. In addition, although online ads try to fill this gap, viewers are increasingly disgusted with online ads, and even use ad blocking software to avoid ads. This trend is becoming more and more common among young audiences. This makes both traditional and emerging media face huge challenges in how to break through the barriers of advertising communication and attract and capture the hearts of audiences.

[0004] In the field of video editing and processing, film production, commercial shooting and virtual reality content creation all involve a lot of resource investment, which makes the process of acquiring or re-capturing video footage often expensive and time-consuming. Especially for non-professionals, it is a very difficult task to seamlessly insert virtual objects into existing videos to create realistic visual effects, which is complex and technically demanding.

[0005] With the rapid development of virtual reality, augmented reality and social media content creation, the market demand for solutions to simplify the video editing process has become more urgent. Today's users want to be able to easily add various objects to videos to enhance visual appeal and interactivity, thereby improving the quality of content and viewing experience.

[0006] However, to achieve this goal, multiple technical and operational challenges must be overcome. First, the spatial consistency between virtual objects and real scenes is crucial. To ensure that the inserted objects perfectly match the characters and environment in the video, factors such as lighting, angles, and shadows must be considered. The second is temporal consistency, that is, the existence of the object in the video must conform to the timeline and dynamic changes of the film, and cannot be abrupt or break the original fluency.

[0007] This series of complex factors has led the ad placement video production industry to a critical point of transformation. Content holders, production companies, advertisers and platforms must work together to find innovative and sustainable models to cope with the increasingly complex market environment and achieve their respective business goals. In this context, cooperation and innovation within the industry have become more important than ever.

[0008] In order to solve many problems in the field of video content production, we need to establish an effective mechanism to enable more funds to flow reasonably to video content production companies and create more value for advertisers and viewers. To this end, we have developed a new technology based on 3D product placement in advertising videos, and verified it with advanced artificial intelligence technology developed independently. This innovation will not only enable funds to be better allocated to content production companies, but also provide advertisers with accurate and efficient promotion channels, while bringing a more natural and interesting viewing experience to viewers.

[0009] Through this new technology, we are committed to optimizing the interests of all parties: content producers can obtain more stable and substantial income, thereby improving the quality of their creations; advertisers can expand brand exposure through more influential and immersive advertising formats; and viewers can enjoy higher-quality film and television content without being disturbed by commercial advertisements.

[0010] To this end, our 3D product video advertising placement system came into being, which combines 3D models of products and three-dimensional reconstruction technology to allow users to customize their 3D assets and integrate them seamlessly into any video without any knowledge of 3D rendering. The system simplifies the video editing process, making it more efficient and easier to use, while providing high-quality visual output. As a result, users can easily incorporate their imagination or objects from their surroundings into any video creation process. Through this AI-driven product placement platform, our company aims to connect various companies and solve the problems of video content-related industries, and increase revenue by digitally placing products in works after video production or after television screening / public release. This unique advertising method not only attracts more traffic, but also enhances advertising effectiveness by showcasing actual work or products through character props or backgrounds in movies and TV series. Summary of the invention

[0011] The purpose of the present invention is to provide a 3D product video advertising implantation system that combines a 3D model of the product with three-dimensional reconstruction technology, allowing users to customize their 3D assets and seamlessly integrate them into any video without any knowledge of 3D rendering. It is designed to simplify the video editing process, improve efficiency and ease of use, while providing high-quality visual output. By utilizing this AI-driven platform, the present invention hopes to connect various companies and solve difficult problems in video content-related industries, especially after the video is produced or after the TV / movie is shown, through digital means to implant products, thereby increasing revenue. This unique advertising method can attract more traffic and enhance the advertising effect by displaying in character props or backgrounds in movies and TV series.

[0012] The technical solution of the present invention to achieve the above-mentioned purpose is a 3D product placement video algorithm and system based on 3D reconstruction and rendering, comprising the following steps:

[0013] Step 1: Use UV unfolding technology to unfold the surface of the 3D product model onto a two-dimensional plane to create UV coordinates for it, thereby generating a seamless UV layout. After UV unfolding, the surface texture and material information can be mapped and applied through these UV coordinates. Next, extract the color, bumpiness, and reflective material parameters of the surface from the 3D model, and save this information as a 2D texture image for subsequent mapping;

[0014] Step 2: Using the deep learning-based video 3D reconstruction method, the inserted video is subjected to three steps of data collection, feature extraction, and optimization of internal parameters. The camera internal parameters are adjusted by the gradient descent method to minimize the reprojection error, and finally the camera internal parameters are estimated.

[0015] Step 3: Combine the intrinsic parameters of the camera calculated in step 2 with the motion changes of consecutive frames in the video to be inserted, and use the deep learning-based video 3D reconstruction method to estimate the extrinsic parameters of the camera;

[0016] Step 4: Based on the camera internal and external parameters obtained in steps 2 and 3, a deep learning-based method is used to perform 3D reconstruction. According to the estimated pose and depth information, the depth map of the current frame is converted into a 3D point cloud. The point cloud of the current frame is fused with the global point cloud to generate more complete and denser point cloud data. Finally, these sparse point cloud data provide a basis for 3D reconstruction.

[0017] Step 5: Convert the points in the two-dimensional plane coordinate system into coordinates in three-dimensional space to correspond to the depth information and spatial position relationship;

[0018] Step 6: Use the RANSAC plane estimation function to calculate the plane normal vector of the point to be implanted in the three-dimensional space, so as to constrain the direction of the object to be implanted, ensure that it is perpendicular to the plane of the contact point, avoid flipping or tilting, and obtain a more natural visual effect;

[0019] Step 7: Convert the camera coordinate system in the deep learning-based video 3D reconstruction method to the coordinate system under the PyTorch3D framework so as to perform operations and calculations on the rotation matrix and translation vector under this framework;

[0020] Step 8: In the PyTorch3D coordinate system, the movement angle and displacement of the camera in each frame of the video to be embedded are used to calculate the movement angle and displacement of the 3D model in the current frame to accurately depict the dynamic changes of the object.

[0021] Step 9: Combine the texture map obtained in step 1 and use the PyTorch3D framework to perform multi-view rendering to generate a 3D model image with high realism;

[0022] Step 10: Use style transfer technology to make the brightness of the 3D model closer to the style of the original video and seamlessly blend with the environment, thereby improving overall visual consistency;

[0023] Step 11: The user selects the location where the target object needs to be implanted in the operation interface, which will serve as the basic positioning point of the first frame in the subsequent insertion operation;

[0024] Step 12: Use the optical flow estimation method to calculate the pixel position of the product in the next frame so that it can dynamically follow the actual background movement of the video;

[0025] Step 13: Based on the 3D model extracted in step 1, use the plane normal vector in step 6 to constrain its direction, and combine the motion prediction position obtained in step 12, the 3D model pose in step 8 and the style transfer result in step 10 to perform multi-level video overlay processing, and finally obtain a natural product implantation video effect.

[0026] Preferably, the step 2 is specifically:

[0027] The estimation of camera intrinsic parameters is achieved through a deep learning-based video 3D reconstruction method. The camera intrinsic parameters are fixed after leaving the factory and will not change during use. The camera intrinsic parameters are mainly related to the camera imaging process, that is, the process of the camera mapping the coordinate points in the 3D world to the 2D image plane, including the focal length and the optical center coordinates. The derivation of the camera intrinsic parameters is closely integrated with the deep learning network in this system and optimized during the training process through the loss function. The specific steps include:

[0028] S1: Initialize the intrinsic parameters: initialize the camera intrinsic parameters according to the physical characteristics of the camera or use approximate values;

[0029] S2: Training deep neural network: extracting disparity image and depth map through convolutional neural network;

[0030] S3: Error back propagation: Use the objective function to evaluate the deviation of the current output depth and disparity from the true value. This error is back-propagated to update the network parameters and camera intrinsic parameters.

[0031] S4: Sequence optimization: The system simultaneously considers the visual error information of consecutive frames within a certain period of time in the time series to further optimize the camera pose and intrinsic parameters. Through continuous iterative training, the system is able to optimize the camera's intrinsic parameters to make them as accurate as possible.

[0032] Preferably, the step 3 is specifically:

[0033] The estimation of camera extrinsic parameters is achieved through a deep learning-based video 3D reconstruction method. It describes the projection relationship from world coordinates to pixel coordinates, including the camera's pose, i.e., position and translation. Compared with the unchanged intrinsic parameters, the extrinsic parameters change with the camera's motion. They are also the target to be estimated in the deep learning-based video 3D reconstruction method. The position and pose of the camera in the environment are estimated through deep learning and optimization techniques. The derivation is carried out through the following steps:

[0034] S1: Feature extraction: Use the encoder to capture and extract high-level features from the input image and generate a dense feature map with a resolution of 1 / 8 of the input image. These high-level features include the shape, structure, and semantic information of the object.

[0035] S2: Constructing a correlation pyramid: The correlation pyramid is a data structure used to calculate the correlation between image features in the deep learning-based video 3D reconstruction method system. It plays a core role in the feature extraction and matching process. Its main purpose is to capture and compare the similarities between image features at different scales, thereby helping to estimate the motion and depth information between images. For each pair of frames (i, j) in the frame map, a 4D correlation volume is constructed by calculating the dot product of all feature vectors in the two frames. This volume captures the similarity information between the features of the two frames. The specific formula is:

[0036]

[0037] S3: Average pooling: By performing an average pooling operation on the last two dimensions of the correlation volume, this pooling operation helps improve the system's ability to adapt to image changes, especially when dealing with fast-moving or dynamic scenes;

[0038] S4: Feature matching and correspondence estimation: Through the lookup operator, it uses a grid with radius r to index on the correlation volume and retrieve the feature points most relevant to the current pixel position, thereby estimating a dense correspondence field. This dense correspondence field provides rich information for subsequent camera pose and depth estimation;

[0039] S5: Pose Estimation: The core component of the deep learning based video 3D reconstruction method is a learned update operator, which is a 3*3 convolutional gated recurrent unit with a hidden state. Each application of the operator updates the hidden state and also generates a pose update, which is applied to the current pose estimate by contraction and vector addition on the 3D special Euclidean group manifold respectively;

[0040] S6: Loop closure detection: The purpose of loop closure detection is to identify repeated locations, thereby correcting the accumulated pose errors and ensuring the accuracy and consistency of mapping. The deep learning-based video 3D reconstruction method adopts a cyclic optimization strategy, continuously updating the pose graph and bundling adjustments when new frames are added, ensuring that the system maintains high accuracy during long-term operation.

[0041] Preferably, the step 4 is specifically:

[0042] The conversion of points in a two-dimensional plane coordinate system into coordinates in a three-dimensional space is achieved through a video three-dimensional reconstruction method based on deep learning. It can generate a high-precision three-dimensional point cloud from an input image sequence and realize three-dimensional reconstruction of the environment. Specifically, three-dimensional reconstruction and point cloud generation are performed through the following steps:

[0043] S1: Representation of depth information: Inverse depth parameterization is used to represent depth information. Inverse depth refers to the reciprocal of depth. Specifically, the unit of depth is distance, such as meter, and inverse depth is 1 / depth, with a unit of 1 / meter. Inverse depth is a way to represent the distance of an object. In some cases, it is more convenient than using depth directly, especially in the process of optimization and calculation. It can bring the advantage of numerical stability in some algorithms, which helps the numerical stability of the optimization process;

[0044] S2: Depth Estimation: The depth map is the basis of 3D reconstruction. Each update of the depth map is generated by a differentiable dense bundle adjustment layer, which calculates a dense and deep Gauss-Newton update for each pixel to maximize its compatibility with the current optical flow estimate. This dense bundle adjustment layer takes advantage of geometric constraints to improve accuracy and robustness. In addition, the dense bundle adjustment layer directly optimizes the depth of each pixel without being hindered by the depth basis, allowing the network to better generalize to new data sets because the depth representation is not bound to the training data set;

[0045] S3: Pose estimation: camera extrinsics obtained in step 3;

[0046] S4: Camera intrinsic parameters: Camera intrinsic parameters obtained in step 2;

[0047] S5: 3D point cloud generation: Using the estimated depth information and camera pose, the depth value of each pixel is converted into a point in 3D space. The specific formula is:

[0048]

[0049] in, is a point in three-dimensional space, is the depth value, is the camera intrinsic parameter matrix, is the camera extrinsic matrix, is the homogeneous representation of pixel coordinates;

[0050] S6: Point cloud fusion: fuse the 3D point clouds of different frames together to form a global point cloud map. Use the pose graph optimization result to transform the point cloud of each frame into the global coordinate system.

[0051] S7: Post-processing: Filter and denoise the generated point cloud to remove isolated points and noise.

[0052] Preferably, the step 6 is specifically:

[0053] By constraining the normal vector, ensure that the 3D model is placed upright to avoid flipping or tilting. Constrain the plane normal vector through the following steps:

[0054] S1: Projecting 3D points: According to step 5, obtaining the projection points of the points in the original data into the three-dimensional space;

[0055] S2: Estimate plane function using RANSAC: Apply the RANSAC algorithm to estimate a plane function on these projected 3D points;

[0056] S3: Constrained normal vector: estimated plane normal vector Apply constraints to ensure the model is upright;

[0057] S4: Get the camera position and a point on the plane: Determine the camera position and a point on the plane

[0058] S5: Verify normal vector direction: Check the normal vector With vector (From the camera position Point to a point on a plane Is the dot product of the vector of ) greater than 0, that is ;

[0059] S6: Adjust the direction of the normal vector: If the dot product is not greater than 0, adjust the direction of the normal vector to meet the conditions.

[0060] Preferably, the step 7 is specifically:

[0061] The conversion between the camera coordinate system and the pytorch3D coordinate system and the calculation of the motion angle and displacement of the 3D model in each frame in step 8 are implemented together. In the deep learning-based video 3D reconstruction method, several different coordinate systems are involved, and they represent the camera posture differently. Therefore, the camera orientation parameters need to be adapted and converted.

[0062] S1: Find out the orientation of the camera coordinate system used in Pytorch3D and deep learning-based video 3D reconstruction methods. Pytorch3D uses a right-handed coordinate system, with the camera's Z axis facing forward, X axis facing left, and Y axis facing up. However, the camera orientation in the camera pose obtained by the deep learning-based video 3D reconstruction method is Z axis facing forward, X axis facing right, and Y axis facing down;

[0063] S2: Adapt and transform the camera’s orientation parameters. According to the rotation matrix and translation vector obtained from the deep learning-based video 3D reconstruction method, adjust the transformation matrix format required by PyTorch3D, invert the 0th and 1st columns of the rotation matrix and keep the column order unchanged, keep the 2nd column as it is, and then horizontally splice these columns into a new rotation matrix, leaving the translation vector unchanged;

[0064] S3: Concatenate the new matrix, concatenate the new rotation matrix and translation vector to obtain a new transformation matrix, and then import it into PyTorch3D for use.

[0065] Preferably, the step 9 is specifically as follows:

[0066] Multi-view rendering is implemented through PyTorch3D, which allows users to render the same 3D scene from different camera angles to generate multi-view images. The specific process includes importing 3D models, setting up cameras, lighting and renderers, and then performing rendering operations. The key points include:

[0067] S1: Load 3D model;

[0068] S2: Set the camera view. You can use the rotation matrices and translation vectors of multiple viewpoints obtained in steps 7 and 8, or generate a camera pose by setting the distance between the camera and the origin, as well as the pitch angle and azimuth angle so that the camera axis points to the origin.

[0069] S3: Set the light source, define the position and properties of the light source, including diffuse light that affects the brightness and color of the object surface, and specular light that forms highlights on the physical surface, and then create a light source object;

[0070] S4: Define rasterization and shading configuration, including configuring rasterization parameters such as image size, blur radius, and number of faces per pixel;

[0071] S5: Build the renderer, assemble the camera, light source, rasterization settings and shader together, and create a MeshRenderer object;

[0072] S6: Rendering an image, using a renderer to render the 3D model and generate a planar image under a specified viewing angle and motion posture;

[0073] S7: Display or save the result.

[0074] Preferably, the step 10 is specifically as follows:

[0075] The style transfer technique is to adjust the color of the inserted object by inserting it into the scene, specifically through color correction and pixel blurring, so as to adapt to the background lighting. The specific steps are as follows:

[0076] S1: Select a style template and use the complete background as the style template;

[0077] S2: Feature extraction: Use the pre-trained convolutional neural network to extract the feature representation of the product's 2D texture image and target style image;

[0078] S3: Feature transformation: whiten and recolor the feature representations of the two-dimensional texture image of the product and the background image. The whitening process converts the feature representation of the two-dimensional texture image of the product into a zero-mean unit variance form. The recoloring process uses the feature representation of the background image to recolor the whitened two-dimensional texture image features of the product to match the mean and covariance of the background image.

[0079] S4: Generate output image: The processed feature representation is converted back to a set of images with similar brightness to the background image through a deconvolution layer or other reconstruction method.

[0080] S5: Adjust the parameter alpha to control the degree of style transfer, ranging from 0 to 1.

[0081] Preferably, the step 12 is specifically as follows:

[0082] The optical flow estimation in step 12 is a method for describing the movement of pixels between images over time. As time passes, the same pixel will move in the image. According to the grayscale invariance assumption:

[0083]

[0084] Its movement process can be tracked, and the optical flow method can be used to track the position of the implant point in the video 3D reconstruction method based on deep learning.

[0085] The optical flow method specifically includes the following steps:

[0086] S1: Extract features, using deep learning methods to extract features of each frame image in the video;

[0087] S2: Feature point matching, finding points in adjacent frames that correspond to feature points in the initial frame;

[0088] S3: Calculate the optical flow, calculate the optical flow information between two consecutive frames, and use bilinear interpolation to accurately obtain the offset of the floating point number, that is, the position change of the feature point;

[0089] S4: Update position: update the position of the feature point in the next frame according to the calculated optical flow.

[0090] A 3D product placement video algorithm and system based on 3D reconstruction and rendering is produced using the technical solution of the present invention. The system combines the 3D model of the product with 3D reconstruction technology, allowing users to customize their 3D assets and seamlessly integrate them into any video without any 3D rendering knowledge. It aims to simplify the video editing process, improve efficiency and ease of use, while providing high-quality visual output. By utilizing this AI-driven platform, the present invention hopes to connect various companies and solve difficult problems in video content-related industries, especially after the video is produced or after the TV / movie is shown, product placement is performed through digital means to increase revenue. This unique advertising method can attract more traffic and enhance advertising effects by displaying in character props or backgrounds in movies and TV series. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 It is a flow chart of a 3D product placement video algorithm and system based on three-dimensional reconstruction and rendering according to the present invention;

[0092] Figure 2 It is a flowchart of the position estimation and map generation of a 3D product placement video algorithm and system based on three-dimensional reconstruction and rendering according to the present invention;

[0093] Figure 3 It is a flowchart of Pytorch3D multi-view rendering of a 3D product placement video algorithm and system based on three-dimensional reconstruction and rendering described in the present invention;

[0094] Figure 4It is a flow chart of a 3D commodity placement video algorithm based on 3D reconstruction and rendering and a system's optical flow method for calculating the position relationship between adjacent frames according to the present invention;

[0095] Figure 5 It is the definition of the coordinate system in the deep learning-based video 3D reconstruction method of the 3D product implantation video algorithm and system based on 3D reconstruction and rendering described in the present invention.

[0096] Figure 6 It is the definition of the orientation of the camera coordinate system of the 3D product implantation video algorithm and system based on three-dimensional reconstruction and rendering described in the present invention.

[0097] Figure 7 This is the definition of the optical flow method of the 3D product implantation video algorithm and system based on three-dimensional reconstruction and rendering described in the present invention.

[0098] Figure 8 This is the definition of bilinear interpolation of a 3D commodity implantation video algorithm and system based on three-dimensional reconstruction and rendering described in the present invention. DETAILED DESCRIPTION

[0099] The present invention will be described in detail below in conjunction with the accompanying drawings. Figure 1-8 As shown, a 3D product placement video algorithm and system based on 3D reconstruction and rendering specifically includes the following steps:

[0100] Step 1: Use UV unfolding technology to unfold the surface of a 3D product model (such as GLB, FBX model) onto a two-dimensional plane in order to create UV coordinates for it, thereby generating a seamless UV layout. After UV unfolding, surface texture and material information can be mapped and applied through these UV coordinates. Next, extract the color, bumpiness, and reflective material parameters of the surface from the 3D model, and save this information as a 2D texture image for subsequent mapping.

[0101] Step 2: Use the deep learning-based video 3D reconstruction method to perform three steps on the inserted video: data collection (including data with rich textures), feature extraction (optical flow estimation method), and optimization of internal parameters. Use the gradient descent method to adjust the camera internal parameters to minimize the reprojection error, and finally estimate the camera's internal parameters (including focal length, optical center coordinates, and distortion coefficient).

[0102] Step 3: Combine the intrinsic parameters of the camera calculated in step 2 with the motion changes of consecutive frames in the video to be inserted, and use the deep learning-based video 3D reconstruction method to estimate the extrinsic parameters of the camera (such as posture, rotation matrix, and translation vector).

[0103] Step 4: Based on the camera internal and external parameters obtained in steps 2 and 3, perform 3D reconstruction using a deep learning-based method. Convert the depth map of the current frame into a 3D point cloud based on the estimated pose and depth information. Fuse the point cloud of the current frame with the global point cloud to generate more complete and denser point cloud data. Ultimately, these sparse point cloud data provide a basis for 3D reconstruction.

[0104] Step 5: Convert the points in the two-dimensional plane coordinate system into coordinates in three-dimensional space to correspond to the depth information and spatial position relationship.

[0105] Step 6: Use RANSAC to estimate the plane function and calculate the plane normal vector of the point to be implanted in the three-dimensional space, so as to constrain the direction of the object to be implanted, ensure that it is perpendicular to the plane of the contact point, avoid flipping or tilting, and obtain a more natural visual effect.

[0106] Step 7: Convert the camera coordinate system in the deep learning-based video 3D reconstruction method to the coordinate system under the PyTorch3D framework so that the rotation matrix and translation vector operations can be performed under this framework.

[0107] Step 8: In the PyTorch3D coordinate system, the movement angle and displacement of the camera in each frame of the video to be embedded are used to calculate the movement angle and displacement of the 3D model in the current frame to accurately depict the dynamic changes of the object.

[0108] Step 9: Combined with the texture map obtained in step 1, use the PyTorch3D framework to perform multi-view rendering to generate a more realistic 3D model image.

[0109] Step 10: Through style transfer technology, the brightness and darkness of the 3D model are made closer to the style of the original video and seamlessly integrated with the environment, thereby improving the overall visual consistency.

[0110] Step 11: The user selects the location where the target object needs to be implanted in the operation interface, which will serve as the basic positioning point of the first frame in the subsequent insertion operation.

[0111] Step 12: Use the optical flow estimation method to calculate the pixel position of the product in the next frame so that it can dynamically follow the actual background movement of the video.

[0112] Step 13: Based on the 3D model extracted in step 1, use the plane normal vector in step 6 to constrain its direction, and combine the motion prediction position obtained in step 12, the 3D model pose in step 8 and the style transfer result in step 10 to perform multi-level video overlay processing, and finally obtain a natural product implantation video effect.

[0113] The estimation of the camera intrinsic parameters in step 2 is achieved through a deep learning-based video 3D reconstruction method. The camera's intrinsic parameters are fixed after leaving the factory and will not change during use. The camera's intrinsic parameters are mainly related to the camera's imaging process, that is, the process by which the camera maps coordinate points in the 3D world (in meters) to the 2D image plane (in pixels), including focal length and optical center coordinates. The derivation of the camera's intrinsic parameters is closely integrated with the deep learning network in this system and optimized during the training process through the loss function. The specific steps include:

[0114] S1: Initialize the intrinsic parameters: initialize the camera intrinsic parameters according to the physical characteristics of the camera or use approximate values;

[0115] S2: Training deep neural network: extracting disparity image and depth map through convolutional neural network;

[0116] S3: Error back propagation: Use the objective function to evaluate the deviation of the current output depth and disparity from the true value. This error is back-propagated to update the network parameters and camera intrinsic parameters.

[0117] S4: Sequence Optimization: Considering the visual error information of consecutive frames within a certain period of time in the time series, the camera pose and internal parameters are further optimized. Through continuous iteration training, the system is able to optimize the camera's internal parameters to make them as accurate as possible.

[0118] The estimation of the camera extrinsic parameters in step 3 is achieved through a deep learning-based video 3D reconstruction method. It describes the projection relationship from world coordinates to pixel coordinates, including the camera's pose, i.e., position and translation. Compared with the constant intrinsic parameters, the extrinsic parameters change with the camera movement, and are also the targets to be estimated in the deep learning-based video 3D reconstruction method. The position and pose of the camera in the environment are estimated through deep learning and optimization techniques. The derivation is specifically carried out through the following steps:

[0119] S1: Feature extraction: Use the encoder to capture and extract high-level features from the input image and generate a dense feature map with a resolution of 1 / 8 of the input image. These high-level features include the shape, structure, and semantic information of the object.

[0120] S2: Constructing a correlation pyramid: The correlation pyramid is a data structure used to calculate the correlation between image features in a deep learning-based video 3D reconstruction method system. It plays a core role in the feature extraction and matching process. Its main purpose is to capture and compare the similarities between image features at different scales, thereby helping to estimate motion and depth information between images. For each pair of frames (i, j) in the frame map, a 4D correlation volume is constructed by calculating the dot product of all feature vectors in the two frames. This volume captures the similarity information between the features of the two frames. The specific formula is:

[0121]

[0122] S3: Average pooling: Average pooling is performed on the last two dimensions of the correlation volume. This pooling operation helps improve the system's ability to adapt to image changes, especially when dealing with fast motion or dynamic scenes.

[0123] S4: Feature matching and correspondence estimation: Through the lookup operator, it uses a grid of radius r to index on the correlation volume and retrieve the feature points that are most relevant to the current pixel position, thereby estimating a dense correspondence field. This dense correspondence field provides rich information for subsequent camera pose and depth estimation.

[0124] S5: Pose Estimation: The core component of the deep learning-based video 3D reconstruction method is a learned update operator, which is a 3*3 convolutional gated recurrent unit with a hidden state. Each application of the operator updates the hidden state and also generates a pose update, which is applied to the current pose estimate by contraction and vector addition on the 3D special Euclidean group manifold, respectively.

[0125] S6: Loop detection: The purpose of loop detection is to identify repeated locations, thereby correcting the accumulated pose errors and ensuring the accuracy and consistency of the map. The deep learning-based video 3D reconstruction method adopts a cyclic optimization strategy to continuously update the pose graph and bundle adjustments when new frames are added, ensuring that the system maintains high accuracy during long-term operation.

[0126] The 3D reconstruction in step 4 and the conversion of points in the 2D plane coordinate system into coordinates in 3D space in step 5 are achieved by a video 3D reconstruction method based on deep learning. It can generate a high-precision 3D point cloud from the input image sequence and realize the 3D reconstruction of the environment. Specifically, the 3D reconstruction and point cloud generation are performed through the following steps.

[0127] S1: Representation of depth information: Inverse depth parameterization is used to represent depth information. Inverse depth is the reciprocal of depth. Specifically, the unit of depth is distance, such as meters, and the inverse depth is 1 / depth, with a unit of 1 / meter. Inverse depth is a way to represent the distance of an object, which is more convenient than using depth directly in some cases, especially in the process of optimization and calculation, it can bring numerical stability advantages in some algorithms. This helps numerical stability in the optimization process.

[0128] S2: Depth Estimation: The depth map is the basis for 3D reconstruction. Each update of the depth map is produced by a differentiable dense bundle adjustment layer, which computes a dense and deep Gauss-Newton update for each pixel to maximize its compatibility with the current optical flow estimate. This dense bundle adjustment layer exploits geometric constraints and improves accuracy and robustness. In addition, the dense bundle adjustment layer directly optimizes the depth of each pixel without being hindered by the depth basis, allowing the network to better generalize to new datasets because the depth representation is not bound to the training dataset.

[0129] S3: Pose estimation: camera extrinsics obtained in step 3;

[0130] S4: Camera intrinsic parameters: Camera intrinsic parameters obtained in step 2;

[0131] S5: 3D point cloud generation: Using the estimated depth information and camera pose, the depth value of each pixel is converted into a point in 3D space. The specific formula is:

[0132]

[0133] in, is a point in three-dimensional space, is the depth value, is the camera intrinsic parameter matrix, is the camera extrinsic matrix, is the homogeneous representation of pixel coordinates;

[0134] S6: Point cloud fusion: fuse the 3D point clouds of different frames together to form a global point cloud map. Use the pose graph optimization results to transform the point cloud of each frame into the global coordinate system;

[0135] S7: Post-processing: Filter and denoise the generated point cloud to remove isolated points and noise.

[0136] Step 6: Ensure that the 3D model is placed upright by constraining the normal vector to avoid flipping or tilting. Constrain the plane normal vector by following the steps below:

[0137] S1: Projecting 3D points: According to step 5, obtaining the projection points of the points in the original data into the three-dimensional space;

[0138] S2: Estimate plane function using RANSAC: Apply the RANSAC algorithm to estimate a plane function on these projected 3D points;

[0139] S3: Constrained normal vector: estimated plane normal vector Apply constraints to ensure the model is upright;

[0140] S4: Get the camera position and a point on the plane: Determine the camera position and a point on the plane ;

[0141] S5: Verify normal vector direction: Check the normal vector With vector (From the camera position Point to a point on a plane Is the dot product of the vector of ) greater than 0, that is ;

[0142] S6: Adjust the direction of the normal vector: If the dot product is not greater than 0, adjust the direction of the normal vector to meet the conditions.

[0143] The conversion between the camera coordinate system and the pytorch3D coordinate system in step 7 and the calculation of the motion angle and displacement of the 3D model in each frame in step 8 are implemented together. In the deep learning-based video 3D reconstruction method, several different coordinate systems are involved, and they represent the camera posture differently, so the camera orientation parameters need to be adapted and converted. The specific steps for the conversion between the camera coordinate system and the pytorch3D coordinate system are as follows: Figure 5 shown.

[0144] S1: Find out the orientation of the camera coordinate system used in Pytorch3D and deep learning based video 3D reconstruction methods. Figure 6 The last column shows that Pytorch3D uses a right-handed coordinate system, with the camera's Z axis facing forward, X axis facing left, and Y axis facing up (LUF). However, the camera pose obtained by the deep learning-based video 3D reconstruction method has the camera's Z axis facing forward, X axis facing right, and Y axis facing down (RDF).

[0145] S2: Adapt and transform the camera's orientation parameters. According to the rotation matrix and translation vector obtained from the deep learning-based video 3D reconstruction method, adjust the transformation matrix format required by PyTorch3D, invert the 0th and 1st columns of the rotation matrix and keep the column order unchanged, keep the 2nd column as it is, and then splice these columns horizontally into a new rotation matrix, and keep the translation vector unchanged;

[0146] S3: Concatenate the new matrix. Concatenate the new rotation matrix and translation vector to obtain a new transformation matrix (4x4 homogeneous transformation matrix), and then import it into PyTorch3D for use.

[0147] The multi-view rendering in step 9 is implemented by PyTorch3D. PyTorch3D allows users to render the same 3D scene from different camera angles, thereby generating multi-view images. The specific process includes importing 3D models, setting up cameras, lighting, and renderers, and then performing rendering operations. The key points include:

[0148] S1: Load 3D model.

[0149] S2: Set the camera view. You can use the rotation matrices and translation vectors of multiple viewpoints obtained in steps 7 and 8, or generate a camera pose by setting the distance between the camera and the origin, as well as the pitch angle and azimuth angle so that the camera axis points to the origin.

[0150] S3: Set up the light source. Define the position and properties of the light source, including diffuse light that affects the brightness and color of the surface of the object, and specular light that forms highlights on the physical surface, and then create a light source object;

[0151] S4: Define rasterization and shading configuration. Specifically, it includes configuring rasterization parameters, such as image size, blur radius, and number of faces per pixel;

[0152] S5: Build the renderer. Assemble the camera, light source, rasterization settings and shader together to create a MeshRenderer object;

[0153] S6: Rendering an image. Use a renderer to render the 3D model and generate a planar image under a specified viewing angle and motion posture;

[0154] S7: Display or save the result.

[0155] The style transfer technique in step 10 is to adjust the color of the inserted object by inserting it into the scene, specifically by color correction and pixel blurring, so as to adapt to the background lighting. The specific steps are as follows:

[0156] S1: Select a style template. Use the full background as the style template;

[0157] S2: Feature extraction: Use a pre-trained convolutional neural network to extract feature representations of the product’s 2D texture image and target style image (background image).

[0158] S3: Feature transformation: whiten and recolor the feature representations of the product's 2D texture image and background image. The whitening process converts the feature representation of the product's 2D texture image into a zero-mean unit variance form. The recoloring process uses the feature representation of the background image to recolor the whitened 2D texture image features of the product to match the mean and covariance of the background image.

[0159] S4: Generate output images: The processed feature representation is converted back into a set of images with similar brightness to the background image through a deconvolution layer or other reconstruction method.

[0160] S5: Adjust the parameter alpha to control the degree of style transfer (ranging from 0 to 1).

[0161] The optical flow estimation in step 12 is a method to describe the movement of pixels between images over time. Figure 7 As shown in the figure, the same pixel will move in the image over time. According to the grayscale invariance assumption:

[0162]

[0163] Its movement process can be tracked. The optical flow method can be used to track the position of the implant point in the deep learning-based video 3D reconstruction method.

[0164] The optical flow method specifically includes the following steps:

[0165] S1: Extract features. Use deep learning methods to extract features of each frame in the video;

[0166] S2: Feature point matching. Find points in adjacent frames that correspond to feature points in the initial frame;

[0167] S3: Calculate optical flow. Calculate the optical flow information between two consecutive frames using bilinear interpolation. Figure 8 Accurately obtain the floating point offset, that is, the position change of the feature point.

[0168] S4: Update position. Update the position of the feature point in the next frame based on the calculated optical flow.

[0169] The above technical solutions only reflect the preferred technical solutions of the technical solutions of the present invention. Some changes that may be made to certain parts thereof by technicians in this technical field all reflect the principles of the present invention and fall within the protection scope of the present invention.

Claims

1. A 3D product placement video algorithm and system based on 3D reconstruction and rendering, comprising the following steps: Step 1: Use UV unfolding technology to unfold the surface of the 3D product model onto a two-dimensional plane to create UV coordinates for it, thereby generating a seamless UV layout. After UV unfolding, the surface texture and material information can be mapped and applied through these UV coordinates. Next, extract the color, bumpiness, and reflective material parameters of the surface from the 3D model, and save this information as a 2D texture image for subsequent mapping; Step 2: Using the deep learning-based video 3D reconstruction method, the inserted video is subjected to three steps of data collection, feature extraction, and optimization of internal parameters. The camera internal parameters are adjusted by the gradient descent method to minimize the reprojection error, and finally the camera internal parameters are estimated. Step 3: Combine the intrinsic parameters of the camera calculated in step 2 with the motion changes of consecutive frames in the video to be inserted, and use the deep learning-based video 3D reconstruction method to estimate the extrinsic parameters of the camera; Step 4: Based on the camera internal and external parameters obtained in steps 2 and 3, a deep learning-based method is used to perform 3D reconstruction. According to the estimated pose and depth information, the depth map of the current frame is converted into a 3D point cloud. The point cloud of the current frame is fused with the global point cloud to generate more complete and denser point cloud data. Finally, these sparse point cloud data provide a basis for 3D reconstruction. Step 5: Convert the points in the two-dimensional plane coordinate system into coordinates in three-dimensional space to correspond to the depth information and spatial position relationship; Step 6: Use the RANSAC plane estimation function to calculate the plane normal vector of the point to be implanted in the three-dimensional space, so as to constrain the direction of the object to be implanted, ensure that it is perpendicular to the plane of the contact point, avoid flipping or tilting, and obtain a more natural visual effect; Step 7: Convert the camera coordinate system in the deep learning-based video 3D reconstruction method to the coordinate system under the PyTorch3D framework so as to perform operations and calculations on the rotation matrix and translation vector under this framework; Step 8: In the PyTorch3D coordinate system, the movement angle and displacement of the camera in each frame of the video to be embedded are used to calculate the movement angle and displacement of the 3D model in the current frame to accurately depict the dynamic changes of the object. Step 9: Combine the texture map obtained in step 1 and use the PyTorch3D framework to perform multi-view rendering to generate a 3D model image with high realism; Step 10: Use style transfer technology to make the brightness of the 3D model closer to the style of the original video and seamlessly blend with the environment, thereby improving overall visual consistency; Step 11: The user selects the location where the target object needs to be implanted in the operation interface, which will serve as the basic positioning point of the first frame in the subsequent insertion operation; Step 12: Use the optical flow estimation method to calculate the pixel position of the product in the next frame so that it can dynamically follow the actual background movement of the video; Step 13: Based on the 3D model extracted in step 1, use the plane normal vector in step 6 to constrain its direction, and combine the motion prediction position obtained in step 12, the 3D model pose in step 8 and the style transfer result in step 10 to perform multi-level video overlay processing, and finally obtain a natural product implantation video effect.

2. The 3D product placement video algorithm and system based on 3D reconstruction and rendering according to claim 1, characterized in that: The step 2 is specifically as follows: The estimation of camera intrinsic parameters is achieved through a deep learning-based video 3D reconstruction method. The camera intrinsic parameters are fixed after leaving the factory and will not change during use. The camera intrinsic parameters are mainly related to the camera imaging process, that is, the process of the camera mapping the coordinate points in the 3D world to the 2D image plane, including the focal length and the optical center coordinates. The derivation of the camera intrinsic parameters is closely integrated with the deep learning network in this system and optimized during the training process through the loss function. The specific steps include: S1: Initialize the intrinsic parameters: initialize the camera intrinsic parameters according to the physical characteristics of the camera or use approximate values; S2: Training deep neural network: extracting disparity image and depth map through convolutional neural network; S3: Error back propagation: Use the objective function to evaluate the deviation of the current output depth and disparity from the true value. This error is back-propagated to update the network parameters and camera intrinsic parameters. S4: Sequence optimization: The visual error information of consecutive frames within a certain period of time in the time series is considered simultaneously to further optimize the camera pose and intrinsic parameters. Through continuous iterative training, the system is able to optimize the camera's intrinsic parameters to make them as accurate as possible.

3. The 3D product placement video algorithm and system based on 3D reconstruction and rendering according to claim 1, characterized in that: The step 3 is specifically as follows: The estimation of camera extrinsic parameters is achieved through a deep learning-based video 3D reconstruction method. It describes the projection relationship from world coordinates to pixel coordinates, including the camera's pose, i.e., position and translation. Compared with the unchanged intrinsic parameters, the extrinsic parameters change with the camera's motion. They are also the target to be estimated in the deep learning-based video 3D reconstruction method. The position and pose of the camera in the environment are estimated through deep learning and optimization techniques. The derivation is carried out through the following steps: S1: Feature extraction: Use the encoder to capture and extract high-level features from the input image and generate a dense feature map with a resolution of 1 / 8 of the input image. These high-level features include the shape, structure, and semantic information of the object. S2: Constructing a correlation pyramid: The correlation pyramid is a data structure used to calculate the correlation between image features in the deep learning-based video 3D reconstruction method system. It plays a core role in the feature extraction and matching process. Its main purpose is to capture and compare the similarities between image features at different scales, thereby helping to estimate the motion and depth information between images. For each pair of frames (i, j) in the frame map, a 4D correlation volume is constructed by calculating the dot product of all feature vectors in the two frames. This volume captures the similarity information between the features of the two frames. The specific formula is: ； S3: Average pooling: Average pooling is performed on the last two dimensions of the correlation volume. This pooling operation helps improve the system's ability to adapt to image changes, especially when dealing with fast-moving or dynamic scenes. S4: Feature matching and correspondence estimation: Through the lookup operator, it uses a grid with radius r to index on the correlation volume and retrieve the feature points most relevant to the current pixel position, thereby estimating a dense correspondence field. This dense correspondence field provides rich information for subsequent camera pose and depth estimation; S5: Pose Estimation: The core component of the deep learning based video 3D reconstruction method is a learned update operator, which is a 3*3 convolutional gated recurrent unit with a hidden state. Each application of the operator updates the hidden state and also generates a pose update, which is applied to the current pose estimate by contraction and vector addition on the 3D special Euclidean group manifold respectively; S6: Loop closure detection: The purpose of loop closure detection is to identify repeated locations, thereby correcting the accumulated pose errors and ensuring the accuracy and consistency of mapping. The deep learning-based video 3D reconstruction method adopts a cyclic optimization strategy, continuously updating the pose graph and bundling adjustments when new frames are added, ensuring that the system maintains high accuracy during long-term operation.

4. The 3D product placement video algorithm and system based on 3D reconstruction and rendering according to claim 1, characterized in that: The step 4 is specifically as follows: The conversion of points in a two-dimensional plane coordinate system into coordinates in a three-dimensional space is achieved through a video three-dimensional reconstruction method based on deep learning. It can generate a high-precision three-dimensional point cloud from an input image sequence and realize three-dimensional reconstruction of the environment. Specifically, three-dimensional reconstruction and point cloud generation are performed through the following steps: S1: Representation of depth information: Inverse depth parameterization is used to represent depth information. Inverse depth refers to the reciprocal of depth. Specifically, the unit of depth is distance, such as meter, and inverse depth is 1 / depth, with a unit of 1 / meter. Inverse depth is a way to represent the distance of an object. In some cases, it is more convenient than using depth directly, especially in the process of optimization and calculation. It can bring the advantage of numerical stability in some algorithms, which helps the numerical stability of the optimization process; S2: Depth Estimation: The depth map is the basis of 3D reconstruction. Each update of the depth map is generated by a differentiable dense bundle adjustment layer, which calculates a dense and deep Gauss-Newton update for each pixel to maximize its compatibility with the current optical flow estimate. This dense bundle adjustment layer takes advantage of geometric constraints to improve accuracy and robustness. In addition, the dense bundle adjustment layer directly optimizes the depth of each pixel without being hindered by the depth basis, allowing the network to better generalize to new datasets because the depth representation is not bound to the training dataset; S3: Pose estimation: camera extrinsics obtained in step 3; S4: Camera intrinsic parameters: Camera intrinsic parameters obtained in step 2; S5: 3D point cloud generation: Using the estimated depth information and camera pose, the depth value of each pixel is converted into a point in 3D space. The specific formula is: ； in, is a point in three-dimensional space, is the depth value, is the camera intrinsic parameter matrix, is the camera extrinsic matrix, is the homogeneous representation of pixel coordinates; S6: Point cloud fusion: fuse the 3D point clouds of different frames together to form a global point cloud map. Use the pose graph optimization result to transform the point cloud of each frame into the global coordinate system. S7: Post-processing: Filter and denoise the generated point cloud to remove isolated points and noise.

5. The 3D product placement video algorithm and system based on 3D reconstruction and rendering according to claim 1, characterized in that: The step 6 is specifically as follows: By constraining the normal vector, ensure that the 3D model is placed upright to avoid flipping or tilting. Constrain the plane normal vector through the following steps: S1: Projecting 3D points: According to step 5, obtaining the projection points of the points in the original data into the three-dimensional space; S2: Estimate plane function using RANSAC: Apply the RANSAC algorithm to estimate a plane function on these projected 3D points; S3: Constrained normal vector: estimated plane normal vector Apply constraints to ensure the model is upright; S4: Get the camera position and a point on the plane: Determine the camera position A point on a plane S5: Verify normal vector direction: Check the normal vector With vector Is the dot product greater than 0, that is ; S6: Adjust the direction of the normal vector: If the dot product is not greater than 0, adjust the direction of the normal vector to meet the conditions.

6. The 3D product placement video algorithm and system based on 3D reconstruction and rendering according to claim 1, characterized in that: The step 7 is specifically as follows: The conversion between the camera coordinate system and the pytorch3D coordinate system and the calculation of the motion angle and displacement of the 3D model in each frame in step 8 are implemented together. In the deep learning-based video 3D reconstruction method, several different coordinate systems are involved, and they represent the camera posture differently. Therefore, the camera orientation parameters need to be adapted and converted. S1: Find out the orientation of the camera coordinate system used in Pytorch3D and deep learning-based video 3D reconstruction methods. Pytorch3D uses a right-handed coordinate system, with the camera's Z axis facing forward, X axis facing left, and Y axis facing up. However, the camera orientation in the camera pose obtained by the deep learning-based video 3D reconstruction method is Z axis facing forward, X axis facing right, and Y axis facing down. S2: Adapt and transform the camera’s orientation parameters. According to the rotation matrix and translation vector obtained from the deep learning-based video 3D reconstruction method, adjust the transformation matrix format required by PyTorch3D, invert the 0th and 1st columns of the rotation matrix and keep the column order unchanged, keep the 2nd column as it is, and then horizontally splice these columns into a new rotation matrix, leaving the translation vector unchanged; S3: Concatenate the new matrix, concatenate the new rotation matrix and translation vector to obtain a new transformation matrix, and then import it into PyTorch3D for use.

7. The 3D product placement video algorithm and system based on 3D reconstruction and rendering according to claim 1, characterized in that: The step 9 is specifically as follows: Multi-view rendering is implemented through PyTorch3D, which allows users to render the same 3D scene from different camera angles to generate multi-view images. The specific process includes importing 3D models, setting up cameras, lighting and renderers, and then performing rendering operations. The key points include: S1: Load 3D model; S2: Set the camera view. You can use the rotation matrices and translation vectors of multiple viewpoints obtained in steps 7 and 8, or generate a camera pose by setting the distance between the camera and the origin, as well as the pitch angle and azimuth angle so that the camera axis points to the origin. S3: Set the light source, define the position and properties of the light source, including diffuse light that affects the brightness and color of the object surface, and specular light that forms highlights on the physical surface, and then create a light source object; S4: Define rasterization and shading configuration, including configuring rasterization parameters such as image size, blur radius, and number of faces per pixel; S5: Build the renderer, assemble the camera, light source, rasterization settings and shader together, and create a MeshRenderer object; S6: Rendering an image, using a renderer to render the 3D model and generate a planar image under a specified viewing angle and motion posture; S7: Display or save the result.

8. The 3D product placement video algorithm and system based on 3D reconstruction and rendering according to claim 1, characterized in that: The step 10 is specifically as follows: The style transfer technique is to adjust the color of the inserted object by inserting it into the scene, specifically through color correction and pixel blurring, so as to adapt to the background lighting. The specific steps are as follows: S1: Select a style template and use the complete background as the style template; S2: Feature extraction: Use the pre-trained convolutional neural network to extract the feature representation of the product's 2D texture image and target style image; S3: Feature transformation: whiten and recolor the feature representations of the two-dimensional texture image of the product and the background image. The whitening process converts the feature representation of the two-dimensional texture image of the product into a zero-mean unit variance form. The recoloring process uses the feature representation of the background image to recolor the whitened two-dimensional texture image features of the product to match the mean and covariance of the background image. S4: Generate output image: The processed feature representation is converted back to a set of images with similar brightness to the background image through a deconvolution layer or other reconstruction method. S5: Adjust the parameter alpha to control the degree of style transfer, ranging from 0 to 1.

9. The 3D product placement video algorithm and system based on 3D reconstruction and rendering according to claim 1, characterized in that: The step 12 is specifically as follows: The optical flow estimation in step 12 is a method for describing the movement of pixels between images over time. As time passes, the same pixel will move in the image. According to the grayscale invariance assumption: ； Its movement process can be tracked, and the optical flow method can be used to track the position of the implant point in the video 3D reconstruction method based on deep learning. The optical flow method specifically includes the following steps: S1: Extract features, using deep learning methods to extract features of each frame image in the video; S2: Feature point matching, finding points in adjacent frames that correspond to feature points in the initial frame; S3: Calculate the optical flow, calculate the optical flow information between two consecutive frames, and use bilinear interpolation to accurately obtain the offset of the floating point number, that is, the position change of the feature point; S4: Update position: update the position of the feature point in the next frame according to the calculated optical flow.

Images