Three-dimensional video fusion method and application thereof in video security system

Abstract

This application relates to the field of video processing technology and discloses a three-dimensional video fusion method and its application in a video security system. The method includes: processing static and dynamic scenes to construct a three-dimensional scene model; obtaining the optical flow field corresponding to each layer of the image pyramid and obtaining multi-scale optical flow information; matching the image feature points of the video frame with the corresponding image feature points of the three-dimensional scene model to generate feature point matching results; finding the correspondence between the video frame and the three-dimensional scene model through the feature point matching results, providing accurate positional information for subsequent fusion; calculating fusion weights based on the feature point matching results; and processing the video according to the fusion weights to obtain the fused three-dimensional video. In a video security system, the fused three-dimensional video can provide more comprehensive and accurate scene information, helping to improve the accuracy and real-time performance of security monitoring and meeting the high reliability requirements of video security systems.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a three-dimensional video fusion method and its application in video security systems. Background Technology

[0002] In today's digital age, the demand for security is growing rapidly. Video security systems, as a crucial line of defense for protecting people and property, are undeniably important. With the continuous expansion of cities, the increasing complexity of public places, and the diversification of criminal methods, video security systems integrating multiple key technologies such as video surveillance, alarms, access control, IP intercom, power and environmental data collection, patrol management, and fire protection have become a vital safeguard. However, traditional video security systems face severe challenges in data processing. The diverse sources of data collected at the underlying level result in extremely complex data, which also exhibits significant heterogeneity, leading to numerous difficulties in data integration and analysis. A large amount of valuable data is buried in the complex information and cannot be efficiently utilized. Furthermore, the two-dimensional video surveillance used in existing video security systems suffers from a single perspective and susceptibility to interference from complex environments. When faced with changes in lighting or obstructions, blind spots and missing information easily occur, making it impossible to accurately monitor target areas and failing to meet the current requirements for high reliability and accuracy in security systems.

[0003] To address these issues, existing common video fusion methods mainly focus on stitching and overlaying two-dimensional video images. While these methods can expand the monitoring perspective to some extent, they cannot construct monitoring scenes with depth and a three-dimensional feel. They are not accurate enough in judging the spatial position and movement trajectory of objects, lack in-depth consideration of scene details and environmental factors, and are difficult to play an effective role in complex security scenarios, thus failing to provide continuous and reliable support for security systems. Summary of the Invention

[0004] This application provides a three-dimensional video fusion method and its application in video security systems. Through high-precision three-dimensional fusion of multi-scale optical flow, it achieves the high reliability requirements of security and improves the high precision and real-time performance of security monitoring.

[0005] In a first aspect, this application provides a three-dimensional video fusion method, which includes: acquiring a multi-angle video sequence of a target scene, the video sequence including several video frames; when the target scene is a static scene, analyzing the video sequence through a structured light three-dimensional reconstruction module to obtain a static three-dimensional mesh model; when the target scene is a dynamic scene, analyzing the video sequence through a motion recovery structure to obtain a dynamic three-dimensional mesh model; constructing a three-dimensional scene model based on the static three-dimensional mesh model and the dynamic three-dimensional mesh model; constructing an image pyramid for each video frame; obtaining the optical flow field corresponding to each layer of the image pyramid through an optical flow algorithm; fusing the optical flow field through a multi-scale convolutional neural network to obtain multi-scale optical flow information; extracting corresponding image feature points from the video frames and the three-dimensional scene model and combining them with the multi-scale optical flow information; matching the image feature points of the video frames with the image feature points of the three-dimensional scene model to generate feature point matching results; calculating fusion weights based on the feature point matching results; and performing weighted fusion of the video frames and the three-dimensional scene model based on the fusion weights to obtain a fused three-dimensional video.

[0006] Secondly, this application provides an application of a three-dimensional video fusion method in a video security system. The video security system includes: an acquisition module for acquiring multi-angle video sequences of a target scene, the video sequence comprising several video frames; a construction module for analyzing the video sequence and obtaining a static three-dimensional mesh model using a structured light three-dimensional reconstruction module when the target scene is static, and for analyzing the video sequence and obtaining a dynamic three-dimensional mesh model using a motion recovery structure when the target scene is dynamic; the construction module is used to construct a three-dimensional scene model based on the static and dynamic three-dimensional mesh models; and a multi-scale optical flow module for processing each... The system constructs an image pyramid from the video frames, obtains the optical flow field corresponding to each layer of the image pyramid using an optical flow algorithm, and fuses the optical flow fields using a multi-scale convolutional neural network to obtain multi-scale optical flow information. A feature point matching module extracts corresponding image feature points from the video frames and the 3D scene model, and combines them with the multi-scale optical flow information to match the image feature points of the video frames with the image feature points of the 3D scene model to generate feature point matching results. A fusion module calculates fusion weights based on the feature point matching results, and performs weighted fusion of the video frames and the 3D scene model according to the fusion weights to obtain a fused 3D video.

[0007] In the technical solution provided in this application, after acquiring a multi-angle video sequence of a target scene, a three-dimensional scene model is constructed by processing and combining the static and dynamic scenes in the target scene separately. The static scene in the target scene is analyzed using a structured light 3D reconstruction module to obtain a static 3D mesh model from the video sequence. The dynamic scene in the target scene is analyzed using a motion recovery structure to obtain a dynamic 3D mesh model from the video sequence. A 3D scene model is then constructed based on the static and dynamic 3D mesh models, resulting in a complete 3D scene model containing both static and dynamic elements. This provides a unified 3D spatial framework for subsequent video fusion. Furthermore, an image pyramid is constructed for each video frame. The optical flow field corresponding to each layer of the image pyramid is obtained using an optical flow algorithm. The optical flow field is then fused using a multi-scale convolutional neural network to obtain multi-scale optical flow information. Images at different layers can capture objects of different sizes and motion details, which helps improve the accuracy of optical flow calculation. The fused multi-scale optical flow information provides a more reliable basis for subsequent feature point matching. Then, from the... Image feature points are extracted from the video frame and the 3D scene model, and combined with the multi-scale optical flow information. The image feature points of the video frame are matched with the image feature points of the 3D scene model to generate feature point matching results. Feature point extraction can reduce the amount of data and help improve the efficiency of subsequent matching. The correspondence between the video frame and the 3D scene model can be found through the feature point matching results, providing accurate positional information for subsequent fusion. Finally, the fusion weight is calculated based on the feature point matching results, and the video frame and the 3D scene model are weighted and fused according to the fusion weight to obtain the fused 3D video. The fusion weight calculated based on the feature point matching results not only reflects the importance of the video frame and the 3D scene model at different positions, but also more reasonably allocates the contributions of the video frame and the 3D scene model, making the fused image more natural and realistic. The fused 3D video presents a richer and more realistic visual effect. In video security systems, it can provide more comprehensive and accurate scene information, which helps to improve the accuracy and real-time performance of security monitoring and meets the high reliability requirements of video security systems. Attached Figure Description

[0008] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a schematic diagram of one embodiment of the three-dimensional video fusion method in this application.

[0010] Figure 2 This is a schematic diagram of an embodiment of a video security system, illustrating the application of the three-dimensional video fusion method in this application. Detailed Implementation

[0011] This application provides a three-dimensional video fusion method and its application in a video security system. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0012] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 One embodiment of the three-dimensional video fusion method in this application includes:

[0013] Step S101: Obtain a multi-angle video sequence of the target scene, wherein the video sequence includes several video frames;

[0014] Step S102: When the target scene is a static scene, the video sequence is analyzed by the structured light 3D reconstruction module to obtain a static 3D mesh model. When the target scene is a dynamic scene, the video sequence is analyzed by the motion recovery structure to obtain a dynamic 3D mesh model. A 3D scene model is constructed based on the static 3D mesh model and the dynamic 3D mesh model.

[0015] Step S103: Construct an image pyramid for each video frame, obtain the optical flow field corresponding to each layer of the image pyramid using an optical flow algorithm, and fuse the optical flow field using a multi-scale convolutional neural network to obtain multi-scale optical flow information.

[0016] Step S104: Extract corresponding image feature points from the video frame and the three-dimensional scene model, and combine them with the multi-scale optical flow information to match the image feature points of the video frame with the image feature points of the three-dimensional scene model to generate feature point matching results;

[0017] Step S105: Calculate the fusion weight based on the feature point matching result, and perform weighted fusion on the video frame and the three-dimensional scene model according to the fusion weight to obtain the fused three-dimensional video.

[0018] It is understood that the entity executing this application can be a video security system, a terminal, or a server; no specific limitation is made here.

[0019] In one specific embodiment, acquiring multi-angle video sequences of the target scene is the initial step of the entire 3D video fusion method, providing basic data for subsequent 3D reconstruction and fusion. To acquire multi-angle information of the target scene, video acquisition equipment needs to be rationally arranged, with cameras distributed at different positions and angles according to the shape and characteristics of the scene. Appropriate acquisition parameters, such as frame rate, exposure time, and gain, are set based on factors such as the lighting conditions and motion speed of the target scene. For fast-moving scenes, the frame rate needs to be increased to capture clear motion trajectories; for dimly lit scenes, the exposure time or gain needs to be appropriately increased, but care must be taken to avoid overexposure or excessive noise in the images.

[0020] During the acquisition process, it is essential to ensure stable camera operation and avoid video shaking caused by vibration or other interference. At the same time, it is necessary to record information such as the time, location, and scene description of the acquisition for subsequent processing and analysis. The acquired video data should be stored in a suitable storage device, such as a hard drive array or cloud storage, ensuring that the storage device has sufficient capacity and reliable performance to prevent data loss or damage.

[0021] In one specific embodiment, the structured light 3D reconstruction module projects a structured light pattern with coded information, such as sinusoidal stripes, onto a static target scene using a specific projection device. In each frame of the video frame, the camera captures the reflected structured light pattern. Through image preprocessing steps, including noise reduction and contrast enhancement, preparation is made for the subsequent accurate extraction of structured light coded information. Finally, the structured light coded information is extracted from the preprocessed image using a decoding algorithm. After the structured light coded information is extracted, the absolute phase value of each pixel in the image is calculated using a phase unrolling algorithm. Taking sinusoidal stripe projection as an example, the initial value obtained is the wrapped phase value. Through the phase unrolling algorithm, the wrapped phase is unrolled into the absolute phase to obtain continuous and accurate phase information.

[0022] Based on the calculated absolute phase value of each pixel and the calibration parameters of the structured light, the three-dimensional spatial coordinates of each pixel are calculated using the principle of triangulation. Triangulation is based on the propagation and projection relationship of light rays, and uses the known position information of the camera and projector to map the pixels on the two-dimensional image into three-dimensional space. The three-dimensional spatial coordinates of all pixels are combined to form point cloud data. The point cloud data is a collection of a large number of discrete three-dimensional points, each point representing information about a location in the scene.

[0023] The Poisson surface reconstruction algorithm is used to process the generated point cloud data to generate a static 3D mesh model. The Poisson surface reconstruction algorithm treats the point cloud data as a directed set of sampling points and estimates an implicit surface function by solving the Poisson equation, thereby generating a continuous 3D mesh model. It can effectively handle noise and incompleteness in point cloud data and generate a smooth and continuous surface.

[0024] For dynamic scenes, the Structure for Motion Reconstruction (SMR) algorithm is used to process video frames. First, feature extraction algorithms are used to extract feature points from each frame. These feature points have unique local features and can be stably detected across different image frames. Then, the K-nearest neighbor algorithm is used to match feature points from adjacent video frames. The K-nearest neighbor algorithm finds the K nearest neighbors of each feature point in adjacent frames by calculating the distance between feature points. Reliable matching point pairs are selected based on a distance threshold. Based on the matched feature point pairs, combined with camera calibration parameters and multi-view geometry principles, the 3D coordinates of the feature points are calculated using triangulation. Triangulation utilizes the projection relationships of feature points from different viewpoints to solve for their positions in 3D space.

[0025] The calculated 3D coordinates of the feature points are combined to form a sparse point cloud. Since the sparse point cloud only contains 3D information of some points with obvious features in the image, a multi-view stereo network algorithm is used to further generate a dense point cloud to obtain a more complete 3D model. The multi-view stereo network algorithm utilizes image information from multiple perspectives and uses techniques such as stereo matching to supplement more 3D points on the basis of the sparse point cloud, thus obtaining denser and more complete point cloud data. Similarly, the Poisson surface reconstruction algorithm is used to process the dense point cloud to generate a dynamic 3D mesh model. Poisson surface reconstruction can convert discrete point cloud data into a continuous 3D mesh, making the model have better visualization effects and easier subsequent processing.

[0026] The Iterative Closest Point Algorithm (ICP) aligns the static and dynamic 3D mesh models to the same coordinate system. The ICP algorithm iteratively searches for the optimal rotation and translation transformation between the two models, minimizing the sum of distances between corresponding points. The specific steps are as follows:

[0027] Choose an initial rotation matrix and translation vector;

[0028] For each vertex in the static 3D mesh model, find the vertex closest to it in the dynamic 3D mesh model to form a corresponding point pair;

[0029] Based on the current corresponding point pairs, use the least squares method to calculate the rotation matrix and translation vector that can make the two models closer;

[0030] Transform one of the models using the calculated rotation matrix and translation vector;

[0031] Repeat the above steps until the convergence condition is met.

[0032] The iterative nearest point algorithm is used again to match the boundary vertices of the static and dynamic 3D mesh models. By finding corresponding vertices on the boundaries of the two models, the algorithm provides a basis for subsequent triangulation and merging operations.

[0033] Delaunay triangulation is performed on the matched boundary vertices to generate triangular patches in the stitching region. Delaunay triangulation ensures that the generated triangular patches are as close as possible to equilateral triangles, avoiding the formation of elongated triangles, thereby improving the quality of the mesh. The boundary vertices of the static 3D mesh model and the dynamic 3D mesh model are merged, and duplicate vertices are removed to ensure mesh consistency.

[0034] The texture coordinates of the static 3D mesh model and the dynamic 3D mesh model after merging the boundary vertices are unified. Since the two models may have different texture mapping methods, the texture coordinates need to be adjusted so that they can correctly map the texture after merging. Poisson image editing technology is used to transition the textures of the static 3D mesh model and the dynamic 3D mesh model. Poisson image editing solves the Poisson equation to smoothly transition the textures of the two models, eliminate obvious boundaries at the texture splicing, and make the texture look more natural.

[0035] By connecting the static 3D mesh model and the dynamic 3D mesh model after connecting the generated triangular facets with the transition texture, a complete 3D scene model is constructed. During the connection process, the topological structure of the mesh is ensured to be correct and the connection relationship between vertices and edges is reasonable, thus obtaining a high-quality 3D scene model containing both static and dynamic elements.

[0036] Image pyramids are a method for representing images at multiple scales. By performing Gaussian blur and downsampling operations on the original image, a series of image layers with different resolutions are generated, just like a pyramid, with the resolution decreasing as you go up. This multi-scale representation helps to capture the features and motion information of the image at different scales, improving the accuracy and robustness of subsequent optical flow calculations and feature fusion.

[0037] A Gaussian filter is applied to the input video frame image. The Gaussian filter replaces each pixel value in the image with a weighted average of its neighboring pixels through a convolution operation. The weights are determined by a Gaussian distribution. The kernel size and standard deviation of the Gaussian filter are important parameters, usually chosen based on the image characteristics and application requirements. For example, for images with a lot of noise, a larger kernel size and standard deviation can be chosen to enhance the smoothing effect. A downsampling operation is then performed on the Gaussian-blurred image, typically using an interlaced sampling method, that is, only retaining pixels from every other row and every other column, thus reducing the image resolution by half. This Gaussian blurring and downsampling operation is repeated on the first layer image to obtain the second layer image. The second layer image is then subjected to Gaussian blurring and downsampling operations again to obtain the third layer image.

[0038] Optical flow refers to the trend of pixel grayscale changes caused by the motion of objects in an image. The optical flow field describes the motion speed and direction of each pixel in the image. By calculating the optical flow field of image layers with different resolutions, motion information of objects can be captured at different scales, providing a foundation for subsequent multi-scale feature fusion.

[0039] Optical flow fields corresponding to the first, second, and third layer images are calculated using an optical flow algorithm. Optical flow fields at different resolutions contain motion information at different scales, but their resolutions are different and cannot be directly fused. They need to be upsampled to the same resolution first, and then their respective features are extracted using convolutional kernels of different sizes. Finally, a dynamic weighted fusion is performed using the squeezing and activation (SE) module of a multi-scale convolutional neural network to generate multi-scale optical flow information with consistent image features. Interpolation methods are used to upsample the optical flow fields corresponding to the second and third layer images to make their resolution the same as the optical flow field corresponding to the first layer image. For the aligned optical flow fields, convolution operations are performed using convolutional kernels of different sizes to extract image features. The squeezing and activation (SE) module dynamically weights and fuses the three feature maps. The SE module adaptively learns the importance weights of each channel through global average pooling, fully connected layers, and the sigmoid activation function, and then applies these weights to the feature maps.

[0040] By processing video frames using the Scale Invariant Feature Transform (SIFT) algorithm, a set of feature points is obtained through scale space extremum detection, feature point localization, orientation assignment, and feature descriptor generation. First, the 3D scene model is projected into a 2D image based on camera parameters. Then, the SIFT algorithm is used to extract the feature point set from the projected image. Combined with multi-scale optical flow information, the positions of the video frame feature points in the projected image are predicted, narrowing the matching search range. The KNN algorithm is used to calculate the feature descriptor distance, filtering out reliable matching pairs. The RANSAC algorithm is used to calculate the homography matrix, and multiple iterations are performed for optimization. Inlier points are filtered out based on projection error. The verified matching pairs are the final feature point matching results. Accurate and reliable feature point matching results provide crucial correspondences for subsequent 3D video fusion. Through these matching pairs, the spatial positional relationship between the video frame and the 3D scene model can be determined, thereby achieving accurate fusion and generating high-quality 3D video. For example, in security monitoring systems, real-time video can be fused with a 3D scene model, providing more comprehensive and accurate scene information and improving the accuracy and real-time performance of monitoring.

[0041] Gaussian difference operation is performed on video frame images to construct a scale space. Local extrema are searched in the scale space, and these points may be candidate points for feature points. The location and scale of feature points are accurately determined by fitting a three-dimensional quadratic function. At the same time, points with low contrast and edge response are removed to improve the stability of feature points. One or more principal directions are assigned to each feature point. Based on the gradient direction histogram in the neighborhood of the feature point, a neighborhood is selected around the feature point and divided into multiple sub-regions. The gradient direction histogram in each sub-region is calculated. These histograms are combined into a high-dimensional vector as the feature descriptor of the feature point.

[0042] The 3D scene model is projected onto the viewpoint of the video frame to generate a projected image. Then, the SIFT algorithm is used to extract feature points in the projected image, so as to achieve feature matching between the video frame and the 3D scene model under the same viewpoint. Specifically, according to the camera parameters of the video frame, the 3D scene model is projected onto a 2D plane to generate a projected image. The SIFT algorithm is used to extract feature points from the projected image. The steps are the same as those for extracting feature points from the video frame, and will not be repeated here.

[0043] Multi-scale optical flow information can predict the motion trajectory of feature points between different frames, narrowing the search range for feature point matching and improving matching accuracy and efficiency. The K-Nearest Neighbors (KNN) algorithm calculates the distance between feature point descriptors to find the K nearest neighbors, thereby determining the matching pair. Specifically, based on multi-scale optical flow information, the possible positions of feature points in the video frame in the next frame are predicted. The KNN algorithm is used to match the feature point set of the video frame with the feature point set of the 3D scene model. Typically, K=2 is chosen, meaning that the two nearest neighbors are found for each feature point. By comparing the distance ratio between the nearest and second nearest neighbors, reliable matching pairs are selected. If the distance ratio between the nearest and second nearest neighbors is less than a certain threshold (e.g., 0.8), the matching pair is considered reliable.

[0044] The homography matrix describes the projection transformation relationship between two planes. If a matching pair is correct, then they satisfy the homography transformation. By calculating the homography matrix and verifying the consistency of the matching pairs, mismatched points can be removed. Specifically, the Random Sample Consensus (RANSAC) algorithm is used to calculate the homography matrix from the matching pairs. The RANSAC algorithm randomly selects a set of matching pairs, calculates the homography matrix, and then verifies the consistency of other matching pairs based on the matrix. This process is iterated until the optimal homography matrix is ​​found. Based on the calculated homography matrix, each matching pair is verified. If the projection error of the matching pair is less than a certain threshold, the matching pair is considered to be consistent.

[0045] The confidence score of a matching pair can be calculated based on factors such as the matching distance between feature point descriptors and the projection error of the homography matrix. The smaller the matching distance and the smaller the projection error, the more reliable the matching pair is, and the higher the confidence score is. For each matching pair, the distance between its feature point descriptors is calculated, the distance is normalized, and then the reciprocal is taken as the matching distance score. The projection error of each matching pair is calculated based on the homography matrix, the error is normalized, and then the reciprocal is taken as the projection error score. The matching distance score and the projection error score are weighted and averaged to obtain the confidence score of the matching pair.

[0046] Local weights consider the distribution of matching pairs within local regions, while global weights consider the overall situation of all matching pairs. By calculating local and global weights, the importance of matching pairs can be reflected more comprehensively. Specifically, the image is divided into multiple local regions, and the sum of the confidence scores of matching pairs within each local region is calculated and then normalized to obtain the local weights. Similarly, the sum of the confidence scores of all matching pairs is calculated and normalized to obtain the global weights. Different coefficients are assigned to the local and global weights to obtain the fusion weights. The selection of these coefficients can be adjusted according to the specific application scenario. The specific calculation formula is as follows:

[0047] W = α * w local +β*w glocal

[0048] Where W is the fusion weight, w local For the local weight, w glocal Let α be the global weight, β be the coefficient of the local weight, and β be the coefficient of the global weight, where α + β = 1.

[0049] At each pixel location in the image, the pixel values ​​of the video frame and the image in the 3D scene model are weighted and fused according to the fusion weights to obtain the fused pixel values. These fused pixel values ​​are then arranged into a sequence to form the fused 3D video. The fused pixel values ​​are calculated based on the pixel values ​​of the video frame and the image in the 3D scene model. The specific calculation formula is as follows:

[0050] I fused =W*I model +(1-W)*I video

[0051] Among them, I fused I represents the fused pixel value. video I represents the pixel value of the video frame. model Let W be the pixel value of the image in the 3D scene model, and W be the fusion weight. By traversing each pixel of the video frame and the image in the 3D scene model, the fused pixel value is calculated according to the fusion weight. The fused pixel values ​​are then arranged into a sequence to form the 3D video, which is saved as a video file using a video processing library.

[0052] In this embodiment, specifically, a three-dimensional scene model is constructed by separately processing and combining the static and dynamic scenes in the target scene. The static scene in the target scene is analyzed by a structured light 3D reconstruction module to obtain a static 3D mesh model from the video sequence. The dynamic scene in the target scene is analyzed by a motion recovery structure to obtain a dynamic 3D mesh model from the video sequence. A three-dimensional scene model is then constructed based on the static and dynamic 3D mesh models, resulting in a complete 3D scene model containing both static and dynamic elements. This provides a unified 3D spatial framework for subsequent video fusion. Furthermore, an image pyramid is constructed for each video frame. The optical flow field corresponding to each layer of the image pyramid is obtained using an optical flow algorithm. The optical flow field is then fused using a multi-scale convolutional neural network to obtain multi-scale optical flow information. Images at different layers can capture objects of different sizes and motion details, which helps improve the accuracy of optical flow calculation. The fused multi-scale optical flow information provides a more reliable basis for subsequent feature point matching. Then, the video frames and the 3D field... The corresponding image feature points are extracted from the scene model and combined with the multi-scale optical flow information. The image feature points of the video frame are matched with the image feature points of the 3D scene model to generate feature point matching results. Feature point extraction can reduce the amount of data and help improve the efficiency of subsequent matching. The correspondence between the video frame and the 3D scene model can be found through the feature point matching results, providing accurate positional information for subsequent fusion. Finally, the fusion weight is calculated based on the feature point matching results. The video frame and the 3D scene model are weighted and fused according to the fusion weight to obtain the fused 3D video. The fusion weight calculated based on the feature point matching results not only reflects the importance of the video frame and the 3D scene model at different positions, but also more reasonably allocates the contributions of the video frame and the 3D scene model, making the fused image more natural and realistic. The fused 3D video presents a richer and more realistic visual effect. In video security systems, it can provide more comprehensive and accurate scene information, which helps to improve the accuracy and real-time performance of security monitoring and meet the high reliability requirements of video security systems.

[0053] In one specific embodiment, acquiring multi-angle video sequences of the target scene is the initial step of the entire 3D video fusion method, providing basic data for subsequent 3D reconstruction and fusion. To acquire multi-angle information of the target scene, video acquisition equipment needs to be rationally arranged, with cameras distributed at different positions and angles according to the shape and characteristics of the scene. Appropriate acquisition parameters, such as frame rate, exposure time, and gain, are set based on factors such as the lighting conditions and motion speed of the target scene. For fast-moving scenes, the frame rate needs to be increased to capture clear motion trajectories; for dimly lit scenes, the exposure time or gain needs to be appropriately increased, but care must be taken to avoid overexposure or excessive noise in the images.

[0054] During the acquisition process, it is essential to ensure stable camera operation and avoid video shaking caused by vibration or other interference. At the same time, it is necessary to record information such as the time, location, and scene description of the acquisition for subsequent processing and analysis. The acquired video data should be stored in a suitable storage device, such as a hard drive array or cloud storage, ensuring that the storage device has sufficient capacity and reliable performance to prevent data loss or damage.

[0055] In one specific embodiment, the process of performing step S102 may specifically include the following steps:

[0056] (1) When the target scene is a static scene, the structured light 3D reconstruction module is used to extract the structured light coding information in each frame of the video frame, and the absolute phase value of each pixel of the image is calculated by the phase unfolding algorithm.

[0057] (2) Calculate the three-dimensional spatial coordinates corresponding to each pixel based on the absolute phase value, combine the three-dimensional spatial coordinates of the pixels into point cloud data, and use Poisson surface reconstruction to generate a static three-dimensional mesh model based on the point cloud data.

[0058] (3) When the target scene is a dynamic scene, the feature points of each frame of the video frame are extracted using the motion recovery structure and the feature points of adjacent video frames are matched using the K-nearest neighbor algorithm;

[0059] (4) Calculate the three-dimensional coordinates of the feature points based on the feature points, combine the three-dimensional coordinates of the feature points into a sparse point cloud, use a multi-view stereo network algorithm to generate a dense point cloud from the sparse point cloud, and use Poisson surface reconstruction to generate a dynamic three-dimensional mesh model from the dense point cloud.

[0060] (5) The static three-dimensional mesh model and the dynamic three-dimensional mesh model are aligned to the same coordinate system by the iterative nearest point algorithm, and the static three-dimensional mesh model and the dynamic three-dimensional mesh model are merged by the mesh stitching algorithm to construct the three-dimensional scene model.

[0061] Specifically, the step of using a mesh stitching algorithm to merge the static 3D mesh model and the dynamic 3D mesh model to construct the 3D scene model includes:

[0062] (1) Match the boundary vertices of the static 3D mesh model and the dynamic 3D mesh model using the iterative nearest point algorithm;

[0063] (2) Perform Delaunay triangulation on the matched boundary vertices to generate triangular patches of the stitching region, and merge the boundary vertices of the static three-dimensional mesh model and the dynamic three-dimensional mesh model.

[0064] (3) Unify the texture coordinates of the static 3D mesh model and the dynamic 3D mesh model after merging the boundary vertices, and use Poisson image editing to transition the textures of the static 3D mesh model and the dynamic 3D mesh model;

[0065] (4) The static three-dimensional mesh model after connecting the transition texture of the triangular facets and the mesh of the dynamic three-dimensional mesh model are used to construct the three-dimensional scene model.

[0066] In one specific embodiment, the structured light 3D reconstruction module projects a structured light pattern with coded information, such as sinusoidal stripes, onto a static target scene using a specific projection device. In each frame of the video frame, the camera captures the reflected structured light pattern. Through image preprocessing steps, including noise reduction and contrast enhancement, preparation is made for the subsequent accurate extraction of structured light coded information. Finally, the structured light coded information is extracted from the preprocessed image using a decoding algorithm. After the structured light coded information is extracted, the absolute phase value of each pixel in the image is calculated using a phase unrolling algorithm. Taking sinusoidal stripe projection as an example, the initial value obtained is the wrapped phase value. Through the phase unrolling algorithm, the wrapped phase is unrolled into the absolute phase to obtain continuous and accurate phase information.

[0067] Based on the calculated absolute phase value of each pixel and the calibration parameters of the structured light, the three-dimensional spatial coordinates of each pixel are calculated using the principle of triangulation. Triangulation is based on the propagation and projection relationship of light rays, and uses the known position information of the camera and projector to map the pixels on the two-dimensional image into three-dimensional space. The three-dimensional spatial coordinates of all pixels are combined to form point cloud data. The point cloud data is a collection of a large number of discrete three-dimensional points, each point representing information about a location in the scene.

[0068] The Poisson surface reconstruction algorithm is used to process the generated point cloud data to generate a static 3D mesh model. The Poisson surface reconstruction algorithm treats the point cloud data as a directed set of sampling points and estimates an implicit surface function by solving the Poisson equation, thereby generating a continuous 3D mesh model. It can effectively handle noise and incompleteness in point cloud data and generate a smooth and continuous surface.

[0069] For dynamic scenes, the Structure for Motion Reconstruction (SMR) algorithm is used to process video frames. First, feature extraction algorithms are used to extract feature points from each frame. These feature points have unique local features and can be stably detected across different image frames. Then, the K-nearest neighbor algorithm is used to match feature points from adjacent video frames. The K-nearest neighbor algorithm finds the K nearest neighbors of each feature point in adjacent frames by calculating the distance between feature points. Reliable matching point pairs are selected based on a distance threshold. Based on the matched feature point pairs, combined with camera calibration parameters and multi-view geometry principles, the 3D coordinates of the feature points are calculated using triangulation. Triangulation utilizes the projection relationships of feature points from different viewpoints to solve for their positions in 3D space.

[0070] The calculated 3D coordinates of the feature points are combined to form a sparse point cloud. Since the sparse point cloud only contains 3D information of some points with obvious features in the image, a multi-view stereo network algorithm is used to further generate a dense point cloud to obtain a more complete 3D model. The multi-view stereo network algorithm utilizes image information from multiple perspectives and uses techniques such as stereo matching to supplement more 3D points on the basis of the sparse point cloud, thus obtaining denser and more complete point cloud data. Similarly, the Poisson surface reconstruction algorithm is used to process the dense point cloud to generate a dynamic 3D mesh model. Poisson surface reconstruction can convert discrete point cloud data into a continuous 3D mesh, making the model have better visualization effects and easier subsequent processing.

[0071] The Iterative Closest Point Algorithm (ICP) aligns the static and dynamic 3D mesh models to the same coordinate system. The ICP algorithm iteratively searches for the optimal rotation and translation transformation between the two models, minimizing the sum of distances between corresponding points. The specific steps are as follows:

[0072] Choose an initial rotation matrix and translation vector;

[0073] For each vertex in the static 3D mesh model, find the vertex closest to it in the dynamic 3D mesh model to form a corresponding point pair;

[0074] Based on the current corresponding point pairs, use the least squares method to calculate the rotation matrix and translation vector that can make the two models closer;

[0075] Transform one of the models using the calculated rotation matrix and translation vector;

[0076] Repeat the above steps until the convergence condition is met.

[0077] The iterative nearest point algorithm is used again to match the boundary vertices of the static and dynamic 3D mesh models. By finding corresponding vertices on the boundaries of the two models, the algorithm provides a basis for subsequent triangulation and merging operations.

[0078] Delaunay triangulation is performed on the matched boundary vertices to generate triangular patches in the stitching region. Delaunay triangulation ensures that the generated triangular patches are as close as possible to equilateral triangles, avoiding the formation of elongated triangles, thereby improving the quality of the mesh. The boundary vertices of the static 3D mesh model and the dynamic 3D mesh model are merged, and duplicate vertices are removed to ensure mesh consistency.

[0079] The texture coordinates of the static 3D mesh model and the dynamic 3D mesh model after merging the boundary vertices are unified. Since the two models may have different texture mapping methods, the texture coordinates need to be adjusted so that they can correctly map the texture after merging. Poisson image editing technology is used to transition the textures of the static 3D mesh model and the dynamic 3D mesh model. Poisson image editing solves the Poisson equation to smoothly transition the textures of the two models, eliminate obvious boundaries at the texture splicing, and make the texture look more natural.

[0080] By connecting the static 3D mesh model and the dynamic 3D mesh model after connecting the generated triangular facets with the transition texture, a complete 3D scene model is constructed. During the connection process, the topological structure of the mesh is ensured to be correct and the connection relationship between vertices and edges is reasonable, thus obtaining a high-quality 3D scene model containing both static and dynamic elements.

[0081] In one specific embodiment, the process of executing step S103 may specifically include the following steps:

[0082] (1) Gaussian blur and downsampling are applied to the images of the video frames to generate multi-layer images, which form the image pyramid. The image pyramid includes at least a first layer image, a second layer image, and a third layer image. The resolution of the first layer image is greater than the resolution of the second layer image, and the resolution of the second layer image is greater than the resolution of the third layer image.

[0083] (2) The optical flow fields corresponding to the first layer image, the second layer image, and the third layer image are calculated respectively using the optical flow algorithm;

[0084] (3) Multi-scale optical flow information with consistent image features is generated by fusing the optical flow fields corresponding to the first layer image, the second layer image, and the third layer image through a multi-scale convolutional neural network.

[0085] Specifically, the image of the video frame is Gaussian blurred and downsampled to generate multi-layer images, which form the image pyramid. The image pyramid includes at least a first layer image, a second layer image, and a third layer image. The resolution of the first layer image is greater than the resolution of the second layer image, and the resolution of the second layer image is greater than the resolution of the third layer image.

[0086] The first layer image is obtained by performing Gaussian blur and downsampling on the image of the video frame;

[0087] The first layer image is subjected to Gaussian blur and downsampling to obtain the second layer image, and the resolution of the second layer image is lower than that of the first layer image;

[0088] The second layer image is subjected to Gaussian blur and downsampling to obtain the third layer image, the resolution of the third layer image being lower than that of the second layer image.

[0089] Specifically, the step of fusing the optical flow fields corresponding to the first layer image, the second layer image, and the third layer image using a multi-scale convolutional neural network to generate multi-scale optical flow information with consistent image features includes:

[0090] The optical flow fields corresponding to the first layer image, the second layer image, and the third layer image are upsampled and aligned to the same resolution.

[0091] The first feature map is obtained by extracting image features from the optical flow field corresponding to the aligned first layer image through 3×3 convolution; the second feature map is obtained by extracting image features from the optical flow field corresponding to the aligned second layer image through 5×5 convolution; and the third feature map is obtained by extracting image features from the optical flow field corresponding to the aligned third layer image through 7×7 convolution.

[0092] The first feature map, the second feature map, and the third feature map are dynamically weighted and fused through the squeezing and excitation module in a multi-scale convolutional neural network, and then 3×3 convolution is used to generate multi-scale optical flow information with consistent image features.

[0093] Image pyramids are a method for representing images at multiple scales. By performing Gaussian blur and downsampling operations on the original image, a series of image layers with different resolutions are generated, just like a pyramid, with the resolution decreasing as you go up. This multi-scale representation helps to capture the features and motion information of the image at different scales, improving the accuracy and robustness of subsequent optical flow calculations and feature fusion.

[0094] A Gaussian filter is applied to the input video frame image. The Gaussian filter replaces each pixel value in the image with a weighted average of its neighboring pixels through a convolution operation. The weights are determined by a Gaussian distribution. The kernel size and standard deviation of the Gaussian filter are important parameters, usually chosen based on the image characteristics and application requirements. For example, for images with a lot of noise, a larger kernel size and standard deviation can be chosen to enhance the smoothing effect. A downsampling operation is then performed on the Gaussian-blurred image, typically using an interlaced sampling method, that is, only retaining pixels from every other row and every other column, thus reducing the image resolution by half. This Gaussian blurring and downsampling operation is repeated on the first layer image to obtain the second layer image. The second layer image is then subjected to Gaussian blurring and downsampling operations again to obtain the third layer image.

[0095] Optical flow refers to the trend of pixel grayscale changes caused by the motion of objects in an image. The optical flow field describes the motion speed and direction of each pixel in the image. By calculating the optical flow field of image layers with different resolutions, motion information of objects can be captured at different scales, providing a foundation for subsequent multi-scale feature fusion.

[0096] Optical flow fields corresponding to the first, second, and third layer images are calculated using an optical flow algorithm. Optical flow fields at different resolutions contain motion information at different scales, but their resolutions are different and cannot be directly fused. They need to be upsampled to the same resolution first, and then their respective features are extracted using convolutional kernels of different sizes. Finally, a dynamic weighted fusion is performed using the squeezing and activation (SE) module of a multi-scale convolutional neural network to generate multi-scale optical flow information with consistent image features. Interpolation methods are used to upsample the optical flow fields corresponding to the second and third layer images to make their resolution the same as the optical flow field corresponding to the first layer image. For the aligned optical flow fields, convolution operations are performed using convolutional kernels of different sizes to extract image features. The squeezing and activation (SE) module dynamically weights and fuses the three feature maps. The SE module adaptively learns the importance weights of each channel through global average pooling, fully connected layers, and the sigmoid activation function, and then applies these weights to the feature maps.

[0097] In one specific embodiment, the process of executing step S104 may specifically include the following steps:

[0098] (1) Use the scale-invariant feature transform algorithm to extract feature points in the video frame to obtain the feature point set of the video frame;

[0099] (2) Project the three-dimensional scene model onto the viewpoint of the video frame to generate a projection image, and use the scale-invariant feature transformation algorithm to extract feature points in the projection image and generate a feature point set of the three-dimensional scene model;

[0100] (3) Combine the multi-scale optical flow information to predict the feature point position, and use the K-nearest neighbor algorithm to match the feature point set of the video frame with the feature point set of the three-dimensional scene model to obtain a set of matching pairs;

[0101] (4) Verify the consistency of the matching pair set through the homography matrix, and the matching pair set that passes the verification is the feature point matching result.

[0102] By processing video frames using the Scale Invariant Feature Transform (SIFT) algorithm, a set of feature points is obtained through scale space extremum detection, feature point localization, orientation assignment, and feature descriptor generation. First, the 3D scene model is projected into a 2D image based on camera parameters. Then, the SIFT algorithm is used to extract the feature point set from the projected image. Combined with multi-scale optical flow information, the positions of the video frame feature points in the projected image are predicted, narrowing the matching search range. The KNN algorithm is used to calculate the feature descriptor distance, filtering out reliable matching pairs. The RANSAC algorithm is used to calculate the homography matrix, and multiple iterations are performed for optimization. Inlier points are filtered out based on projection error. The verified matching pairs are the final feature point matching results. Accurate and reliable feature point matching results provide crucial correspondences for subsequent 3D video fusion. Through these matching pairs, the spatial positional relationship between the video frame and the 3D scene model can be determined, thereby achieving accurate fusion and generating high-quality 3D video. For example, in security monitoring systems, real-time video can be fused with a 3D scene model, providing more comprehensive and accurate scene information and improving the accuracy and real-time performance of monitoring.

[0103] In one specific embodiment, the process of executing step S105 may specifically include the following steps:

[0104] (1) Calculate the confidence score of the matching pair based on the feature point matching results;

[0105] (2) Calculate the local weight and global weight based on the confidence score of the matching pair;

[0106] (3) The fusion weight is obtained by assigning different coefficients to the local weights and the global weights, and then linearly adding them together. The calculation formula is as follows:

[0107] W = α * wlocal +β*w glocal

[0108] Where W is the fusion weight, w local For the local weight, w glocal Let α be the global weight, β be the coefficient of the local weight, and α + β = 1.

[0109] (4) For the video frame and the image of the three-dimensional scene model, weighted fusion is performed at the pixel position of each image according to the fusion weight to obtain the fused three-dimensional video.

[0110] Specifically, the step of weighted fusion at each pixel location of the image according to the fusion weight to obtain the fused 3D video includes:

[0111] Based on the pixel values ​​of the video frame and the pixel values ​​of the image in the 3D scene model, the fused pixel value is calculated using the following formula:

[0112] I fused =W*I model +(1-W)*I video

[0113] Among them, I fused I represents the fused pixel value. video I represents the pixel value of the video frame. model , where is the pixel value of the image in the 3D scene model, and W is the fusion weight;

[0114] The calculated fused pixel values ​​are organized into a sequence to form the three-dimensional video.

[0115] Gaussian difference operation is performed on video frame images to construct a scale space. Local extrema are searched in the scale space, and these points may be candidate points for feature points. The location and scale of feature points are accurately determined by fitting a three-dimensional quadratic function. At the same time, points with low contrast and edge response are removed to improve the stability of feature points. One or more principal directions are assigned to each feature point. Based on the gradient direction histogram in the neighborhood of the feature point, a neighborhood is selected around the feature point and divided into multiple sub-regions. The gradient direction histogram in each sub-region is calculated. These histograms are combined into a high-dimensional vector as the feature descriptor of the feature point.

[0116] The 3D scene model is projected onto the viewpoint of the video frame to generate a projected image. Then, the SIFT algorithm is used to extract feature points from the projected image, enabling feature matching between the video frame and the 3D scene model from the same viewpoint. Specifically, based on the camera parameters of the video frame, the 3D scene model is projected onto a 2D plane to generate a projected image. The SIFT algorithm is then used to extract feature points from the projected image. The steps are the same as those for extracting feature points from the video frame, and will not be repeated here.

[0117] Multi-scale optical flow information can predict the motion trajectory of feature points between different frames, narrowing the search range for feature point matching and improving matching accuracy and efficiency. The K-Nearest Neighbors (KNN) algorithm calculates the distance between feature point descriptors to find the K nearest neighbors, thereby determining the matching pair. Specifically, based on multi-scale optical flow information, the possible positions of feature points in the video frame in the next frame are predicted. The KNN algorithm is used to match the feature point set of the video frame with the feature point set of the 3D scene model. Typically, K=2 is chosen, meaning that the two nearest neighbors are found for each feature point. By comparing the distance ratio between the nearest and second nearest neighbors, reliable matching pairs are selected. If the distance ratio between the nearest and second nearest neighbors is less than a certain threshold (e.g., 0.8), the matching pair is considered reliable.

[0118] The homography matrix describes the projection transformation relationship between two planes. If a matching pair is correct, then they satisfy the homography transformation. By calculating the homography matrix and verifying the consistency of the matching pairs, mismatched points can be removed. Specifically, the Random Sample Consensus (RANSAC) algorithm is used to calculate the homography matrix from the matching pairs. The RANSAC algorithm randomly selects a set of matching pairs, calculates the homography matrix, and then verifies the consistency of other matching pairs based on the matrix. This process is iterated until the optimal homography matrix is ​​found. Based on the calculated homography matrix, each matching pair is verified. If the projection error of the matching pair is less than a certain threshold, the matching pair is considered to be consistent.

[0119] The confidence score of a matching pair can be calculated based on factors such as the matching distance between feature point descriptors and the projection error of the homography matrix. The smaller the matching distance and the smaller the projection error, the more reliable the matching pair is, and the higher the confidence score is. For each matching pair, the distance between its feature point descriptors is calculated, the distance is normalized, and then the reciprocal is taken as the matching distance score. The projection error of each matching pair is calculated based on the homography matrix, the error is normalized, and then the reciprocal is taken as the projection error score. The matching distance score and the projection error score are weighted and averaged to obtain the confidence score of the matching pair.

[0120] Local weights consider the distribution of matching pairs within local regions, while global weights consider the overall situation of all matching pairs. By calculating local and global weights, the importance of matching pairs can be reflected more comprehensively. Specifically, the image is divided into multiple local regions, and the sum of the confidence scores of matching pairs within each local region is calculated and then normalized to obtain the local weights. Similarly, the sum of the confidence scores of all matching pairs is calculated and normalized to obtain the global weights. Different coefficients are assigned to the local and global weights, and then they are linearly added to obtain the fusion weight. The choice of coefficients can be adjusted according to the specific application scenario. The calculation formula is as follows:

[0121] W = α * w local +β*w glocal

[0122] Where W is the fusion weight, w local For the local weight, w glocal Let α be the global weight, β be the coefficient of the local weight, and β be the coefficient of the global weight, where α + β = 1.

[0123] At each pixel location in the image, the pixel values ​​of the video frame and the pixel values ​​of the image in the 3D scene model are weighted and fused according to the fusion weights to obtain the fused pixel values. These fused pixel values ​​are then arranged into a sequence to form the fused 3D video. The fused pixel values ​​are calculated based on the pixel values ​​of the video frame and the pixel values ​​of the image in the 3D scene model, using the following formula:

[0124] I fused =W*I model +(1-W)*I video

[0125] Among them, I fused I represents the fused pixel value. video I represents the pixel value of the video frame. model Let W be the pixel value of the image in the 3D scene model, and W be the fusion weight. By traversing each pixel of the video frame and the image in the 3D scene model, the fused pixel value is calculated according to the fusion weight. The fused pixel values ​​are then arranged into a sequence to form the 3D video, which is saved as a video file using a video processing library.

[0126] The above describes the 3D video fusion method in the embodiments of this application. The following describes the application of the 3D video fusion method in a video security system. Please refer to [link / reference]. Figure 2 This application describes the application of a 3D video fusion method in a video security system, wherein the video security system includes:

[0127] Acquisition module 201 is used to acquire a multi-angle video sequence of a target scene, the video sequence including a number of video frames;

[0128] The construction module 202, when the target scene is a static scene, analyzes the video sequence through the structured light 3D reconstruction module to obtain a static 3D mesh model; when the target scene is a dynamic scene, it analyzes the video sequence through the motion recovery structure to obtain a dynamic 3D mesh model. The construction module 202 is used to construct a 3D scene model based on the static 3D mesh model and the dynamic 3D mesh model.

[0129] The multi-scale optical flow module 203 is used to construct an image pyramid for each video frame, obtain the optical flow field corresponding to each layer of the image pyramid through an optical flow algorithm, and fuse the optical flow field through a multi-scale convolutional neural network to obtain multi-scale optical flow information.

[0130] Feature point matching module 204 is used to extract corresponding image feature points from the video frame and the three-dimensional scene model, and combine them with the multi-scale optical flow information to match the image feature points of the video frame with the image feature points of the three-dimensional scene model to generate feature point matching results.

[0131] The fusion module 205 is used to calculate the fusion weight based on the feature point matching result, and to perform weighted fusion of the video frame and the three-dimensional scene model according to the fusion weight to obtain the fused three-dimensional video.

[0132] In this embodiment, a 3D scene model is constructed by processing and combining static and dynamic scenes in the target scene separately. The static scene in the target scene is analyzed using a structured light 3D reconstruction module to obtain a static 3D mesh model from the video sequence. The dynamic scene in the target scene is analyzed using a motion recovery structure to obtain a dynamic 3D mesh model from the video sequence. A 3D scene model is then constructed based on the static and dynamic 3D mesh models, resulting in a complete 3D scene model containing both static and dynamic elements. This provides a unified 3D spatial framework for subsequent video fusion. Furthermore, an image pyramid is constructed for each video frame. The optical flow field corresponding to each layer of the image pyramid is obtained using an optical flow algorithm. The optical flow field is then fused using a multi-scale convolutional neural network to obtain multi-scale optical flow information. Images at different layers can capture objects of different sizes and motion details, which helps improve the accuracy of optical flow calculation. The fused multi-scale optical flow information provides a more reliable basis for subsequent feature point matching. Finally, the video frames and the 3D scene model are analyzed... The corresponding image feature points are extracted from the model and combined with the multi-scale optical flow information. The image feature points of the video frame are matched with the image feature points of the 3D scene model to generate feature point matching results. Feature point extraction can reduce the amount of data and help improve the efficiency of subsequent matching. The correspondence between the video frame and the 3D scene model can be found through the feature point matching results, providing accurate positional information for subsequent fusion. Finally, the fusion weight is calculated based on the feature point matching results. The video frame and the 3D scene model are weighted and fused according to the fusion weight to obtain the fused 3D video. The fusion weight calculated based on the feature point matching results not only reflects the importance of the video frame and the 3D scene model at different positions, but also more reasonably allocates the contributions of the video frame and the 3D scene model, making the fused image more natural and realistic. The fused 3D video presents a richer and more realistic visual effect. In video security systems, it can provide more comprehensive and accurate scene information, which helps to improve the accuracy and real-time performance of security monitoring and meet the high reliability requirements of video security systems.

[0133] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0134] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method of three-dimensional video fusion, characterized by, The three-dimensional video fusion method comprises: acquiring a multi-angle video sequence of a target scene, the video sequence comprising a plurality of video frames; when the target scene is a static scene, analyzing the video sequence by a structured light three-dimensional reconstruction module to obtain a static three-dimensional mesh model, and when the target scene is a dynamic scene, analyzing the video sequence by a motion recovery structure to obtain a dynamic three-dimensional mesh model, and constructing a three-dimensional scene model according to the static three-dimensional mesh model and the dynamic three-dimensional mesh model; constructing an image pyramid for each video frame, obtaining a flow field corresponding to each layer of the image pyramid by a flow algorithm, and fusing the flow field by a multi-scale convolutional neural network to obtain multi-scale flow information; extracting corresponding image feature points from the video frames and the three-dimensional scene model, combining the multi-scale flow information, matching the image feature points of the video frames with the image feature points of the three-dimensional scene model to obtain a feature point matching result; calculating a fusion weight according to the feature point matching result, and performing weighted fusion on the video frames and the three-dimensional scene model according to the fusion weight to obtain a fused three-dimensional video.

2. The three-dimensional video fusion method of claim 1, wherein, When the target scene is a static scene, the structured light three-dimensional reconstruction module is used to extract structured light encoding information in each image of the video frames, and the absolute phase value of each pixel of the image is calculated by a phase unwrapping algorithm. According to the absolute phase value, the three-dimensional space coordinates corresponding to each pixel are calculated, the three-dimensional space coordinates of the pixels are combined into point cloud data, and a static three-dimensional mesh model is generated from the point cloud data using Poisson surface reconstruction. When the target scene is a dynamic scene, the motion recovery structure is used to extract feature points of each image of the video frames, and the feature points of adjacent images of the video frames are matched by a K nearest neighbor algorithm. According to the feature points, the three-dimensional coordinates of the feature points are calculated, and the three-dimensional coordinates of the feature points are combined into a sparse point cloud. A multi-view stereo network algorithm is used to generate a dense point cloud from the sparse point cloud, and Poisson surface reconstruction is used to generate a dynamic three-dimensional mesh model from the dense point cloud. The static three-dimensional mesh model and the dynamic three-dimensional mesh model are aligned to the same coordinate system by an iterative closest point algorithm, and the static three-dimensional mesh model and the dynamic three-dimensional mesh model are merged to construct the three-dimensional scene model by a mesh stitching algorithm.

3. The three-dimensional video fusion method of claim 2, wherein, The mesh stitching algorithm is used to merge the static three-dimensional mesh model and the dynamic three-dimensional mesh model to construct the three-dimensional scene model, comprising: matching the boundary vertices of the meshes of the static three-dimensional mesh model and the dynamic three-dimensional mesh model by an iterative closest point algorithm; Delaunay triangulation is performed on the matched boundary vertices to generate triangular patches of the stitching area, and the static three-dimensional mesh model is merged with the boundary vertices of the mesh of the dynamic three-dimensional mesh model; The static three-dimensional mesh model after merging the boundary vertices is unified with the texture coordinates of the mesh of the dynamic three-dimensional mesh model, and the textures of the static three-dimensional mesh model and the dynamic three-dimensional mesh model are transitioned using Poisson image editing; According to the triangular patches, the static three-dimensional mesh model and the mesh of the dynamic three-dimensional mesh model after transition of the texture are connected to construct the three-dimensional scene model.

4. The three-dimensional video fusion method of claim 1, wherein, The image pyramid is constructed for each video frame, the optical flow field corresponding to each layer image of the image pyramid is obtained through an optical flow algorithm, and the optical flow field is fused through a multi-scale convolutional neural network to obtain multi-scale optical flow information, including: Gaussian blur and downsampling are performed on the image of the video frame to generate multiple layers of images, and the image pyramid is formed by the multiple layers of images, the image pyramid at least including a first layer image, a second layer image and a third layer image, the resolution of the first layer image being greater than the resolution of the second layer image, and the resolution of the second layer image being greater than the resolution of the third layer image; The first layer image, the second layer image and the third layer image are respectively calculated through an optical flow algorithm to obtain the optical flow field corresponding to the first layer image, the optical flow field corresponding to the second layer image and the optical flow field corresponding to the third layer image; The optical flow field corresponding to the first layer image, the optical flow field corresponding to the second layer image and the optical flow field corresponding to the third layer image are fused through a multi-scale convolutional neural network to generate multi-scale optical flow information with consistent image features.

5. The three-dimensional video fusion method of claim 4, wherein, The image of the video frame is subjected to Gaussian blur and downsampling to generate multiple layers of images, and the image pyramid is formed by the multiple layers of images, the image pyramid at least including a first layer image, a second layer image and a third layer image, the resolution of the first layer image being greater than the resolution of the second layer image, and the resolution of the second layer image being greater than the resolution of the third layer image, including: The image of the video frame is subjected to Gaussian blur and downsampling to generate multiple layers of images, and the image pyramid is formed by the multiple layers of images, the image pyramid at least including a first layer image, a second layer image and a third layer image, the resolution of the first layer image being greater than the resolution of the second layer image, and the resolution of the second layer image being greater than the resolution of the third layer image, including: The first layer image is subjected to Gaussian blur and downsampling to obtain the second layer image, and the resolution of the second layer image is lower than that of the first layer image; The second layer image is subjected to Gaussian blur and downsampling to obtain the third layer image, and the resolution of the third layer image is lower than that of the second layer image.

6. The three-dimensional video fusion method of claim 4, wherein, The optical flow field corresponding to the first layer image, the optical flow field corresponding to the second layer image and the optical flow field corresponding to the third layer image are fused through a multi-scale convolutional neural network to generate multi-scale optical flow information with consistent image features, including: The optical flow field corresponding to the first layer image, the optical flow field corresponding to the second layer image and the optical flow field corresponding to the third layer image are aligned to the same resolution through upsampling; The first feature map is obtained by extracting image features through 3*3 convolution for the optical flow field corresponding to the aligned first layer image, the second feature map is obtained by extracting image features through 5*5 convolution for the optical flow field corresponding to the aligned second layer image, and the third feature map is obtained by extracting image features through 7*7 convolution for the optical flow field corresponding to the aligned third layer image; The first feature map, the second feature map and the third feature map are dynamically weighted and fused through the squeezing and excitation module in the multi-scale convolutional neural network, and then 3*3 convolution is used to generate multi-scale optical flow information with consistent image features.

7. The three-dimensional video fusion method of claim 1, wherein, The corresponding image feature points are extracted from the video frame and the three-dimensional scene model, and the image feature points of the video frame are matched with the image feature points of the three-dimensional scene model to generate a feature point matching result, including: The feature points in the video frame are extracted using a scale-invariant feature transform algorithm to obtain a feature point set of the video frame; The three-dimensional scene model is projected into the perspective of the video frame to generate a projection image, and the feature points in the projection image are extracted using a scale-invariant feature transform algorithm to generate a feature point set of the three-dimensional scene model; The feature point position is predicted in combination with the multi-scale optical flow information, and the feature point set of the video frame is matched with the feature point set of the three-dimensional scene model using a K-nearest neighbor algorithm to obtain a matching pair set; The consistency of the matching pair set is verified through a homography matrix, and the matching pair set that passes the verification is the feature point matching result.

8. The three-dimensional video fusion method of claim 1, wherein, The fusion weight is calculated according to the feature point matching result, and the video frame and the three-dimensional scene model are weighted and fused according to the fusion weight to obtain a fused three-dimensional video, including: The confidence score of the matching pair is calculated based on the feature point matching result; The local weight and the global weight are calculated based on the confidence score of the matching pair; The fusion weight is obtained by assigning different coefficients to the local weight and the global weight; For the images of the video frame and the three-dimensional scene model, the fused three-dimensional video is obtained by weighting and fusing the images at each pixel position according to the fusion weight.

9. The three-dimensional video fusion method of claim 8, wherein, For the images of the video frame and the three-dimensional scene model, the fused three-dimensional video is obtained by weighting and fusing the images at each pixel position according to the fusion weight. The fused pixel value is calculated based on the pixel value of the video frame and the pixel value of the image in the three-dimensional scene model; The fused pixel value is arranged into a sequence and forms the three-dimensional video.

10. A three-dimensional video fusion system for implementing the three-dimensional video fusion method according to any one of claims 1-9, characterized in that, The system comprises: An acquisition module is configured to acquire a target scene multi-angle video sequence, wherein the video sequence comprises a plurality of video frames; A construction module is configured to analyze the video sequence by a structured light three-dimensional reconstruction module to obtain a static three-dimensional mesh model when the target scene is a static scene, and analyze the video sequence by a motion recovery structure to obtain a dynamic three-dimensional mesh model when the target scene is a dynamic scene, and construct a three-dimensional scene model based on the static three-dimensional mesh model and the dynamic three-dimensional mesh model. a multi-scale optical flow module, configured to construct an image pyramid for each of the video frames, derive an optical flow field corresponding to each layer of the image pyramid by an optical flow algorithm, and fuse the optical flow fields by a multi-scale convolutional neural network to obtain multi-scale optical flow information; a feature point matching module, configured to extract corresponding image feature points from the video frames and the three-dimensional scene model, combine the multi-scale optical flow information, match the image feature points of the video frames with the image feature points of the three-dimensional scene model to generate a feature point matching result; a fusion module, configured to calculate a fusion weight according to the feature point matching result, and perform weighted fusion on the video frames and the three-dimensional scene model according to the fusion weight to obtain a fused three-dimensional video.

Images