Real-time fusion processing method and system for dynamic video stream based on three-dimensional scene

Abstract

The application relates to the technical field of computer vision, and provides a dynamic video stream real-time fusion processing method and system based on a three-dimensional scene. By extracting 2D key views and establishing a mapping table of 2D key view pixels and three-dimensional model vertex coordinates, reverse mapping of a two-dimensional registration result to three-dimensional coordinates is realized, a deterministic data index structure is constructed on a three-dimensional model surface, and the corresponding relationship between a video frame picture and a three-dimensional model surface point can be directly obtained by table lookup. A matching pair set is obtained through a LoFTR feature matching module, and an improved RANSAC algorithm is used to solve a target homography matrix, so that the mapping relationship between two-dimensional images and the matching constraint and data basis used for two-dimensional-three-dimensional coordinate query are consistent. A perspective projection model is constructed to form a space correlation, and the space correlation is taken as a structured data object, so that a uniform 3D-2D correlation data source is provided for a GPU acceleration unit to generate dynamic texture.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and system for real-time fusion processing of dynamic video streams based on three-dimensional scenes. Background Technology

[0002] Current video fusion technologies in 3D scenes mainly include dynamic texture mapping fusion, visual feature-based registration and projection fusion, and cloud streaming rendering fusion. Dynamic texture mapping fusion typically maps video frames as texture resources onto the surface of a 3D model, achieving front-end fusion rendering through the texture update interface of the 3D rendering engine. Visual feature-based registration and projection fusion extracts feature points from 2D images and calculates projection relationships using a camera projection model to complete geometric registration and texture fusion from video frames to the 3D model surface. Cloud streaming rendering fusion captures and encodes the rendered image of the 3D application on the server side, then streams it over the network to the front-end terminal for display and compositing.

[0003] However, existing technologies for reverting 2D image registration results to 3D geometric relationships typically rely on manually selecting control points, matching based on sparse features, or recalculating 3D projection correspondences during the front-end rendering stage. Due to the lack of a direct indexing mechanism between key view sets and view pixel-3D point coordinate mapping tables, the pixel correspondences obtained from 2D registration cannot directly retrieve 3D coordinates consistent with points on the 3D model surface through queries. Consequently, the texture projection correspondence between video frame images and the 3D model surface cannot form a standardized data input structure supported by queries in the method chain. Summary of the Invention

[0004] Therefore, it is necessary to provide a method and system for real-time fusion processing of dynamic video streams based on three-dimensional scenes to address the aforementioned technical problems.

[0005] A real-time fusion processing method for dynamic video streams based on a 3D scene is used in a real-time fusion device for dynamic video streams. The device includes a data preprocessing layer, a static registration layer, and a real-time fusion rendering layer. The static registration layer is equipped with a LoFTR feature matching module, and the real-time fusion rendering layer includes a GPU acceleration unit and a WebGL rendering engine.

[0006] The method includes the following steps: Multiple video streams are input and a 3D model is loaded in the data preprocessing layer; Decode multiple video streams to obtain video frame images and preprocess the video frame images; convert the 3D model into a triangular patch structure, read the UV coordinates of the 3D model and store the UV coordinates in the UV channel, extract the 2D key view, and establish a mapping table between the pixels of the 2D key view and the vertex coordinates of the 3D model. The LoFTR feature matching module is used to match feature points between the video frame image and the 2D key view to obtain a set of matching pairs. The target homography matrix of the matching pair set is solved by the improved RANSAC algorithm. Based on the target homography matrix, the pixel coordinates of the video frame image are mapped to the pixel coordinates in the 2D key view. The corresponding 3D point coordinates are obtained by querying the mapping table between the 2D key view pixels and the 3D model vertex coordinates. A perspective projection model is constructed based on the three-dimensional point coordinates and the corresponding video frame pixel coordinates. The perspective projection matrix is ​​obtained by solving the perspective projection matrix. The correspondence between the three-dimensional point coordinates and the video frame image pixel coordinates is established based on the perspective projection matrix. Combined with the UV coordinates of the three-dimensional model vertices, a spatial relationship between three-dimensional points, pixels, and UV is formed. The GPU acceleration unit establishes a mapping relationship between the pixel coordinates of the video frame image and the UV coordinates of the 3D model based on the spatial association, samples the video frame information to generate dynamic textures, and the WebGL rendering engine samples and renders the dynamic textures to output a real-time fused image of the video and the 3D scene.

[0007] In one embodiment, decoding multiple video streams to obtain video frame images and preprocessing the video frame images includes: Decode multiple video streams to obtain video frame images, add UTC timestamps to the video frame images, and perform timing alignment based on the added UTC timestamps; The aligned video frame images are then subjected to image quality enhancement processing.

[0008] In one embodiment, a UTC timestamp is added to the video frame image, and timing alignment is performed based on the added UTC timestamp, including: Obtain the UTC timestamp corresponding to each video stream and determine the target video frame image of each video stream based on the same fusion time. When the time difference between the UTC timestamp of the target video frame image and the fusion time is greater than a preset time difference threshold, the target video frame image is discarded or waited for; when the time difference is not greater than the preset time difference threshold, the target video frame image is used as input to the LoFTR feature matching module and the GPU acceleration unit.

[0009] In one embodiment, the 3D model is converted into a triangular patch structure, the UV coordinates of the 3D model are read and stored in the UV channel, 2D key views are extracted, and a mapping table between the pixels of the 2D key views and the vertex coordinates of the 3D model is established, including: The 3D model is converted into a triangular patch structure, the UV coordinates of the converted 3D model are read, and the UV coordinates are stored in the UV channel. A spherical coordinate system is established with the geometric center of the 3D model as the origin. Multiple virtual camera positions are set in the spherical coordinate system according to the preset azimuth angle interval, and a 2D key view is generated by the renderer. Record the pixel coordinates of each 2D key view and the 3D point coordinates of the corresponding visible surface points in the renderer, and establish a mapping table between the pixels of the 2D key view and the vertex coordinates of the 3D model.

[0010] In one embodiment, the LoFTR feature matching module performs feature point matching between the video frame image and the 2D key view to obtain a set of matching pairs, including: The video frame image and the 2D key view are scaled according to a preset input size and then input into the LoFTR model image encoder. The cosine similarity of all pixel position feature descriptors is calculated to obtain candidate matching pairs. The candidate matching pairs are threshold-filtered and output to form a matching pair set. Each matching pair in the matching pair set contains the pixel coordinates of the video frame image and the pixel coordinates of the 2D key view.

[0011] In one embodiment, solving the target homography matrix of the matching pair set using an improved RANSAC algorithm includes: Construct the local spatial neighborhood of the matching pair set and generate a seed point set; Random sampling consistency iteration is performed using the seed point set as the sampling object; For each iteration of the candidate homography matrix, the pixel coordinate error is calculated based on the reprojection model. The set of inliers that meet the preset error threshold is statistically analyzed, and the candidate homography matrix with the highest inlier rate is selected as the target homography matrix.

[0012] In one embodiment, a correspondence between 3D point coordinates and video frame image pixel coordinates is established based on a perspective projection matrix, and combined with the UV coordinates of the 3D model vertices, a spatial relationship of 3D point-pixel-UV is formed, including: The three-dimensional point coordinates are projected back to the video frame image based on the perspective projection matrix to obtain two-dimensional coordinates. Establish a correspondence between the two-dimensional coordinates and the pixel coordinates of the video frame image to form a spatial association.

[0013] In one embodiment, the GPU acceleration unit establishes a mapping relationship between the pixel coordinates of the video frame image and the UV coordinates of the 3D model based on the spatial correlation, and samples the video frame information to generate dynamic textures, including: Upload video frame images to the GPU texture cache and read the UV channel vertex UV coordinates and spatial relationships of the 3D model; Based on spatial correlation, determine the pixel coordinates of the UV channel vertex UV coordinates of the 3D model in the GPU texture cache, and perform bilinear interpolation sampling on the pixel coordinates to obtain the sampled pixel values. The sampled pixel values ​​are written into a dynamic texture cache object that matches the UV coordinates of the UV channel vertices of the 3D model to form a dynamic texture.

[0014] In one embodiment, the WebGL rendering engine samples dynamic textures and outputs a blended image, including: The vertex coordinates of the 3D model are read by the vertex shader and the modelView projection transformation is performed. The dynamic texture is read by the fragment shader and texture sampling is performed based on the UV coordinates after bilinear interpolation. Input the light direction vector, normal vector, and line-of-sight vector into the Phong lighting model to calculate the color; Perform frustum culling in the renderer and output the blended image.

[0015] A real-time fusion processing system for dynamic video streams based on a 3D scene is used in a real-time fusion device for dynamic video streams. The device includes a data preprocessing layer, a static registration layer, and a real-time fusion rendering layer. The static registration layer is equipped with a LoFTR feature matching module, and the real-time fusion rendering layer includes a GPU acceleration unit and a WebGL rendering engine. The system includes: The data loading module is used to access multiple video streams and load 3D models in the data preprocessing layer; The data processing module is used to decode multiple video streams to obtain video frame images and preprocess the video frame images; convert the 3D model into a triangular patch structure, read the UV coordinates of the 3D model and store the UV coordinates in the UV channel, extract the 2D key view, and establish a mapping table between the pixels of the 2D key view and the vertex coordinates of the 3D model. The static registration module is used to perform feature point matching between the video frame image and the 2D key view using the LoFTR feature matching module to obtain a set of matching pairs; the target homography matrix of the matching pair set is solved by the improved RANSAC algorithm; the pixel coordinates of the video frame image are mapped to the pixel coordinates in the 2D key view based on the target homography matrix; and the corresponding 3D point coordinates are obtained by querying the mapping table between the 2D key view pixels and the 3D model vertex coordinates. The spatial association module is used to construct a perspective projection model based on the three-dimensional point coordinates and the corresponding video frame pixel coordinates, solve for the perspective projection matrix, establish the correspondence between the three-dimensional point coordinates and the video frame image pixel coordinates based on the perspective projection matrix, and combine the three-dimensional model vertex UV coordinates to form a three-dimensional point-pixel-UV spatial association. The real-time fusion rendering module is used by the GPU acceleration unit to establish a mapping relationship between the pixel coordinates of the video frame image and the UV coordinates of the 3D model based on the spatial association, sample the video frame information to generate dynamic textures, and then use the WebGL rendering engine to sample and render the dynamic textures to output a real-time fused image of the video and the 3D scene.

[0016] The aforementioned method and system for real-time fusion processing of dynamic video streams based on 3D scenes extracts 2D key views and establishes a mapping table between the pixels of the 2D key views and the coordinates of the 3D model vertices, thereby realizing the reverse mapping of the 2D registration results to the 3D coordinates. This constructs a deterministic data index structure on the surface of the 3D model, allowing the correspondence between video frame images and points on the surface of the 3D model to be directly obtained by looking up the table, without the need for real-time iterative solution of 3D control points or manual intervention.

[0017] The LoFTR feature matching module obtains a set of matching pairs, and the improved RANSAC algorithm is used to solve the target homography matrix, ensuring that the mapping relationship between two-dimensional images uses the same matching constraints and data foundation as the two-dimensional-to-three-dimensional coordinate query.

[0018] By constructing a perspective projection model to form spatial relationships, and using these spatial relationships as structured data objects, a unified 3D–2D relational data source is provided for the subsequent generation of dynamic textures by the GPU acceleration unit. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a real-time fusion processing method for dynamic video streams based on a 3D scene in one embodiment. Figure 2 This is a structural block diagram of a real-time fusion processing system for dynamic video streams based on a 3D scene, as described in one embodiment. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] In one embodiment, such as Figure 1 As shown, a real-time fusion processing method for dynamic video streams based on a 3D scene is provided for use in a real-time fusion device for dynamic video streams. The device includes a data preprocessing layer, a static registration layer, and a real-time fusion rendering layer. The static registration layer is equipped with a LoFTR feature matching module, and the real-time fusion rendering layer includes a GPU acceleration unit and a WebGL rendering engine.

[0022] It should be noted that the layers in the real-time fusion device for dynamic video streams exchange data objects through memory queues, shared caches, or interface calls and execute them in a unified timing sequence.

[0023] The method includes the following steps: Step 201: In the data preprocessing layer, multiple video streams are accessed and a 3D model is loaded.

[0024] Step 202: Decode multiple video streams to obtain video frame images and preprocess the video frame images; convert the 3D model into a triangular patch structure, read the UV coordinates of the 3D model and store the UV coordinates in the UV channel, extract the 2D key view, and establish a mapping table between the pixels of the 2D key view and the vertex coordinates of the 3D model.

[0025] It should be noted that the triangular patch structure includes a vertex set, a patch index set, and a vertex coordinate set. The read UV coordinates are used to represent the parameter correspondence between the 3D model surface and the 2D texture. When reading the UV coordinates, parametric solutions are performed on the triangular patch structure to obtain the UV coordinates corresponding to each vertex and write them into the UV channel. The UV channel maintains a consistent mapping relationship with the vertex index.

[0026] Step 203: The LoFTR feature matching module is used to match feature points between the video frame image and the 2D key view to obtain a set of matching pairs; the target homography matrix of the matching pair set is solved by the improved RANSAC algorithm; the pixel coordinates of the video frame image are mapped to the pixel coordinates in the 2D key view based on the target homography matrix; and the corresponding 3D point coordinates are obtained by querying the mapping table between the 2D key view pixels and the 3D model vertex coordinates.

[0027] Step 204: Construct a perspective projection model based on the three-dimensional point coordinates and the corresponding video frame pixel coordinates, solve for the perspective projection matrix, establish the correspondence between the three-dimensional point coordinates and the video frame image pixel coordinates based on the perspective projection matrix, and combine the three-dimensional model vertex UV coordinates to form a three-dimensional point-pixel-UV spatial association.

[0028] Step 205: The GPU acceleration unit establishes a mapping relationship between the pixel coordinates of the video frame image and the UV coordinates of the 3D model based on the spatial association, samples the video frame information to generate dynamic textures, and the WebGL rendering engine samples and renders the dynamic textures to output a real-time fused image of the video and the 3D scene.

[0029] In the aforementioned real-time fusion processing method for dynamic video streams based on 3D scenes, a mapping table is established between 2D key views and the coordinates of 3D model vertices. This enables the reverse mapping of 2D registration results to 3D coordinates, thereby constructing a deterministic data index structure on the 3D model surface. The correspondence between video frame images and points on the 3D model surface can be directly obtained through table lookup, eliminating the need for real-time iterative solving of 3D control points or manual intervention. A set of matching pairs is obtained through the LoFTR feature matching module, and the target homography matrix is ​​solved using an improved RANSAC algorithm, ensuring consistent matching constraints and data foundation for the mapping relationship between 2D images and the 2D-3D coordinate lookup. Spatial associations are formed by constructing a perspective projection model, and these spatial associations are used as structured data objects, providing a unified 3D-2D association data source for subsequent dynamic texture generation by the GPU acceleration unit.

[0030] In one embodiment, step 202 involves decoding multiple video streams to obtain video frame images, and preprocessing the video frame images, including: Decode multiple video streams to obtain video frame images, add UTC timestamps to the video frame images, and perform timing alignment based on the added UTC timestamps; The aligned video frame images are then subjected to image quality enhancement processing.

[0031] In this embodiment, global timing alignment of multiple video streams is performed using a unified UTC timestamp to eliminate time offsets caused by acquisition latency, transmission delay, and frame rate differences among the video streams. This ensures that the multiple video frames are strictly synchronized on the timeline, thereby accurately selecting the target video frame image corresponding to the 3D scene fusion moment from each video stream. Furthermore, image quality enhancement processing is applied to the timing-aligned target video frame image to improve the clarity and texture consistency of the video frames, providing a high-quality, timing-synchronized image data source for subsequent GPU dynamic texture generation and real-time fusion of the 3D scene.

[0032] Specifically, any one of the multiple video streams is denoted as the i-th video stream, where, Decoding multiple video streams involves using a video decoder to convert the compressed bitstream into raw frames, and then converting the raw frames into a unified pixel format to form video frame images. The i-th video stream is decoded to obtain the i-th... Video frame frame image denoted as The video frame image contains a pixel matrix and channel information; the UTC timestamp added to the video frame image is... The UTC timestamp is bound to the video frame image and stored, denoted as... .

[0033] In one embodiment, a UTC timestamp is added to the video frame image, and timing alignment is performed based on the added UTC timestamp, including: Obtain the UTC timestamp corresponding to each video stream and determine the target video frame image of each video stream based on the same fusion time. When the time difference between the UTC timestamp of the target video frame image and the fusion time is greater than a preset time difference threshold, the target video frame image is discarded or waited for; when the time difference is not greater than the preset time difference threshold, the target video frame image is used as input to the LoFTR feature matching module and the GPU acceleration unit.

[0034] Specifically, in terms of fusion moment Based on this, the target video frame images of each video stream are determined. ; for the first The target frame index is defined as follows for the video stream:

[0035] Where m is the frame index of all video frames in the i-th video stream; the preset time difference threshold is... ;when When, a discard or wait strategy is applied to the target video frame image: when At that time, the target video frame image is output to the subsequent processing queue and used as a participant. The inputs are to the FTR feature matching module and the GPU acceleration unit. The discard or wait strategy is implemented using a time-aligned state machine, which controls the writing to the output queue based on the current fusion time, the timestamp in the cache queue, and threshold conditions.

[0036] In one embodiment, the aligned video frame images are subjected to image quality enhancement processing, including Gaussian filtering for noise reduction, histogram matching to unify color distribution, and white balance adjustment.

[0037] After image enhancement processing, the video frame images are output to the input queue of the LoFTR feature matching module.

[0038] It should be noted that the image enhancement processing is performed on each target video frame image in a fixed processing order. Specifically, for each target video frame image... Gaussian filtering noise reduction performs convolution operations in the pixel domain, and the convolution kernel parameters are determined by the standard deviation of the Gaussian kernel. The kernel size parameter is determined together with the image size parameter; histogram matching uses the channel cumulative distribution function of the reference frame as the target distribution and transforms the pixel values ​​according to the monotonic mapping relationship; white balance adjustment calculates the three-channel gain coefficients according to the gray-world method and linearly scales the channel pixel values; the video image after image enhancement processing is denoted as... and will Output to The input queue of the FTR feature matching module.

[0039] In one embodiment, step 202 involves converting the 3D model into a triangular patch structure, reading the UV coordinates of the 3D model and storing the UV coordinates in the UV channel, extracting 2D key views, and establishing a mapping table between 2D key view pixels and 3D model vertex coordinates, including: The 3D model is converted into a triangular patch structure, the UV coordinates of the 3D model are read, and the UV coordinates are stored in the UV channel. A spherical coordinate system is established with the geometric center of the 3D model as the origin. Multiple virtual camera positions are set in the spherical coordinate system according to the preset azimuth angle interval, and a 2D key view is generated by the renderer. Record the pixel coordinates of each 2D key view and the 3D point coordinates of the corresponding visible surface points in the renderer, and establish a mapping table between the pixels of the 2D key view and the vertex coordinates of the 3D model.

[0040] It should be noted that during the rendering process, a depth value is output for each pixel, and the visible surface point information corresponding to that pixel is retained.

[0041] Specifically, the extracted 2D key views are denoted as ,in For each 2D key view, a view pixel-to-3D point coordinate mapping table is established. The mapping table uses the 2D key view pixel coordinates as an index to store the corresponding 3D point coordinates. For the first... indivual The key view, its pixel coordinates are The view pixel-to-3D point coordinate mapping table is... Used as an index to store the 3D point coordinates corresponding to that pixel. The view pixel-to-3D point coordinate mapping table is implemented using a one-dimensional array or an equivalent index structure, and is linked with... Key view identifiers are associated with storage.

[0042] In one embodiment, in step 203, the LoFTR feature matching module performs feature point matching between the video frame image and the 2D key view to obtain a set of matching pairs, including: The video frame image and the 2D key view are scaled according to a preset input size and then input into the LoFTR model image encoder. The cosine similarity of all pixel position feature descriptors is calculated to obtain candidate matching pairs. The candidate matching pairs are threshold-filtered and output to form a matching pair set. Each matching pair in the matching pair set contains the pixel coordinates of the video frame image and the pixel coordinates of the 2D key view.

[0043] Specifically, the video frame image and the 2D key view are scaled according to a preset input size to obtain a scaled image. The scaling relationship is recorded as a scale parameter for subsequent mapping of the matching results back to the original resolution coordinate system. The LoFTR feature matching module loads the pre-trained LoFTR model parameters and uses the LoFTR model image encoder to... Extract feature descriptors and calculate cosine similarity to generate a candidate matching pair set; let the elements of the candidate matching pair be... ,in These are the pixel coordinates of the video frame image. for The pixel coordinates of the key view; candidate matching pairs are filtered according to a preset similarity threshold to obtain a set of matching pairs. .

[0044] In one embodiment, solving the target homography matrix of the matching pair set using an improved RANSAC algorithm includes: Construct the local spatial neighborhood of the matching pair set and generate a seed point set; Random sampling consistency iteration is performed using the seed point set as the sampling object; For each iteration of the candidate homography matrix, the pixel coordinate error is calculated based on the reprojection model. The set of inliers that meet the preset error threshold is statistically analyzed, and the candidate homography matrix with the highest inlier rate is selected as the target homography matrix.

[0045] Specifically, the improved RANSAC algorithm is applied to the set of matching pairs. A local spatial neighborhood is constructed and a seed point set is generated. The seed point set is generated based on the pixel coordinates of the video frame images. The neighborhood radius parameter is used to complete the aggregation; random sampling consistency iteration is performed on each seed point set, and in each iteration, the smallest sample set is selected from the seed point set to solve for the candidate homography matrix H; the homography matrix is ​​represented in homogeneous coordinates, let The mapping relationship is then:

[0046] in Indicate homogeneous proportional equivalence; calculate the reprojection error for each pair of matching points, let:

[0047] The reprojection error is:

[0048] in, This is the homogeneous coordinate normalization result for the reprojected points. The set of interior points is determined based on a preset error threshold, and the interior point ratio is calculated. The candidate homography matrix with the highest interior point ratio is selected as the target homography matrix. .

[0049] It should be noted that a local spatial neighborhood refers to defining all image patches within a fixed spatial radius R (number of pixels) centered at the pixel center coordinates (x, y) of each image patch as its local spatial neighborhood. Generating a seed point set involves calculating the L2 norm of the corresponding feature vectors of all image patches within each local spatial neighborhood. A larger norm indicates higher feature discrimination of the image patch. The K image patches (e.g., K=8) are sorted in descending order of feature saliency, and their center coordinates and corresponding feature matching pairs are selected as seed points for that neighborhood. The seed points selected from all local spatial neighborhoods are then aggregated, deduplicated, and a global seed point set is obtained.

[0050] The reprojection model refers to using the homography matrix calculated in the current iteration to project the coordinates of the feature points in the first image onto the second image, and then subtracting the true coordinates of the actual matching points in the second image.

[0051] In step 203, the pixel coordinates of the video frame image are mapped to the pixel coordinates in the 2D key view based on the target homography matrix, and the corresponding 3D point coordinates are obtained by querying the mapping table between the 2D key view pixels and the 3D model vertex coordinates.

[0052] Specifically, for any video frame image pixel coordinates Construct homogeneous coordinates ,calculate:

[0053] Then normalize to obtain the 2D key view pixel coordinates ;by As an index in the By querying the mapping table between the pixels of each 2D key view and the vertex coordinates of the 3D model, the corresponding 3D point coordinates can be obtained. ; and This creates a single data record, which is then used to build spatial relationships later.

[0054] In one embodiment, step 204 involves obtaining the perspective projection matrix; specifically, reading the camera intrinsic parameter matrix. With camera external parameters ,in, Let be a rotation matrix. The translation vector is used; the perspective projection matrix is ​​obtained by combining the camera intrinsic and extrinsic parameters. .

[0055] In one embodiment, step 204 involves establishing a correspondence between 3D point coordinates and video frame image pixel coordinates based on the perspective projection matrix, and combining this with the UV coordinates of the 3D model vertices to form a spatial association between 3D points, pixels, and UV coordinates, including: The three-dimensional point coordinates are projected back to the video frame image based on the perspective projection matrix to obtain two-dimensional coordinates. Establish a correspondence between the two-dimensional coordinates and the pixel coordinates of the video frame image to form a spatial association.

[0056] Specifically, for three-dimensional point coordinates Construct homogeneous coordinates calculate ; in, Two-dimensional coordinates; To use the perspective projection matrix homogeneous coordinates of a 3D point The homogeneous coordinate vector obtained after linear transformation. Compared with the pixel coordinates of the original video frame image Establish correspondences and form spatial connections Basic records; spatial association It is stored in structured data format. Spatial associations include video frame image identifiers and video frame image pixel coordinates. Corresponding 3D point coordinates Two-dimensional coordinates calculated from perspective projection matrix And an index field used to subsequently establish UV coordinate mapping.

[0057] In one embodiment, in step 205, the GPU acceleration unit establishes a mapping relationship between the pixel coordinates of the video frame image and the UV coordinates of the 3D model based on the spatial association, and samples the video frame information to generate a dynamic texture, including: Upload video frame images to the GPU texture cache and read the UV channel vertex UV coordinates and spatial relationships of the 3D model; Based on spatial correlation, determine the pixel coordinates of the UV channel vertex UV coordinates of the 3D model in the GPU texture cache, and perform bilinear interpolation sampling on the pixel coordinates to obtain the sampled pixel values. The sampled pixel values ​​are written into a dynamic texture cache object that matches the UV coordinates of the UV channel vertices of the 3D model to form a dynamic texture.

[0058] The resulting dynamic textures are GPU texture resources that can be directly accessed by the WebGL rendering engine.

[0059] In this embodiment, a mapping from the UV parameter domain to the pixel coordinates of the video frame image is constructed based on spatial correlation, and the GPU acceleration unit executes the sampling of pixel values ​​and writes them into the dynamic texture; the sampling results of the dynamic texture are directly written into the GPU texture object parameter domain to form a texture resource that can be accessed by the WebGL rendering engine, avoiding the need to recalculate the coordinate mapping relationship during the front-end rendering stage.

[0060] Specifically, the GPU acceleration unit uploads video frame images to the GPU texture buffer. The GPU texture buffer stores the pixel data of each video frame image in the form of texture objects and establishes an index using frame identifiers and path identifiers. The GPU acceleration unit reads the UV coordinates of the 3D model's UV channels and reads the spatial association A. i,k The corresponding record in the buffer; the GPU acceleration unit determines the corresponding pixel coordinates of the UV coordinates in the GPU texture buffer based on spatial correlation, forming a correspondence between the pixel coordinates of the video frame image and the UV coordinates of the 3D model, and generates dynamic texture GPU texture resources accordingly; the GPU acceleration unit performs bilinear interpolation sampling on the pixel coordinates to obtain sampled pixel values, and writes the sampled pixel values ​​into the dynamic texture GPU texture resources consistent with the UV coordinates; let the sampled coordinates of a certain video frame image texture in the GPU texture buffer be... in Let the coordinates be floating-point coordinates; ; And order ; Let the pixel value of the texture at integer coordinates be... The result of bilinear interpolation is

[0061] ; in, For horizontal interpolation weights; For vertical interpolation weights; The coordinates are the integer coordinates of the lower left neighborhood. The coordinates are the integer coordinates of the lower right neighborhood. The coordinates are integers of the upper left neighborhood. The coordinates are integers in the upper right neighborhood.

[0062] GPU acceleration unit will Write the corresponding UV positions of the dynamic texture GPU texture resource; the dynamic texture GPU texture resource is stored as a GPU texture object and maintains the same parameter field index relationship as the UV channel of the 3D model.

[0063] In one embodiment, in step 205, the WebGL rendering engine samples the dynamic texture and outputs the blended image, including: The vertex coordinates of the 3D model are read by the vertex shader and the modelView projection transformation is performed. The dynamic texture is read by the fragment shader and texture sampling is performed based on the UV coordinates after bilinear interpolation. Input the light direction vector, normal vector, and line-of-sight vector into the Phong lighting model to calculate the color; Perform frustum culling in the renderer and output the blended image.

[0064] Specifically, the WebGL rendering engine initializes the WebGL rendering context, loads the vertex buffer, index buffer, and UV channel data of the 3D model, and binds the dynamic texture GPU texture resource as a texture sampler input accessible to the fragment shader; the vertex shader reads the vertex coordinates of the 3D model and performs modelView projection transformation to transform the 3D vertex coordinates to clip space coordinates; the fragment shader reads the interpolated UV coordinates and performs texture sampling on the dynamic texture GPU texture resource to obtain the fragment color input; the fragment shader calculates the color based on the Phong lighting model, whose input includes the light direction vector, normal vector, and view direction vector, and whose output is the fragment color value; before the rendering call, the renderer performs frustum culling on the 3D model and writes the rendered output into the browser Canvas.

[0065] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0066] In one embodiment, such as Figure 2 As shown, a real-time fusion processing system for dynamic video streams based on a 3D scene is provided for use in a real-time fusion device for dynamic video streams. The device includes a data preprocessing layer, a static registration layer, and a real-time fusion rendering layer. The static registration layer is equipped with a LoFTR feature matching module, and the real-time fusion rendering layer includes a GPU acceleration unit and a WebGL rendering engine. The system includes: The data loading module 901 is used to access multiple video streams and load 3D models in the data preprocessing layer.

[0067] The data processing module 902 is used to decode multiple video streams to obtain video frame images and preprocess the video frame images; convert the 3D model into a triangular patch structure, read the UV coordinates of the 3D model and store the UV coordinates in the UV channel, extract the 2D key view, and establish a mapping table between the pixels of the 2D key view and the vertex coordinates of the 3D model.

[0068] The static registration module 903 is used to perform feature point matching between the video frame image and the 2D key view through the LoFTR feature matching module to obtain a set of matching pairs; solve the target homography matrix of the set of matching pairs by the improved RANSAC algorithm; map the pixel coordinates of the video frame image to the pixel coordinates in the 2D key view based on the target homography matrix; and obtain the corresponding 3D point coordinates by querying the mapping table between the 2D key view pixels and the 3D model vertex coordinates.

[0069] The spatial association module 904 is used to construct a perspective projection model based on the three-dimensional point coordinates and the corresponding video frame pixel coordinates, solve for the perspective projection matrix, establish the correspondence between the three-dimensional point coordinates and the video frame image pixel coordinates based on the perspective projection matrix, and combine the three-dimensional model vertex UV coordinates to form a three-dimensional point-pixel-UV spatial association.

[0070] The real-time fusion rendering module 905 is used by the GPU acceleration unit to establish a mapping relationship between the pixel coordinates of the video frame image and the UV coordinates of the 3D model based on the spatial association, sample the video frame information to generate dynamic textures, and then use the WebGL rendering engine to sample and render the dynamic textures to output a real-time fused image of the video and the 3D scene.

[0071] Specific limitations regarding the real-time fusion processing system for dynamic video streams based on 3D scenes can be found in the limitations of the real-time fusion processing method for dynamic video streams based on 3D scenes mentioned above, and will not be repeated here. Each module in the aforementioned real-time fusion processing system for dynamic video streams based on 3D scenes can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0072] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0073] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for real-time fusion processing of dynamic video streams based on a 3D scene, characterized in that, In a real-time fusion device for dynamic video streams, the device includes a data preprocessing layer, a static registration layer and a real-time fusion rendering layer. The static registration layer is equipped with a LoFTR feature matching module, and the real-time fusion rendering layer includes a GPU acceleration unit and a WebGL rendering engine. The method includes the following steps: Multiple video streams are input and a 3D model is loaded in the data preprocessing layer; Decode multiple video streams to obtain video frame images and preprocess the video frame images; convert the 3D model into a triangular patch structure, read the UV coordinates of the 3D model and store the UV coordinates in the UV channel, extract the 2D key view, and establish a mapping table between the pixels of the 2D key view and the vertex coordinates of the 3D model. The LoFTR feature matching module is used to match feature points between the video frame image and the 2D key view to obtain a set of matching pairs. The target homography matrix of the matching pair set is solved by the improved RANSAC algorithm. Based on the target homography matrix, the pixel coordinates of the video frame image are mapped to the pixel coordinates in the 2D key view. The corresponding 3D point coordinates are obtained by querying the mapping table between the 2D key view pixels and the 3D model vertex coordinates. A perspective projection model is constructed based on the three-dimensional point coordinates and the corresponding video frame pixel coordinates. The perspective projection matrix is ​​obtained by solving the perspective projection matrix. The correspondence between the three-dimensional point coordinates and the video frame image pixel coordinates is established based on the perspective projection matrix. Combined with the UV coordinates of the three-dimensional model vertices, a spatial relationship between three-dimensional points, pixels, and UV is formed. The GPU acceleration unit establishes a mapping relationship between the pixel coordinates of the video frame image and the UV coordinates of the 3D model based on the spatial association, samples the video frame information to generate dynamic textures, and the WebGL rendering engine samples and renders the dynamic textures to output a real-time fused image of the video and the 3D scene.

2. The real-time fusion processing method for dynamic video streams based on a three-dimensional scene according to claim 1, characterized in that, Decode multiple video streams to obtain video frame images, and preprocess the video frame images, including: Decode multiple video streams to obtain video frame images, add UTC timestamps to the video frame images, and perform timing alignment based on the added UTC timestamps; The aligned video frame images are then subjected to image quality enhancement processing.

3. The real-time fusion processing method for dynamic video streams based on a three-dimensional scene according to claim 2, characterized in that, Adding UTC timestamps to the video frame images and performing timing alignment based on the added UTC timestamps includes: Obtain the UTC timestamp corresponding to each video stream and determine the target video frame image of each video stream based on the same fusion time. When the time difference between the UTC timestamp of the target video frame image and the fusion time is greater than a preset time difference threshold, the target video frame image is discarded or waited for; when the time difference is not greater than the preset time difference threshold, the target video frame image is used as input to the LoFTR feature matching module and the GPU acceleration unit.

4. The real-time fusion processing method for dynamic video streams based on a three-dimensional scene according to claim 1, characterized in that, The 3D model is converted into a triangular patch structure. The UV coordinates of the 3D model are read and stored in the UV channel. 2D key views are extracted, and a mapping table is established between the pixels of the 2D key views and the vertex coordinates of the 3D model, including: The 3D model is converted into a triangular patch structure, the UV coordinates of the converted 3D model are read, and the UV coordinates are stored in the UV channel. A spherical coordinate system is established with the geometric center of the 3D model as the origin. Multiple virtual camera positions are set in the spherical coordinate system according to the preset azimuth angle interval, and a 2D key view is generated by the renderer. Record the pixel coordinates of each 2D key view and the 3D point coordinates of the corresponding visible surface points in the renderer, and establish a mapping table between the pixels of the 2D key view and the vertex coordinates of the 3D model.

5. The real-time fusion processing method for dynamic video streams based on a three-dimensional scene according to claim 1, characterized in that, The LoFTR feature matching module performs feature point matching between the video frame image and the 2D key view to obtain a set of matching pairs, including: The video frame image and the 2D key view are scaled according to a preset input size and then input into the LoFTR model image encoder. The cosine similarity of all pixel position feature descriptors is calculated to obtain candidate matching pairs. The candidate matching pairs are threshold-filtered and output to form a matching pair set. Each matching pair in the matching pair set contains the pixel coordinates of the video frame image and the pixel coordinates of the 2D key view.

6. The real-time fusion processing method for dynamic video streams based on a three-dimensional scene according to claim 1, characterized in that, Solving the target homography matrix of the matching pair set using an improved RANSAC algorithm includes: Construct the local spatial neighborhood of the matching pair set and generate a seed point set; Random sampling consistency iteration is performed using the seed point set as the sampling object; For each iteration of the candidate homography matrix, the pixel coordinate error is calculated based on the reprojection model. The set of inliers that meet the preset error threshold is statistically analyzed, and the candidate homography matrix with the highest inlier rate is selected as the target homography matrix.

7. The real-time fusion processing method for dynamic video streams based on a three-dimensional scene according to claim 1, characterized in that, Based on the perspective projection matrix, a correspondence is established between the coordinates of 3D points and the pixel coordinates of video frame images. Combined with the UV coordinates of the 3D model vertices, a spatial relationship between 3D points, pixels, and UVs is formed, including: The three-dimensional point coordinates are projected back to the video frame image based on the perspective projection matrix to obtain two-dimensional coordinates. Establish a correspondence between the two-dimensional coordinates and the pixel coordinates of the video frame image to form a spatial association.

8. The real-time fusion processing method for dynamic video streams based on a three-dimensional scene according to claim 1, characterized in that, The GPU acceleration unit establishes a mapping relationship between the pixel coordinates of the video frame image and the UV coordinates of the 3D model based on the spatial association, and samples the video frame information to generate dynamic textures, including: Upload video frame images to the GPU texture cache and read the UV channel vertex UV coordinates and spatial relationships of the 3D model; Based on spatial correlation, determine the pixel coordinates of the UV channel vertex UV coordinates of the 3D model in the GPU texture cache, and perform bilinear interpolation sampling on the pixel coordinates to obtain the sampled pixel values. The sampled pixel values ​​are written into a dynamic texture cache object that matches the UV coordinates of the UV channel vertices of the 3D model to form a dynamic texture.

9. The real-time fusion processing method for dynamic video streams based on a three-dimensional scene according to claim 8, characterized in that, The WebGL rendering engine samples dynamic textures and outputs a blended image, including: The vertex coordinates of the 3D model are read by the vertex shader and the modelView projection transformation is performed. The dynamic texture is read by the fragment shader and texture sampling is performed based on the UV coordinates after bilinear interpolation. Input the light direction vector, normal vector, and line-of-sight vector into the Phong lighting model to calculate the color; Perform frustum culling in the renderer and output the blended image.

10. A real-time fusion processing system for dynamic video streams based on a three-dimensional scene, characterized in that, In a real-time fusion device for dynamic video streams, the device includes a data preprocessing layer, a static registration layer and a real-time fusion rendering layer. The static registration layer is equipped with a LoFTR feature matching module, and the real-time fusion rendering layer includes a GPU acceleration unit and a WebGL rendering engine. The system includes: The data loading module is used to access multiple video streams and load 3D models in the data preprocessing layer; The data processing module is used to decode multiple video streams to obtain video frame images and preprocess the video frame images; convert the 3D model into a triangular patch structure, read the UV coordinates of the 3D model and store the UV coordinates in the UV channel, extract the 2D key view, and establish a mapping table between the pixels of the 2D key view and the vertex coordinates of the 3D model. The static registration module is used to perform feature point matching between the video frame image and the 2D key view using the LoFTR feature matching module to obtain a set of matching pairs; the target homography matrix of the matching pair set is solved by the improved RANSAC algorithm; the pixel coordinates of the video frame image are mapped to the pixel coordinates in the 2D key view based on the target homography matrix; and the corresponding 3D point coordinates are obtained by querying the mapping table between the 2D key view pixels and the 3D model vertex coordinates. The spatial association module is used to construct a perspective projection model based on the three-dimensional point coordinates and the corresponding video frame pixel coordinates, solve for the perspective projection matrix, establish the correspondence between the three-dimensional point coordinates and the video frame image pixel coordinates based on the perspective projection matrix, and combine the three-dimensional model vertex UV coordinates to form a three-dimensional point-pixel-UV spatial association. The real-time fusion rendering module is used by the GPU acceleration unit to establish a mapping relationship between the pixel coordinates of the video frame image and the UV coordinates of the 3D model based on the spatial association, sample the video frame information to generate dynamic textures, and then use the WebGL rendering engine to sample and render the dynamic textures, and output the real-time fusion image of the video and the 3D scene.

Images