A method and device for three-dimensional video fusion calibration and real-time rendering

Abstract

The application discloses a three-dimensional video fusion calibration and real-time rendering method and device, and the method comprises the following steps: acquiring a video frame image, performing distortion correction, and saving the distortion parameters after the distortion correction; selecting a plurality of feature points in the video frame image and the picture position of a corresponding three-dimensional scene of the video frame image, respectively, and performing registration on the video frame image and the picture position of the three-dimensional scene; acquiring initial internal parameters of a camera for shooting the video, and obtaining optimal internal parameters through an optimization solving mode; determining the rotation and translation vectors of the camera relative to the origin of the world coordinate system based on the optimal internal parameters; determining the world coordinates and rotation angles of the camera in the three-dimensional scene, and projecting the image after the distortion correction onto the three-dimensional face sheet corresponding to the picture position of the three-dimensional scene in the form of texture projection. Even in the case of a large difference between internal parameter templates, the method can quickly find suitable internal parameters, and the accuracy of projection is ensured.

Description

Technical Field

[0001] This invention relates to the field of computer graphics, specifically to a method and apparatus for three-dimensional video fusion calibration and real-time rendering. Background Technology

[0002] In traditional video surveillance systems, monitoring personnel need to view multiple split-screen images, making it difficult to correlate fragmented video footage with their actual geographical locations. This hinders global real-time monitoring of large scenes, rapid retrieval of historical events, and quick video retrieval along specific inspection paths. 3D video fusion technology seamlessly projects real-time camera footage onto a 3D real-world or oblique photography model, stitching adjacent images together to create a higher-resolution image. This fusion process is unaffected by tilting or rotating the 3D model. Thus, in a 3D video fusion platform, video footage from various monitoring points can be seamlessly integrated and displayed with the 3D scene. This allows for rapid video retrieval and scheduling, and switching of video points based on the 3D scene location. This provides a more intuitive and convenient visual experience for both indoor fixed-point and outdoor large-scale video monitoring and path inspection.

[0003] The common method for fusing video footage from monitoring points with 3D scenes is through video texture mapping. This involves accurately projecting the distortion-corrected video texture onto the corresponding location in the 3D scene. This requires fusion calibration of the projection camera to determine its position and rotation angle.

[0004] In existing 3D video fusion calibration solutions, projecting real-time camera footage onto a 3D scene requires calculating and determining the position and orientation of the projecting camera. The accuracy of this calculation directly impacts the fusion precision with the 3D scene. Calibration methods include those based on camera intrinsic parameters, such as the Zhang Zhengyou calibration method. This method calculates camera intrinsic parameters and distortion coefficients by acquiring checkerboard images with real physical information. It is applicable to most scenarios and offers high accuracy. However, in engineering applications, this method is difficult to implement effectively in many scenarios, as the estimated camera intrinsic parameters have significant errors. Another calibration method is camera self-calibration, which does not require acquiring checkerboard images. Common examples include 3D reconstruction calibration methods based on multiple image sequences and camera self-calibration 3D reconstruction methods based on single images of vanishing points. Self-calibration methods can quickly estimate camera intrinsic parameters or directly estimate the camera's position and orientation, generally offering high efficiency. However, this type of calibration is difficult to guarantee accuracy in engineering applications and is only suitable for scenarios with low precision requirements.

[0005] In the process of fusion calibration in actual projects, differences in 3D scenes and modeling methods or technologies make it impossible to perfectly recreate the actual scene and model one-to-one. The projection of video textures onto the 3D scene itself introduces errors. At the same time, a key factor in texture projection is the calibration of the projection camera's extrinsic parameters, that is, determining the camera's position and orientation. The accuracy of the calibration directly affects the accuracy of the projection. Summary of the Invention

[0006] In view of this, the present invention provides a method and apparatus for three-dimensional video fusion calibration and real-time rendering, which can solve the technical problems that the intrinsic parameter calibration process is difficult to implement in engineering and the accuracy cannot be guaranteed when the intrinsic parameter template is used directly in the process of solving the position and attitude of the projection camera in the existing camera intrinsic parameter calibration.

[0007] To solve the above-mentioned technical problems, the present invention is implemented as follows.

[0008] This invention provides a method for three-dimensional video fusion calibration, comprising:

[0009] Step S11: Acquire video frame images, perform distortion correction on the radial distortion of the video frame images, and save the distortion parameters for distortion correction;

[0010] Step S12: Obtain the image position of the three-dimensional scene corresponding to the video frame image, select multiple feature points in the video frame image and the image position of the three-dimensional scene respectively, and register the video frame image and the image position of the three-dimensional scene.

[0011] Step S13: Obtain the initial intrinsic parameters of the camera that captured the video, and find the optimal intrinsic parameters through optimization; based on the optimal intrinsic parameters, determine the rotation and translation vectors of the camera relative to the origin of the world coordinate system; the initial intrinsic parameters are Fx, Fy, U0, V0, where Fx and Fy are the focal lengths of the camera in the X and Y directions, respectively, (U0, V0) is the center of the video frame image, and the initial value of (U0, V0) is (video frame image width * 0.5, video frame image height * 0.5);

[0012] Step S14: Determine the world coordinates and rotation angle of the camera in the 3D scene, and project the distortion-corrected image onto the 3D patch corresponding to the image position in the 3D scene using texture projection.

[0013] Preferably, after step S14, the method further includes:

[0014] Step S15: Determine the fusion method. If it is single-point fusion, perform cropping processing on the 3D patches corresponding to the image positions of the 3D scene; if it is multi-point fusion, perform cropping processing and color difference correction processing on the seams of the 3D patches corresponding to the image positions of the 3D scene.

[0015] Step S16: Save the processed 3D patch, establish a one-to-one correspondence between the vertex coordinates of the 3D patch and the texture coordinates of the distortion-corrected image, and save the 3D patch with the correspondence.

[0016] Among them, a 3D patch refers to an output file in obj or fbx format that stores a one-to-one correspondence between the coordinates of all vertices and the texture coordinates of the image after distortion correction.

[0017] Preferably, step S13, which involves finding the optimal intrinsic parameters through optimization, includes:

[0018] Step S131: Obtain multiple feature points in the video frame image, and multiple feature points in the image position of the three-dimensional scene corresponding to the video frame image; obtain the initial intrinsic parameters of the camera that captured the video;

[0019] Step S132: Based on the solvePnP algorithm, determine the rotation and translation vectors of the camera that captured the video;

[0020] Step S133: Determine the position of the camera in world coordinates and its rotation angle relative to the origin of the world coordinate system;

[0021] Step S134: Based on the position and rotation angle of the camera, determine the minimum reprojection error of multiple feature points in the image position of the 3D scene projected onto the video frame image;

[0022] Specifically, for each feature point in the image position of the three-dimensional scene: the world coordinates are converted into camera coordinates, and then the first error between the pixel coordinates in the video frame image and the pixel coordinates of the corresponding feature point in the video frame image is obtained according to the camera intrinsic parameters;

[0023] The sum of the squares of the first errors of all feature points is accumulated and then the square root is taken to obtain the L2 norm of all feature points. The mean of the L2 norms of all feature points is then calculated, which is the minimum reprojection error. The calculation formula is as follows:

[0024]

[0025] Where ξ represents the minimum reprojection error, n is the number of feature points, and i is the i-th feature point, (x i ,y i (x′) represents the pixel coordinates in the video frame image obtained by projection transformation of the i-th feature point. i y′ i (i) represents the pixel coordinates of the i-th feature point in the video frame image;

[0026] Step S135: If the difference between two consecutive minimum reprojection errors is less than 0.00001, the method ends; otherwise, update the initial intrinsic parameters of the camera and proceed to step S132.

[0027] The intrinsic parameters are updated by presetting a first moving step size Delta1, a second moving step size Delta2, and a moving direction, where the moving direction is either increasing or decreasing. If the current minimum reprojection error between two adjacent frames decreases or remains unchanged compared to the previous minimum reprojection error between two adjacent frames, the moving direction is kept unchanged, and the value of Delta1 is updated to Delta1*0.5, and the value of Delta2 is updated to Delta2*0.5. The values ​​of Fx and Fy are changed according to the moving direction, and the values ​​of U0 and V0 are changed according to the moving direction. If the current minimum reprojection error between two adjacent frames increases compared to the previous minimum reprojection error between two adjacent frames, the moving direction is changed, and the values ​​of Fx and Fy are changed according to the moving direction, and the values ​​of U0 and V0 are changed according to the moving direction.

[0028] Furthermore, Delta1 is initialized to 100, and Delta2 is initialized to 10.

[0029] This invention provides a method for real-time rendering of three-dimensional video. The method is based on the three-dimensional video fusion calibration method described above, and includes the following steps:

[0030] Step S21: Load the 3D scene corresponding to the video and crop the processed and saved 3D patches;

[0031] Step S22: Call the video decoder to decode the current video frame image in real time;

[0032] Step S23: Perform distortion correction processing on the current video frame image based on the distortion parameters;

[0033] Step S24: Use the current video frame image after distortion correction as the texture image to be bound, and replace the texture image bound to the 3D patch with the texture image to be bound;

[0034] Step S25: Based on the correspondence between the vertex coordinates of the three-dimensional patch and the texture coordinates of the bound texture image, update and render the merged image of the video in real time.

[0035] This invention provides a device for three-dimensional video fusion calibration, the device comprising:

[0036] Image acquisition module: configured to acquire video frame images, perform distortion correction on the radial distortion of the video frame images, and save the distortion parameters for distortion correction;

[0037] Registration module: configured to obtain the image position of the three-dimensional scene corresponding to the video frame image, select multiple feature points in the video frame image and the image position of the three-dimensional scene respectively, and register the video frame image and the image position of the three-dimensional scene;

[0038] Optimization module: Configured to obtain the initial intrinsic parameters of the camera capturing the video, and to find the optimal intrinsic parameters through optimization; based on the optimal intrinsic parameters, determine the rotation and translation vectors of the camera relative to the origin of the world coordinate system; the initial intrinsic parameters are Fx, Fy, U0, V0, where Fx and Fy are the focal lengths of the camera in the X and Y directions, respectively, (U0, V0) is the center of the video frame image, and the initial value of (U0, V0) is (video frame image width * 0.5, video frame image height * 0.5);

[0039] Mapping module: configured to determine the world coordinates and rotation angle of the camera in the 3D scene, and project the distortion-corrected image onto the 3D patch corresponding to the image position in the 3D scene using texture projection.

[0040] This invention provides an apparatus for real-time rendering of three-dimensional video. The apparatus is based on the device described above, and includes:

[0041] Loading module: configured to load the 3D scene corresponding to the video and crop the processed and saved 3D patches;

[0042] Calling module: Configured to call the video decoder to decode the current video frame image of the video in real time;

[0043] Distortion removal module: configured to perform distortion removal processing on the current video frame image based on the distortion parameters;

[0044] Update module: Configured to use the current video frame image after distortion correction as the texture image to be bound, and replace the texture image bound to the 3D patch with the texture image to be bound;

[0045] Rendering module: Configured to update and render the merged video frame in real time based on the correspondence between the vertex coordinates of the three-dimensional patch and the texture coordinates of the bound texture image.

[0046] Beneficial effects:

[0047] This invention employs a camera intrinsic parameter calibration optimization approach. Addressing the difficulty of implementing intrinsic parameter calibration, a set of intrinsic parameter templates was accumulated for different cameras during product development and implementation. First, an intrinsic parameter template is selected as the initial parameter. By choosing a feature point registration method, the rotation and translation vectors of the projecting camera are solved, and the reprojection error is also calculated. In other words, by dynamically adjusting the camera intrinsic parameter template parameters, the minimum reprojection error is optimized, achieving efficient 3D video fusion calibration. Then, the minimum reprojection error is optimized by automatically adjusting the camera intrinsic parameter template parameters. This scheme can quickly find suitable intrinsic parameters even when there are significant differences in the intrinsic parameter templates, ensuring projection accuracy. Simultaneously, based on the 3D patches bound to texture coordinates in the calibration output, a real-time update rendering process for the 3D video fusion image is provided.

[0048] It has the following technical effects:

[0049] (1) This invention does not overly rely on the accuracy of camera intrinsic parameters, but only estimates the camera intrinsic parameters through the physical properties of the camera or uses an intrinsic parameter template as the initial intrinsic parameters. By optimizing the camera intrinsic parameters to solve for the minimum reprojection error, the optimal solution of the intrinsic parameters can be quickly approximated. The calculated camera position and attitude accuracy is high. Therefore, the texture projection fusion accuracy is high and can meet the needs of indoor and outdoor, single-point and multi-point fusion.

[0050] (2) The entire fusion calibration process of this invention is simple and easy to operate, and can be implemented in different ways. Through calibration, distortion parameters and three-dimensional patches with bound texture coordinates are obtained. The three-dimensional video fusion platform can be easily adapted to complete the development of three-dimensional video fusion and the dynamic loading and updating of real-time video. It is not limited to specific platforms and has high compatibility and adaptability.

[0051] (3) The present invention can update the three-dimensional video fusion picture in real time.

[0052] (4) The present invention dynamically adjusts the camera intrinsic template parameters, starting from a large initial adjustment, gradually reducing the adjustment range, gradually approaching the optimal solution, until the reprojection error no longer decreases and the optimization process ends, and the solved parameters have high accuracy. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the method for three-dimensional video fusion calibration provided by the present invention;

[0054] Figure 2 This is a schematic diagram of the method for real-time rendering of three-dimensional video provided by the present invention;

[0055] Figure 3 A schematic diagram of the method flow for three-dimensional video fusion calibration and real-time rendering provided by the present invention;

[0056] Figure 4 A schematic diagram of the device structure for three-dimensional video fusion calibration provided by the present invention;

[0057] Figure 5 This is a schematic diagram of the device structure for real-time rendering of three-dimensional video provided by the present invention. Detailed Implementation

[0058] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0059] like Figure 1 As shown, this invention proposes a method for three-dimensional video fusion calibration, comprising the following steps:

[0060] Step S11: Acquire video frame images, perform distortion correction on the radial distortion of the video frame images, and save the distortion parameters for distortion correction;

[0061] Step S12: Obtain the image position of the three-dimensional scene corresponding to the video frame image, select multiple feature points in the video frame image and the image position of the three-dimensional scene respectively, and register the video frame image and the image position of the three-dimensional scene.

[0062] Step S13: Obtain the initial intrinsic parameters of the camera that captured the video, and find the optimal intrinsic parameters through optimization; based on the optimal intrinsic parameters, determine the rotation and translation vectors of the camera relative to the origin of the world coordinate system; the initial intrinsic parameters are Fx, Fy, U0, V0, where Fx and Fy are the focal lengths of the camera in the X and Y directions, respectively, (U0, V0) is the center of the video frame image, and the initial value of (U0, V0) is (video frame image width * 0.5, video frame image height * 0.5);

[0063] Step S14: Determine the world coordinates and rotation angle of the camera in the 3D scene, and project the distortion-corrected image onto the 3D patch corresponding to the image position in the 3D scene using texture projection.

[0064] Furthermore, the initial value of U0 is 0.5 times the width of the video frame image, and the initial value of V0 is 0.5 times the height of the video frame image.

[0065] Furthermore, after step S14, the method further includes:

[0066] Step S15: Determine the fusion method. If it is single-point fusion, perform cropping processing on the 3D patches corresponding to the image positions of the 3D scene; if it is multi-point fusion, perform cropping processing and color difference correction processing on the seams of the 3D patches corresponding to the image positions of the 3D scene.

[0067] Step S16: Save the processed 3D patch. Establish a one-to-one correspondence between the vertex coordinates of the 3D patch and the texture coordinates of the distortion-corrected image, and save the 3D patch with the correspondence. Here, a 3D patch refers to an output file in obj or fbx format that stores the one-to-one correspondence between all vertex coordinates and the texture coordinates of the distortion-corrected image. The 3D patch can be used as an exchange file, imported and edited in platforms such as 3DMAX or UE4, or rendered and displayed. Because the 3D patch stores the one-to-one correspondence between all vertex coordinates and the texture coordinates of the distortion-corrected image, the purpose of updating the blended image in the corresponding 3D scene can be achieved by changing the texture of the video frame image.

[0068] Specifically, when different corrected images are projected into a 3D scene, if they are isolated from each other with almost no overlap, they are considered as single-point fusion; if they overlap and are connected, they are considered as multi-point fusion.

[0069] Single-point fusion is commonly used for indoor surveillance video points. Each point is fused and projected separately, without needing to handle the relationship with other points. Only the projection of the image needs to be cropped accordingly.

[0070] Multi-point fusion is commonly used for continuous video surveillance points on outdoor roads, where adjacent points overlap. After projection, the seams need to be trimmed and aligned. Additionally, due to differences in color exposure and other factors between different surveillance images, color differences exist at the seams, requiring color correction processing.

[0071] Step S11: Acquire video frame images, process only the radial distortion of the video frame images, and perform distortion correction on the video frame images, wherein:

[0072] The distortion correction in this embodiment is used to remove distortions present in the video footage captured by the camera. Distortions generally include radial distortion and tangential distortion. In this embodiment, only radial distortion is considered, that is, only the first term k1 of the Taylor series coefficient and the scaling factor are considered. The corrected image is output for calibration, and the correction parameters k1 and the scaling factor are output.

[0073] Step S12: Obtain the 3D scene corresponding to the video frame image; select multiple feature points in both the video frame image and the 3D scene corresponding to the video frame image; and register the video frame image and the 3D scene corresponding to the video frame image, wherein:

[0074] The system navigates within the entire 3D scene, moving to the position in the 3D scene corresponding to the video frame image. The 3D scene is the entire 3D space, rendered and displayed by loading 3D scene maps and 3D models. Navigation within the 3D scene is possible by updating the position and pose of the viewpoint camera. The position in the 3D scene refers to the location where the video frame image captured by the real camera is fused and projected into the 3D scene. The position and pose of the virtual camera placed in the 3D scene coincide with the position and pose of the real camera; theoretically, the image observed by the virtual camera also coincides with the video frame image captured by the real camera. Multiple feature points are selected in both the video frame image and the corresponding 3D scene position using manual interaction or automatic image recognition. Each feature point selected in the video frame image corresponds one-to-one with the feature points selected in the corresponding 3D scene position. Based on the selected feature points, registration is performed between the video frame image and the corresponding 3D scene position.

[0075] Furthermore, the manual interaction method reduces the error in feature point selection by using auxiliary lines, scaling, etc.; the automatic image recognition method automatically captures the image position of the 3D scene corresponding to the video frame image to obtain a view, extracts and pairs uniformly distributed feature points from the view and the distortion-corrected video frame image, and selects the top N pairs of feature points with the highest confidence (N>=6). The feature points in the view are converted from screen space to world coordinates to obtain the corresponding 3D world coordinates.

[0076] Step S13, which involves finding the optimal intrinsic parameters through optimization, includes:

[0077] Step S131: Obtain multiple feature points in the video frame image, and multiple feature points in the image position of the three-dimensional scene corresponding to the video frame image; obtain the initial intrinsic parameters of the camera that captured the video;

[0078] Step S132: Based on the solvePnP algorithm, determine the rotation and translation vectors of the camera that captured the video;

[0079] Step S133: Determine the position of the camera in world coordinates and its rotation angle relative to the origin of the world coordinate system;

[0080] Step S134: Based on the position and rotation angle of the camera, determine the minimum reprojection error of multiple feature points in the image position of the 3D scene projected onto the video frame image;

[0081] Specifically, for each feature point in the image position of the three-dimensional scene: the world coordinates are converted into camera coordinates, and then the first error between the pixel coordinates in the video frame image and the pixel coordinates of the corresponding feature point in the video frame image is obtained according to the camera intrinsic parameters;

[0082] The sum of the squares of the first errors of all feature points is accumulated and then the square root is taken to obtain the L2 norm of all feature points. The mean of the L2 norms of all feature points is then calculated, which is the minimum reprojection error. The calculation formula is as follows:

[0083]

[0084] Where ξ represents the minimum reprojection error, n is the number of feature points, and i is the i-th feature point, (x i ,y i (x′) represents the pixel coordinates in the video frame image obtained by projection transformation of the i-th feature point. i y′ i (i) represents the pixel coordinates of the i-th feature point in the video frame image;

[0085] Step S135: If the difference between two consecutive minimum reprojection errors is less than 0.00001, the method ends; otherwise, update the initial intrinsic parameters of the camera and proceed to step S132.

[0086] The intrinsic parameters are updated by presetting a first moving step size Delta1, a second moving step size Delta2, and a moving direction, where the moving direction is either increasing or decreasing. If the current minimum reprojection error between two adjacent frames decreases or remains unchanged compared to the previous minimum reprojection error between two adjacent frames, the moving direction is kept unchanged, and the value of Delta1 is updated to Delta1*0.5, and the value of Delta2 is updated to Delta2*0.5. The values ​​of Fx and Fy are changed according to the moving direction, and the values ​​of U0 and V0 are changed according to the moving direction. If the current minimum reprojection error between two adjacent frames increases compared to the previous minimum reprojection error between two adjacent frames, the moving direction is changed, and the values ​​of Fx and Fy are changed according to the moving direction, and the values ​​of U0 and V0 are changed according to the moving direction.

[0087] Furthermore, the optimization solution is essentially an iterative optimization solution. In the first iteration, the difference between the two adjacent minimum reprojection errors and the previous two adjacent minimum reprojection errors are both randomly initialized. In the second iteration, the previous two adjacent minimum reprojection errors are also randomly initialized.

[0088] Delta1 is initialized to 100, and Delta2 is initialized to 10.

[0089] In this embodiment, the minimum reprojection error is optimized and used to adjust the intrinsic parameters of the camera that captures the video.

[0090] Camera intrinsic parameter optimization to find the minimum reprojection error involves iteratively solving for the camera's rotation and translation vectors based on selected feature point pairs and initial camera intrinsic parameters. Using these vectors, the camera's position and rotation angle in world coordinates are calculated. The minimum reprojection error is a quantitative estimation method that compares the pixel coordinates obtained by reprojecting the calculated camera rotation and translation vectors onto feature points in the 3D scene with the pixel coordinates of the marked feature points in the image.

[0091] Optionally, the camera's rotation and translation vectors can be calculated using OpenCV's solvePnP EPNP solution, PnP iterative solution, or PnPRansac iterative solution.

[0092] In this embodiment, the camera intrinsic template parameters are dynamically adjusted, starting with a relatively large adjustment and gradually decreasing the adjustment range to approach the optimal solution until the reprojection error no longer decreases, ending the optimization process. Using the optimized camera intrinsic parameters, the camera's rotation and translation vectors are calculated, and then the world coordinates and rotation angle of the projection camera in the 3D scene are solved.

[0093] Steps S14-S16 involve texture projection and binding output. Using the calculated projection camera position and rotation angle, the corrected image is projected onto a 3D patch corresponding to the image position in the 3D scene using texture projection. A one-to-one correspondence is established between the vertex coordinates of the 3D patch and the texture coordinates of the distortion-corrected image, and the 3D patch containing this correspondence is saved. The 3D patch refers to the output .obj or .fbx format file, which can be used as an exchange file for importing and editing in platforms such as 3ds Max or Unreal Engine 4, or for rendering and display. Since the 3D patch stores the one-to-one correspondence between all vertex coordinates and the texture coordinates of the distortion-corrected image, the blended image in the corresponding 3D scene can be updated by changing the texture of the video frame image.

[0094] Furthermore, the three-dimensional patch is created using modeling tools such as 3DMAX, and is in formats such as osg, osgb, and osgt. It fits the model in the three-dimensional scene and is larger than the projection screen area.

[0095] Furthermore, after processing, the vertex coordinates and texture coordinates of the three-dimensional patch are bound in a one-to-one correspondence, and the output format is obj or fbx, which can be further imported into platforms such as 3DMAX and UE4 for editing and use.

[0096] Optionally, after loading the 3D patch on other platforms, further editing processes such as cropping and stretching can be performed to make the projected texture blend more accurately with the 3D scene.

[0097] In this invention, video frames from camera locations are acquired, and image distortion correction is performed. The corresponding image positions are located in a 3D real-world scene or oblique photogrammetry model. Multiple feature points are selected for registration on the 3D scene and the corrected image. An initial camera intrinsic parameter template is selected, and the camera's rotation and translation vectors are calculated. The reprojection error of the 3D scene feature points is also calculated, and the camera intrinsic parameter template parameters are dynamically adjusted to optimize and minimize the reprojection error. The corrected image is then projected onto 3D patches in the 3D scene using texture projection, processed, and then texture-bound for output.

[0098] Optionally, a 3D scene or oblique photogrammetry model can be created by building a 3D scene at a 1:1 scale using 3D modeling software such as 3DMAX, or by using other tools to generate oblique photogrammetry data of the actual scene. Data formats supported include osg, osgb, osgt, ive, obj, fbx, etc. Oblique photogrammetry data supports LOD layering.

[0099] like Figure 2 As shown, the present invention provides a method for real-time rendering of three-dimensional video. The method includes the following steps.

[0100] Step S21: Load the 3D scene corresponding to the video and crop the processed and saved 3D patches;

[0101] Step S22: Call the video decoder to decode the current video frame image in real time;

[0102] Step S23: Perform distortion correction processing on the current video frame image based on the distortion parameters;

[0103] Step S24: Use the current video frame image after distortion correction as the texture image to be bound, and replace the texture image bound to the 3D patch with the texture image to be bound;

[0104] Step S25: Based on the correspondence between the vertex coordinates of the three-dimensional patch and the texture coordinates of the bound texture image, update and render the merged image of the video in real time.

[0105] In this embodiment, the 3D video fusion platform has 3D videos to be fused. The video fusion rendering process is as follows: load the 3D scene, load the 3D patch with bound texture coordinates output by calibration, read the camera video images of the monitoring points in real time, read the distortion correction parameters for distortion correction processing, and then use the corrected video frame images as the texture images bound to the 3D patch to update the texture coordinates, and update the 3D video fusion image dynamically in real time.

[0106] Optionally, a 3D video fusion platform can refer to various 3D visualization platforms, fusion platforms, or monitoring platforms that support the loading of 3D scenes, models, and video data.

[0107] Optionally, the 3D scene includes a 3D scene created using 3DMAX modeling tools, oblique photogrammetry data produced by the tools, etc., and supports formats such as osg, osgb, osgt, ive, obj, fbx, etc.

[0108] Optionally, the 3D patch is a fusion calibration output, and the supported formats are obj, fbx, osg, osgb, osgt, etc.

[0109] Optionally, the camera monitoring footage can be accessed in various ways, such as through camera device access SDK, REST API, Rtsp, Onvif, GB28181 and other protocol-based video streams.

[0110] Optionally, for texture updates and replacements, the 3D video fusion platform reads the video frame and updates the texture image bound to the 3D patch, without changing the correspondence between vertex coordinates and texture coordinates.

[0111] like Figure 3 As shown, this invention provides a 3D video fusion calibration method and a real-time rendering process. The left side illustrates the process and output of the 3D video fusion calibration method, while the right side shows the real-time rendering and updating process of the 3D video fusion within a 3D video fusion platform. Specifically, the "distortion correction parameter" is used as an output parameter, read during real-time decoding of video frames for distortion correction. The "3D patch" is the calibrated output model, loaded into the 3D video fusion platform, and the video fusion image is updated in real-time by updating the corresponding bound texture image.

[0112] The present invention also provides a device for three-dimensional video fusion calibration, such as... Figure 4 As shown, the device includes:

[0113] Image acquisition module: configured to acquire video frame images, perform distortion correction on the radial distortion of the video frame images, and save the distortion parameters for distortion correction;

[0114] Registration module: configured to obtain the image position of the three-dimensional scene corresponding to the video frame image, select multiple feature points in the video frame image and the image position of the three-dimensional scene respectively, and register the video frame image and the image position of the three-dimensional scene;

[0115] Optimization module: Configured to obtain the initial intrinsic parameters of the camera capturing the video, and to find the optimal intrinsic parameters through optimization; based on the optimal intrinsic parameters, determine the rotation and translation vectors of the camera relative to the origin of the world coordinate system; the initial intrinsic parameters are Fx, Fy, U0, V0, where Fx and Fy are the focal lengths of the camera in the X and Y directions, respectively, (U0, V0) is the center of the video frame image, and the initial value of (U0, V0) is (video frame image width * 0.5, video frame image height * 0.5);

[0116] Mapping module: configured to determine the world coordinates and rotation angle of the camera in the 3D scene, and project the distortion-corrected image onto the 3D patch corresponding to the image position in the 3D scene using texture projection.

[0117] The present invention also provides an apparatus for real-time rendering of three-dimensional video, such as... Figure 5 As shown, the device is implemented based on the aforementioned three-dimensional video fusion calibration device, and the device for real-time three-dimensional video rendering includes:

[0118] Loading module: configured to load the 3D scene corresponding to the video and crop the processed and saved 3D patches;

[0119] Calling module: Configured to call the video decoder to decode the current video frame image of the video in real time;

[0120] Distortion removal module: configured to perform distortion removal processing on the current video frame image based on the distortion parameters;

[0121] Update module: Configured to use the current video frame image after distortion correction as the texture image to be bound, and replace the texture image bound to the 3D patch with the texture image to be bound;

[0122] Rendering module: Configured to update and render the merged video frame in real time based on the correspondence between the vertex coordinates of the three-dimensional patch and the texture coordinates of the bound texture image.

[0123] The specific embodiments described above only illustrate the design principles of the present invention. The shapes and names of the components in this description may differ and are not limited. Therefore, those skilled in the art can modify or make equivalent substitutions to the technical solutions described in the foregoing embodiments; and these modifications and substitutions do not depart from the inventive spirit and technical solutions of the present invention, and should all fall within the protection scope of the present invention.

Claims

1. A method for three-dimensional video fusion calibration, characterized in that, The method comprises the following steps: Step S11: obtaining a video frame image, performing distortion correction on radial distortion of the video frame image, and saving the distortion parameter after the distortion correction; Step S12: obtaining a picture position of a three-dimensional scene corresponding to the video frame image, selecting a plurality of feature points in the video frame image and the picture position of the three-dimensional scene respectively, and registering the video frame image and the picture position of the three-dimensional scene; Step S13: obtaining an initial intrinsic parameter of a camera for shooting the video, solving an optimal intrinsic parameter by optimization, determining a rotation and translation vector of the camera relative to an origin of a world coordinate system based on the optimal intrinsic parameter; the initial intrinsic parameter is Fx, Fy, U0, V0, wherein Fx and Fy are focal lengths in X and Y directions of the camera respectively, (U0, V0) is a center of the video frame image, and an initial value of (U0, V0) is (video frame image width*0.5, video frame image height*0.5); Step S14: determining a world coordinate and a rotation angle of the camera in the three-dimensional scene, and projecting the image after the distortion correction onto a three-dimensional face corresponding to the picture position of the three-dimensional scene in a texture projection manner; In the step S13, the optimal intrinsic parameter is solved by optimization, comprising: Step S131: obtaining a plurality of feature points in the video frame image and a plurality of feature points in the picture position of the three-dimensional scene corresponding to the video frame image; and obtaining an initial intrinsic parameter of the camera for shooting the video; Step S132: determining a rotation and translation vector of the camera for shooting the video based on a solvePnP algorithm; Step S133: determining a position of the camera in the world coordinate and a rotation angle relative to the origin of the world coordinate system; Step S134: determining a minimum re-projection error of the plurality of feature points in the picture position of the three-dimensional scene projected onto the video frame image based on the position and the rotation angle of the camera; For each feature point in the picture position of the three-dimensional scene, the world coordinate is converted into a camera coordinate, and then a first error between a pixel coordinate in the video frame image and a pixel coordinate of a corresponding feature point in the video frame image is obtained according to the camera intrinsic parameter; The sum of squares of the first errors of all the feature points is accumulated to obtain an L2 norm of all the feature points, and then the L2 norm of all the feature points is averaged, that is, the minimum re-projection error, and the calculation formula is as follows: wherein, represents the minimum re-projection error, n is the number of feature points, i is the i-th feature point, (x , ) is the pixel coordinate of the i-th feature point in the video frame image calculated by the projection conversion, and (x , ) is the pixel coordinate of the i-th feature point in the video frame image. Step S135: if a difference between two adjacent minimum re-projection errors is less than 0.00001, the method ends; otherwise, the initial intrinsic parameter of the camera is updated, and the step S132 is entered. The updating manner of the internal reference is a preset first moving step Delta1, a second moving step Delta2 and a moving direction. The moving direction is increasing or decreasing. If the current adjacent two times minimum re-projection error is reduced or remains unchanged relative to the last adjacent two times minimum re-projection error, the moving direction remains unchanged, the value of Delta1 is updated as Delta1*0.5, the value of Delta2 is updated as Delta2*0.5, Fx and Fy are changed by the current value of Delta1 in the moving direction, and U0 and V0 are changed by the current value of Delta2 in the moving direction. If the current adjacent two times minimum re-projection error is increased relative to the last adjacent two times minimum re-projection error, the moving direction is changed, Fx and Fy are changed by the current value of Delta1 in the moving direction, and U0 and V0 are changed by the current value of Delta2 in the moving direction.

2. The method of claim 1, wherein, After the step S14, further comprising: Step S15: determining a fusion manner. If it is single-point fusion, performing cutting processing on the three-dimensional surface patch corresponding to the picture position of the three-dimensional scene. If it is multi-point fusion, performing cutting processing and color difference correction processing on the joint of the three-dimensional surface patch corresponding to the picture position of the three-dimensional scene. Step S16: saving the processed three-dimensional surface patch, establishing one-to-one correspondence between all vertex coordinates of the three-dimensional surface patch and texture coordinates of the image after distortion correction, and saving the three-dimensional surface patch with the correspondence. The three-dimensional surface patch refers to an output obj or fbx format file with one-to-one correspondence between all vertex coordinates and texture coordinates of the image after distortion correction.

3. The method of claim 2, wherein, The initial value of Delta1 is set as 100, and the initial value of Delta2 is set as 10.

4. A three-dimensional video real-time rendering method, which is realized based on the method of any one of claims 2-3, and comprises the following steps: Step S21: loading a three-dimensional scene corresponding to the video, and loading and processing the saved three-dimensional surface patch. Step S22: calling a video decoder to decode a current video frame image of the video in real time. Step S23: performing distortion correction on the current video frame image based on the distortion parameters. Step S24: taking the current video frame image after distortion correction as a to-be-bound texture picture, and replacing the texture picture bound by the three-dimensional surface patch with the to-be-bound texture picture. Step S25: based on the correspondence between each vertex coordinate of the three-dimensional surface patch and the texture coordinate of the bound texture picture, updating and rendering a fusion picture of the video in real time.

5. An apparatus for 3D video fusion calibration for performing the method of claims 2-3, characterized in that, The device comprises: An image acquisition module configured to acquire a video frame image, perform distortion correction on the radial distortion of the video frame image, and save distortion parameters after distortion correction; A registration module configured to acquire a picture position of a three-dimensional scene corresponding to the video frame image, select a plurality of feature points in the video frame image and the picture position of the three-dimensional scene respectively, and register the video frame image and the picture position of the three-dimensional scene. An optimization module is configured to obtain initial intrinsic parameters of a camera for shooting the video, to obtain optimal intrinsic parameters by optimization solving, and to determine rotation and translation vectors of the camera relative to an origin of a world coordinate system based on the optimal intrinsic parameters; the initial intrinsic parameters are Fx, Fy, U0, and V0, where Fx and Fy are focal lengths in X and Y directions of the camera, respectively, and (U0, V0) is a center of the video frame image, and an initial value of (U0, V0) is (video frame image width*0.5, video frame image height*0.5); A mapping module is configured to determine a world coordinate and a rotation angle of the camera in a three-dimensional scene, and to project the distortion-corrected image onto a three-dimensional face sheet corresponding to a picture position of the three-dimensional scene in a texture projection manner.

6. A device for real-time rendering of a three-dimensional video, which is implemented based on the device of claim 5 and comprises: A loading module configured to load a three-dimensional scene corresponding to the video and to load a saved three-dimensional face sheet; A calling module configured to call a video decoder to decode a current video frame image of the video in real time; A distortion removal module configured to perform distortion removal processing on the current video frame image based on the distortion parameters; An updating module configured to replace a texture picture bound to the three-dimensional face sheet with a to-be-bound texture picture, which is the current video frame image after the distortion removal processing; A rendering module configured to update a fusion picture of the video in real time based on a correspondence between vertex coordinates of the three-dimensional face sheet and texture coordinates of the bound texture picture.

Images