Video virtual-real fusion method and system, computer equipment and readable storage medium
By using neural network matching and nonlinear optimization techniques, self-calibration of real-scene cameras was achieved, solving the problem of high manpower and time costs in the camera calibration process in existing technologies, and improving the efficiency and accuracy of video virtual-real fusion.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGZHOU SHIYUAN ELECTRONICS CO LTD
- Filing Date
- 2024-10-25
- Publication Date
- 2026-04-28
AI Technical Summary
In existing video virtual-real fusion technologies, the camera calibration process requires manual disassembly and removal, resulting in high labor and time costs, which affects the accuracy of calibration results and the efficiency of video virtual-real fusion.
By employing a neural network-based matching method and nonlinear optimization technology, the system obtains the initial intrinsic and extrinsic parameters of the real-world camera, combines the feature matching relationship of the rendered image with the 3D point coordinates, and achieves self-calibration of the real-world camera. The system then fuses the video image and the rendered image in the GPU rendering pipeline.
It enables automatic calibration of real-scene cameras, improves the convenience and efficiency of video virtual-real fusion, meets the performance requirements of real-time rendering, and reduces manpower and time costs.
Smart Images
Figure CN121937672A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video virtual-real fusion, and in particular to a video virtual-real fusion method, system, computer device, and readable storage medium. Background Technology
[0002] With the development of GPU hardware and the advancement of image processing algorithms, video fusion technology has been widely applied in various industries. In fields such as film production, virtual reality, security monitoring, and smart inspection, video fusion technology can lower the barrier for users to understand a scene, increase the interactivity of the scene, obtain a more immersive experience, and at the same time help reduce labor costs for enterprises.
[0003] Video fusion technology refers to the fusion of image sequences captured by video acquisition devices with virtual scenes. Existing video fusion technology processes include... Figure 1 As shown, the first step is to reconstruct the 3D model of the scene. This can be done by acquiring images of the scene using a drone or handheld device, based on existing 3D reconstruction technology. Then, using camera calibration, the intrinsic parameters (i.e., focal length parameters, principal point coordinates) and extrinsic parameters (i.e., pose) of the camera in the rendering scene are obtained. The model-view matrix and projection matrix required for GPU rendering can be obtained from the intrinsic and extrinsic parameter information. From this, the texture coordinates corresponding to the vertices on the model surface are calculated. Then, the model is subjected to occlusion, penetration, and clipping processing to finally obtain the blended view.
[0004] Camera calibration in a virtual 3D scene is the first requirement for video fusion. However, in existing technologies, calibration boards are usually used for manual calibration. When the camera is installed in a special location or space is limited, manual calibration in its original position will be hindered and affect the accuracy of the calibration results. To ensure calibration accuracy and personnel safety, the camera needs to be disassembled and moved before manual calibration, which is labor and time costly and affects the efficiency of virtual-real video fusion. Summary of the Invention
[0005] Based on this, the purpose of the present invention is to provide a video virtual-real fusion method, system, computer device and readable storage medium, which has the advantages of realizing self-calibration of real scene camera and improving the efficiency of video virtual-real fusion.
[0006] A video virtual-real fusion method includes the following steps:
[0007] Acquire video images captured by a real-world camera, and acquire a rendered image from a preset 3D map whose viewpoint parameters differ from those of the video images within a preset range;
[0008] Based on a preset neural network matching method, the feature matching relationship between the video image and the rendered image is obtained;
[0009] Obtain the field of view, width, and height information of the rendered image, and calculate the intrinsic and extrinsic parameters of the rendering camera;
[0010] Based on the intrinsic and extrinsic parameters of the rendering camera, the rendered image is back-projected to obtain the 3D point coordinates corresponding to the pixels of the rendered image;
[0011] Obtain the initial intrinsic and extrinsic parameters of the real-view camera;
[0012] Based on the 3D point coordinates and the feature matching relationship, the initial intrinsic and extrinsic parameters are optimized using a nonlinear optimization method to obtain the intrinsic and extrinsic parameters of the real-scene camera;
[0013] Based on the intrinsic and extrinsic parameters of the real-world camera, the video image and the rendered image are fused in the GPU rendering pipeline to obtain a virtual-real fused image.
[0014] A video virtual-real fusion system, comprising:
[0015] The rendering image acquisition module is used to acquire video images captured by a real-world camera and to acquire a rendering image in a preset 3D map whose difference from the viewpoint parameters of the video images is within a preset range.
[0016] The feature matching module is used to obtain the feature matching relationship between the video image and the rendered image based on a preset neural network matching method.
[0017] The rendering camera intrinsic and extrinsic parameter acquisition module is used to acquire the field of view, width information, and height information of the rendered image, and calculate the rendering camera intrinsic and extrinsic parameters.
[0018] The back projection module is used to back project the rendered image based on the intrinsic and extrinsic parameters of the rendering camera to obtain the 3D point coordinates corresponding to the pixels of the rendered image.
[0019] An initial intrinsic and extrinsic parameter acquisition module is used to acquire the initial intrinsic and extrinsic parameters of the real-view camera;
[0020] The real-scene camera intrinsic and extrinsic parameter acquisition module is used to optimize the initial intrinsic and extrinsic parameters based on the 3D point coordinates and the feature matching relationship using a nonlinear optimization method to obtain the real-scene camera intrinsic and extrinsic parameters;
[0021] The video fusion module is used to fuse the video image and the rendered image in the GPU rendering pipeline based on the intrinsic and extrinsic parameters of the real-scene camera to obtain a virtual-real fused image.
[0022] A computer device includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the video virtual-real fusion method described above.
[0023] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the video virtual-real fusion method described above.
[0024] The video virtual-real fusion method described in this application first acquires a video image captured by a real-scene camera and a rendered image from a preset 3D map whose viewpoint parameters differ from the video image within a preset range. Then, a preset neural network matching method is used to obtain the feature matching relationship between the video image and the rendered image. Next, the field of view, width, and height information of the rendered image are acquired to calculate the rendering camera's intrinsic and extrinsic parameters. The rendered image is then back-projected to obtain 3D point coordinates. Next, the initial intrinsic and extrinsic parameters of the real-scene camera are acquired and optimized using the 3D point coordinates and the feature matching relationship to obtain the final virtual-real fusion image. Finally, based on the virtual-real fusion parameters, the video image and the rendered image are fused using the GPU rendering pipeline.
[0025] The video virtual-real fusion method described in this application combines the intrinsic and extrinsic parameters of the rendering camera corresponding to the rendered image and uses a nonlinear optimization method to calibrate the intrinsic and extrinsic parameters of the real-scene camera, realizing the self-calibration of the real-scene camera. It eliminates the need for a calibration board to calibrate the camera, improving the convenience of video virtual-real fusion. Furthermore, it uses a GPU rendering pipeline to fuse video images and rendered images, meeting the performance requirements of real-time rendering and improving the efficiency of virtual-real fusion video generation.
[0026] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description
[0027] Figure 1 This is a flowchart illustrating the steps of existing video fusion technologies.
[0028] Figure 2 This is a flowchart of the video virtual-real fusion method in one embodiment of this application;
[0029] Figure 3 This is a flowchart illustrating the steps for calculating the intrinsic and extrinsic parameters of the rendering camera in one embodiment of this application;
[0030] Figure 4 This is a flowchart of the steps for obtaining 3D point coordinates by back-projecting the rendered image in one embodiment of this application;
[0031] Figure 5 This is a flowchart illustrating the steps for obtaining preset intrinsic and extrinsic parameters of the real-world camera in one embodiment of this application.
[0032] Figure 6 This is a flowchart illustrating the steps of optimizing the initial intrinsic and extrinsic parameters to obtain the intrinsic and extrinsic parameters of a real-world camera using a nonlinear optimization method in one embodiment of this application.
[0033] Figure 7 This is a flowchart illustrating the steps of fusing video images and rendered images to obtain a virtual-real fused image based on the GPU rendering pipeline in one embodiment of this application.
[0034] Figure 8 This is a schematic diagram of the structure of a video virtual-real fusion system in one embodiment of this application;
[0035] Figure 9 This is a schematic diagram of a computer device used in the video virtual-real fusion method described in this application. Detailed Implementation
[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0037] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.
[0038] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."
[0039] Video fusion technology refers to the fusion of image sequences captured by video acquisition devices with virtual scenes. Existing video fusion technology processes include... Figure 1As shown, the first step is to reconstruct the 3D model of the scene. This can be done by acquiring images of the scene using a drone or handheld device, based on existing 3D reconstruction technology. Then, using camera calibration, the intrinsic parameters (i.e., focal length parameters, principal point coordinates) and extrinsic parameters (i.e., pose) of the camera in the rendering scene are obtained. The model-view matrix and projection matrix required for GPU rendering can be obtained from the intrinsic and extrinsic parameter information. From this, the texture coordinates corresponding to the vertices on the model surface are calculated. Then, the model is subjected to occlusion, penetration, and clipping processing to finally obtain the blended view.
[0040] Camera calibration in a virtual 3D scene is the first requirement for video fusion. However, in existing technologies, calibration boards are usually used for manual calibration. When the camera is installed in a special location or space is limited, manual calibration in its original position will be hindered and affect the accuracy of the calibration results. To ensure calibration accuracy and personnel safety, the camera needs to be disassembled and moved before manual calibration, which is labor and time costly and affects the efficiency of virtual-real video fusion.
[0041] In this regard, the embodiments of this application calculate the intrinsic and extrinsic parameters of the real-world camera based on a single rendering perspective, support a general fisheye camera model, and adopt a nonlinear optimization method to achieve automatic calibration of the real-world camera. The calibration process does not require manual calibration using a calibration board. After automatic calibration, pixel-level projection textures are used in real time in the GPU rendering pipeline to accurately render all pixels without performing lossy information processing on the original image, thereby achieving the fusion of video images and rendered images and improving the efficiency of virtual-real fusion video generation.
[0042] Please see Figure 2 , Figure 2 This is a flowchart illustrating the steps of a video virtual-real fusion method in one embodiment of this application. A video virtual-real fusion method includes the following steps:
[0043] S101, acquire video images captured by a real-world camera, and acquire a rendered image in a preset 3D map whose difference from the viewpoint parameters of the video images is within a preset range;
[0044] S102, Based on a preset neural network matching method, obtain the feature matching relationship between the video image and the rendered image;
[0045] S103, obtain the field of view, width and height information of the rendered image, and calculate the intrinsic and extrinsic parameters of the rendering camera;
[0046] S104, based on the intrinsic and extrinsic parameters of the rendering camera, back-project the rendered image to obtain the 3D point coordinates corresponding to the pixels of the rendered image;
[0047] S105, Obtain the initial intrinsic and extrinsic parameters of the real-view camera;
[0048] S106, Based on the 3D point coordinates and the feature matching relationship, the initial intrinsic and extrinsic parameters are optimized using a nonlinear optimization method to obtain the intrinsic and extrinsic parameters of the real-scene camera;
[0049] S107, based on the intrinsic and extrinsic parameters of the real-scene camera, the video image and the rendered image are fused in the GPU rendering pipeline to obtain a virtual-real fused image.
[0050] The video-real fusion method described in this application first acquires a video image captured by a real-world camera and a rendered image from a preset 3D map whose viewpoint parameters differ from the video image within a preset range. Then, a preset neural network matching method is used to obtain the feature matching relationship between the video image and the rendered image. Next, the field of view, width, and height information of the rendered image are acquired to calculate the rendering camera's intrinsic and extrinsic parameters. The rendered image is then back-projected to obtain 3D point coordinates. By acquiring the initial intrinsic and extrinsic parameters of the real-world camera and combining them with the 3D point coordinates and the feature matching relationship, the initial intrinsic and extrinsic parameters are optimized to obtain the final real-world camera intrinsic and extrinsic parameters. Finally, based on these parameters, the video image and the rendered image are fused in the GPU rendering pipeline to obtain a fused virtual-real image.
[0051] The video virtual-real fusion method described in this application combines the intrinsic and extrinsic parameters of the rendering camera corresponding to the rendered image and uses a nonlinear optimization method to calibrate the intrinsic and extrinsic parameters of the real-scene camera, realizing automatic calibration of the real-scene camera without the need for a calibration board, thus improving the convenience of video virtual-real fusion. Furthermore, it uses a GPU rendering pipeline to fuse video images and rendered images, meeting the performance requirements of real-time rendering and improving the efficiency of virtual-real fusion video generation.
[0052] For step S101, acquire the video image captured by the real-scene camera, and acquire the rendered image in the preset 3D map whose difference from the viewpoint parameters of the video image is within a preset range.
[0053] The real-scene camera is a camera with a high ability to reproduce real scenes. In this embodiment, the real-scene camera is a fisheye camera. In other embodiments, the real-scene camera may also be a camera using a wide-angle lens or an ultra-wide-angle lens.
[0054] The 3D map is a pre-constructed 3D map. In one embodiment, the 3D map is constructed based on the area where the real-view camera is located when taking the picture. In other embodiments, the real-view camera is set in the area corresponding to the 3D map to take pictures.
[0055] The viewpoint parameter is a specific value or indicator used to describe and quantify viewpoint characteristics. The difference in viewpoint parameters is the difference between the viewpoint parameters when the real-world camera captures the video image and the viewpoint parameters of the rendered image in the 3D map. The preset range is a pre-set range for the difference in viewpoint parameters. In this embodiment, by obtaining an image from the 3D map whose viewpoint parameter difference with the video image is within the preset range as the rendered image, the content of the rendered image and the video image is ensured to be identical, facilitating the fusion of the video image and the rendered image.
[0056] In this embodiment, the video image is captured by the real-view camera, and a rendered image with a difference of viewpoint parameters between the video image and the rendered image within a preset range is obtained from a preset 3D map. This results in a video image and a rendered image used for virtual-real fusion, ensuring that the video image and the rendered image belong to the same viewpoint and improving the accuracy of virtual-real fusion.
[0057] For step S102, based on a preset neural network matching method, the feature matching relationship between the video image and the rendered image is obtained;
[0058] The preset neural network matching method is an algorithm that is pre-set to perform feature matching on the video image and the rendered image. In one embodiment, the preset neural network method is the LoFTR method.
[0059] In this embodiment, the video image and the rendered image are input into the preset neural network matching method to obtain the feature matching relationship between the video image and the rendered image.
[0060] In one embodiment, after obtaining the feature matching relationship between the video image and the rendered image, the following steps are further included:
[0061] The RANSAC method is used to calculate the fundamental matrix. Based on the fundamental matrix, erroneous matching points in the feature matching relationship are filtered out, and the feature matching relationship is updated.
[0062] The RANSAC (Random Sample Consensus) method is an iterative method for estimating mathematical model parameters from a set of noisy data. The matching point is the point in the feature matching relationship where the video image and the rendered image match. In this embodiment, the RANSAC scheme is used to calculate the fundamental matrix of the feature matching relationship, thereby filtering out erroneous matching points and updating the feature matching relationship, ensuring its accuracy and improving the accuracy of video virtual-real fusion.
[0063] For step S103, the field of view, width information and height information of the rendered image are obtained, and the intrinsic and extrinsic parameters of the rendering camera are calculated;
[0064] The field of view angle describes the range of the camera's or observer's field of view, typically referring to a horizontal angle, but it can also refer to a vertical angle. In this embodiment, the field of view angle refers to the angle of the rendering camera corresponding to the rendered image in the 3D map. The width information is the pixel size or physical size of the rendered image in the horizontal direction, and the height information is the pixel size or physical size of the rendered image in the vertical direction.
[0065] The rendering camera intrinsic and extrinsic parameters are important parameters used in computer graphics and computer vision to describe how a camera captures the three-dimensional world and projects it onto a two-dimensional image plane. These parameters include intrinsic and extrinsic parameters. The intrinsic parameters describe the camera's own properties; in one embodiment, they include focal length, principal point coordinates, and / or distortion parameters. The extrinsic parameters describe the camera's position and orientation in the world coordinate system, i.e., the camera's pose. In this embodiment, the extrinsic parameters are the pose of the rendering camera corresponding to the rendered image in the three-dimensional map.
[0066] In this embodiment, the rendering camera's intrinsic and extrinsic parameters are calculated by obtaining the field of view, width, and height information corresponding to the rendered image from the 3D map.
[0067] Please see Figure 3 , Figure 3 This is a flowchart illustrating the steps for calculating the rendering camera's intrinsic and extrinsic parameters in one embodiment of this application. In one embodiment, calculating the rendering camera's intrinsic and extrinsic parameters includes the following steps:
[0068] S201, the focal length parameter of the rendering camera is calculated based on the field of view and the height information of the rendered image;
[0069] S202, the principal point coordinates of the rendering camera are calculated based on the width information and the height information;
[0070] S203, the focal length parameter and the principal point coordinates are used as the intrinsic and extrinsic parameters of the rendering camera.
[0071] For steps S201 to S203, the focal length parameter describes the distance from the optical center of the lens to the imaging plane. In this embodiment, the focal length parameter is calculated based on the field of view of the rendered image and the width information. For example, the focal length parameter is calculated using the formula f = h / (2 * tan(fov * 0.5)), where f is the focal length parameter, fov is the field of view, and h is the height information.
[0072] The principal point coordinates are the position of a reference point in the coordinate system of the rendering camera within the rendered image. In this embodiment, the principal point coordinates are calculated based on the width and height information of the rendered image. For example, using c... x =w*0.5;c y =h*0.5 to calculate the coordinates of the principal point, where c x ,c y Let w be the coordinates of the principal point, w be the width information, and h be the height information.
[0073] In this embodiment, the focal length parameter of the rendering camera is calculated based on the field of view angle and the width information of the rendered image, and the principal point coordinates of the rendering camera are calculated based on the width and height information of the rendered image. Finally, the focal length parameter and the principal point coordinates are used as the intrinsic and extrinsic parameters of the rendering camera.
[0074] In this embodiment, the calculation of the rendering camera's intrinsic and extrinsic parameters is determined based on the field of view, width, and height information of the rendered image, thereby improving the efficiency of the rendering camera's intrinsic and extrinsic parameter calculation.
[0075] For step S104, the rendered image is back-projected according to the intrinsic and extrinsic parameters of the rendering camera to obtain the 3D point coordinates corresponding to the pixels of the rendered image.
[0076] The back projection is an image reconstruction technique based on mathematical calculations. In this embodiment, the rendered image is back projected by combining the intrinsic and extrinsic parameters of the rendering camera, thereby obtaining the 3D point coordinates corresponding to the pixels of the rendered image.
[0077] Please see Figure 4 , Figure 4 This is a flowchart illustrating the steps of back-projecting the rendered image to obtain 3D point coordinates in one embodiment of this application. In one embodiment, back-projecting the rendered image to obtain 3D point coordinates based on the intrinsic and extrinsic parameters of the rendering camera includes the following steps:
[0078] S301, Obtain the 2D image pixel coordinates of the rendered image;
[0079] S302, Obtain the depth map of the rendered image, and obtain the depth information corresponding to the pixel coordinates of the 2D image based on the depth map;
[0080] S303, the 3D point coordinates are calculated based on the 2D image pixel coordinates, the depth information, and the rendering camera intrinsic and extrinsic parameters.
[0081] For steps S301 to S303, the 2D image pixel coordinates are used to locate the position of pixels in the rendered image. In one embodiment, a coordinate system is established with the upper left corner of the rendered image as the origin, the positive x-axis to the right, and the positive y-axis downward to obtain the pixel coordinates in the rendered image.
[0082] The depth map is an image representation that records the distance information from each point in the scene to the camera. In this embodiment, based on the three-dimensional map, the depth map corresponding to the rendered image is obtained and used to calculate the 3D point coordinates for real-world camera calibration.
[0083] In this embodiment, the 3D point coordinates are obtained by acquiring the 2D image pixel coordinates of the rendered image and calculating them using the intrinsic and extrinsic parameters of the rendering camera. For example, the formula is used. P z =z; calculate to obtain the coordinates of the 3D point, where P x ,P y ,P z The 3D point coordinates are (u, v), and the 2D image pixel coordinates are c. x ,c y The coordinates are the main points, and z represents the depth information obtained from the depth map corresponding to the pixel coordinates of the 2D image.
[0084] In this embodiment, by obtaining the 2D image pixel coordinates of the rendered image and combining them with the corresponding depth information in the depth map, back projection is performed based on the rendering camera's intrinsic and extrinsic parameters, and the 3D point coordinates are calculated. The 3D point coordinates are then used to calibrate the intrinsic and extrinsic parameters of the real-world camera.
[0085] For step S105, obtain the initial intrinsic and extrinsic parameters of the real-scene camera;
[0086] The initial intrinsic and extrinsic parameters are set based on the rendered image and the video image to obtain the intrinsic and extrinsic parameters for camera calibration.
[0087] Please see Figure 5 , Figure 5 This is a flowchart illustrating the steps for obtaining preset intrinsic and extrinsic parameters of the real-view camera in one embodiment of this application. In one embodiment, obtaining the initial intrinsic and extrinsic parameters of the real-view camera includes the following steps:
[0088] S401, obtain the pose of the rendering camera corresponding to the rendered image in the 3D map, and set the pose as the extrinsic parameter of the real-world camera;
[0089] S402, Obtain the width and height information of the video image, and calculate the initial principal point coordinates;
[0090] S403, within the preset camera focal length parameter range, samples with a preset sampling length to obtain the initial focal length parameter;
[0091] S404, the extrinsic parameters of the real-view camera, the initial principal point coordinates, and the initial focal length parameters are used as the preset extrinsic and extrinsic parameters of the real-view camera;
[0092] S405, the preset intrinsic and extrinsic parameters of the real-view camera are adjusted using the PNP algorithm to obtain the initial intrinsic and extrinsic parameters of the real-view camera.
[0093] For steps S401 to S405, the pose refers to the position and orientation of the rendering camera corresponding to the rendered image in the 3D map. Since the difference between the viewpoint parameters of the rendered image and the video image is within a preset range, the pose of the rendering camera corresponding to the rendered image is similar to the pose of the real-world camera. Therefore, in this embodiment, by obtaining the pose of the rendering camera in the 3D map, the pose is used as the extrinsic parameter of the real-world camera.
[0094] The initial principal point coordinates are calculated based on the width and height information of the video image, resulting in the principal point coordinates corresponding to the real-scene camera. In one embodiment, c is used. x =w*0.5;c y =h*0.5 to calculate the coordinates of the principal point, where c x ,c y Let w be the coordinates of the principal point, w be the width information, and h be the height information.
[0095] The initial focal length parameter is the focal length information corresponding to the real-world camera. The preset camera focal length parameter range is a pre-set range for sampling. The preset sampling length is a pre-set sampling length. In this embodiment, the initial focal length parameter is obtained by sampling within the preset camera focal length parameter range within the preset sampling length. For example, the preset camera focal length range is set to 1000 to 10000, and the sampling length is 1000, that is, between 1000 and 10000, sampling is performed once every 1000 to obtain the initial focal length parameter.
[0096] The preset intrinsic and extrinsic parameters are the intrinsic and extrinsic parameters of the real-scene camera that are set in advance. The PNP (Perspective-n-Point) algorithm is a pose estimation method widely used in computer vision. It is a method for solving the motion of 3D to 2D point pairs and is used to estimate the camera pose. In this embodiment, by obtaining the preset intrinsic and extrinsic parameters of the real-scene camera and further adjusting the preset intrinsic and extrinsic parameters using the PNP algorithm, the initial intrinsic and extrinsic parameters are obtained and used for the calibration of the real-scene camera, ensuring the accuracy of the camera calibration.
[0097] In this embodiment, the preset intrinsic and extrinsic parameters are obtained by acquiring the extrinsic parameters of the real-view camera, the initial principal point coordinates, and the initial focal length parameters. The initial intrinsic and extrinsic parameters are then adjusted using the PNP algorithm and used to further calibrate the intrinsic and extrinsic parameters of the real-view camera, thereby improving the accuracy and efficiency of camera calibration.
[0098] For step S106, the initial intrinsic and extrinsic parameters are optimized using a nonlinear optimization method based on the 3D point coordinates and the feature matching relationship to obtain the intrinsic and extrinsic parameters of the real-scene camera.
[0099] The nonlinear optimization is a scheme to optimize the initial intrinsic and extrinsic parameters by finding the optimal solution in a nonlinear function space. In one embodiment, the nonlinear optimization method includes constructing a reprojection error function. The intrinsic and extrinsic parameters of the real-scene camera are the intrinsic and extrinsic parameters obtained by calibrating the real-scene camera. In this embodiment, the initial intrinsic and extrinsic parameters are optimized by combining the 3D point coordinates and the feature matching relationship, thereby obtaining the intrinsic and extrinsic parameters of the real-scene camera.
[0100] Please see Figure 6 , Figure 6 This document presents a flowchart illustrating the steps of optimizing the initial intrinsic and extrinsic parameters using a nonlinear optimization method to obtain the intrinsic and extrinsic parameters of the real-world camera in one embodiment of this application. In one embodiment, the optimization of the initial intrinsic and extrinsic parameters using a nonlinear optimization method based on the 3D point coordinates and the feature matching relationship to obtain the real-world camera intrinsic and extrinsic parameters includes the following steps:
[0101] S501, Construct a fisheye camera KB model based on the 3D point coordinates;
[0102] S502, combine the fisheye camera KB model with the feature matching relationship and the 3D point coordinates to construct a reprojection error function;
[0103] S503, the initial intrinsic and extrinsic parameters are input into the reprojection error function using nonlinear optimization to perform iterative optimization calculation to obtain the reprojection error. When the reprojection error meets the preset threshold, the initial intrinsic and extrinsic parameters are used as the intrinsic and extrinsic parameters of the real-scene camera. The initial intrinsic and extrinsic parameters are obtained through a preset sampling method and include multiple sets of initial intrinsic and extrinsic parameters.
[0104] For steps S501 to S503, the fisheye camera KB model is a camera model describing the imaging process of a fisheye camera. In this embodiment, the fisheye camera KB model is obtained by constructing a Kannala-Brandt model including four distortion parameters. In one embodiment, the fisheye camera KB model is:
[0105]
[0106] θ d =k1θ+k2θ 3 +k3θ 5 +k4θ 7
[0107]
[0108] Among them, P x ,P y ,P z Represents 3D point coordinates, u, v represent pixel coordinates projected onto the image through the KB4 model, c x ,c y Let f be the principal point coordinates of the real-view camera. x ,f y The focal length parameter of the real-view camera is obtained based on the initial intrinsic and extrinsic parameters in this embodiment, where the principal point coordinates of the real-view camera and the focal length parameter are obtained.
[0109] The reprojection error function is a function constructed by combining the fisheye camera KB model with feature matching relationships to optimize the initial intrinsic and extrinsic parameters using the method of minimizing reprojection error. In one embodiment, the reprojection error function is constructed as argmin∑‖π(T_c_r·P i )-x i || 2 Where π represents the fisheye camera KB model, T_c_r represents the pose of the rendered frame relative to the camera, and P i x is the 3D point obtained by backprojection according to step S104. i The x is the pixel coordinate matched by the 3D point in the video image. In this embodiment, the x iIt is the pixel coordinates of the 3D point in the video image obtained based on the feature matching relationship.
[0110] In this embodiment, the initial intrinsic and extrinsic parameters are obtained through steps S401 to S405. The initial focal length parameters of the real-view camera are sampled using a preset sampling length to obtain multiple sets of initial focal length parameters. Multiple sets of initial intrinsic and extrinsic parameters of the real-view camera are calculated by inputting these multiple sets of initial focal length parameters into the fisheye camera KB model.
[0111] The reprojection error is the error obtained by optimizing the initial intrinsic and extrinsic parameters according to the loss function. The preset threshold is a pre-set threshold for whether the reprojection error meets the conditions. In this embodiment, the initial intrinsic and extrinsic parameters are input into the reprojection error function by using a nonlinear optimization method to perform iterative calculation to obtain the reprojection error. The initial intrinsic and extrinsic parameters that satisfy the preset threshold are obtained as the intrinsic and extrinsic parameters of the real-scene camera.
[0112] Based on the fisheye camera KB model, a reprojection error function is constructed by combining the feature matching relationship and the 3D point coordinates. This function is used to calculate the intrinsic and extrinsic parameters of the real-scene camera. In this embodiment, the calculation of the intrinsic and extrinsic parameters of the real-scene camera does not require the use of a calibration board. By combining rendered images and video images, automatic camera calibration is achieved, which improves the accuracy and efficiency of camera calibration, reduces deployment costs, and expands the scope of application.
[0113] In one embodiment, re-acquiring the initial intrinsic and extrinsic parameters and inputting them into the loss function to calculate the reprojection error further includes the following steps:
[0114] If the reprojection error calculated by the reprojection error function based on the initial intrinsic and extrinsic parameters does not meet the preset threshold, then the difference between the video image viewpoint parameter and the reprojection error is within a preset range, and the rendering viewpoint is reselected, and the rendered image in the 3D map is re-acquired based on the reselected rendering viewpoint.
[0115] In this embodiment, when the reprojection error obtained by iterative optimization of the initial intrinsic and extrinsic parameters input into the reprojection error function using nonlinear optimization does not meet the preset threshold, it indicates that the optimization of the initial intrinsic and extrinsic parameters has failed and the intrinsic and extrinsic parameters of the real-scene camera cannot be determined. Then, the rendering perspective is reselected within the preset range when the difference between the rendering perspective parameter and the video image is within the preset range, and the rendered image in the 3D map is re-acquired based on the reselected rendering perspective.
[0116] In this embodiment, by judging the reprojection error corresponding to the initial intrinsic and extrinsic parameters, the problem of the optimization process failing to converge to a solution that meets the preset threshold is solved, thus avoiding the situation where camera calibration cannot be performed due to solution failure, resulting in the failure to fuse video images and rendered images.
[0117] S107, based on the intrinsic and extrinsic parameters of the real-scene camera, the video image and the rendered image are fused using the GPU rendering pipeline to obtain a virtual-real fused image.
[0118] The RGB image is a color model that generates a range of colors by combining different intensities of red, green, and blue. The GPU rendering pipeline is a series of independent parallel processing units within the GPU that process image signals.
[0119] In this embodiment, by acquiring the depth map and RGB map of the rendered image, and combining them with the intrinsic and extrinsic parameters of the real-world camera, the video image and the rendered image are fused in real time using a GPU rendering pipeline to obtain the fused image.
[0120] Please see Figure 7 , Figure 7 This is a flowchart illustrating the steps of fusing video images and rendered images based on the GPU rendering pipeline to obtain a virtual-real fused image in one embodiment of this application. In one embodiment, fusing video images and rendered images based on the GPU rendering pipeline to obtain a virtual-real fused image includes the following steps:
[0121] S601, Based on the intrinsic and extrinsic parameters of the real-scene camera, project the 3D point coordinates onto the video image to obtain the projection position information;
[0122] S602, obtain the depth map corresponding to the rendered image, and filter out the projection points in the projection position information that are not within the field of view or have negative depth values based on the depth map to obtain the target projection points for video fusion.
[0123] S603, based on the target projection point, obtain the RGB color information of the 3D point coordinates in the video image, obtain the pixel position of the 3D point coordinates in the rendered image, and replace the color information of the coordinate pixel position of the 3D point in the rendered image with the RGB color information corresponding to the 3D point coordinates.
[0124] S604, after processing all 3D points, the video image and the rendered image are fused to obtain a virtual-real fused video.
[0125] For steps S601 to S604, the projection position information is obtained by projecting the 3D point coordinates onto the video image, specifically the position information of the projection point. In one embodiment, p_c = π(T_c_k·P_k) is used to project the 3D point coordinates onto the video image, where p_c is the 2D pixel position of the video image, π is the fisheye camera KB model, P_k is the 3D point coordinate corresponding to the 2D pixel position, and T_c_k is the pose of the real-scene camera relative to the rendered image. In one embodiment... Wherein, T_w_c is obtained based on the intrinsic and extrinsic parameters of the real-world camera, and T_w_k is the pose corresponding to the rendered image.
[0126] In this embodiment, based on the real-scene camera intrinsic and extrinsic parameters obtained from the automatic camera calibration in steps S101 to S106, the 3D point coordinates are projected onto the video image to obtain the projection position information. Then, based on the depth map corresponding to the rendered image obtained from the 3D map, projection points that are not within the field of view or have negative depth values are filtered out from the projection position information to obtain target projection points for video fusion. Finally, the color information of the coordinate pixel positions corresponding to the 3D points is replaced with the RGB color information corresponding to the 3D point coordinates in the rendered image. After replacing the RGB color information of all target projection points corresponding to the 3D points, the video image and the rendered image are fused to obtain the virtual-real fused image.
[0127] In one embodiment, projecting the 3D point coordinates onto the video image includes the following steps:
[0128] The video image is processed to remove distortion using a preset distortion model to obtain the projection position information;
[0129] The preset distortion model is a pre-set model used to remove distortions from the video image. In this embodiment, the preset distortion model is used to remove distortion information from the video image to obtain the projection position information, avoiding interference from distortion information in the video image and improving the accuracy of video fusion.
[0130] In this embodiment, the fusion of video images and rendered images is directly calculated in real time using pixel-level projected textures in the GPU rendering pipeline, accurately rendering all pixels and making better use of scene structure information, thus achieving a better fusion effect.
[0131] The video virtual-real fusion method described in this application first acquires a video image captured by a real-scene camera and a rendered image from a preset 3D map whose viewpoint parameters differ from the video image within a preset range. A preset neural network matching method is used to obtain the feature matching relationship between the video image and the rendered image. Then, the field of view, width, and height information of the rendered image are acquired to calculate the rendering camera's intrinsic and extrinsic parameters. The rendered image is then back-projected to obtain 3D point coordinates. Next, preset intrinsic and extrinsic parameters of the real-scene camera are acquired, and initial intrinsic and extrinsic parameters of the real-scene camera are obtained using a PNP algorithm. Furthermore, by constructing a fisheye camera KB model, combining the feature matching relationship and 3D point coordinates, a loss function is constructed using a method that minimizes the reprojection error. The initial intrinsic and extrinsic parameters are input into the loss function to calculate the reprojection error. Based on the reprojection error, the initial intrinsic and extrinsic parameters are iteratively optimized using nonlinear optimization. Initial intrinsic and extrinsic parameters whose reprojection error satisfies a preset threshold are obtained as the real-scene camera's intrinsic and extrinsic parameters. Finally, based on the real-scene camera's intrinsic and extrinsic parameters, the video image and the rendered image are fused using the GPU rendering pipeline to obtain a virtual-real fused image.
[0132] The video virtual-real fusion method described in this application adopts a fisheye camera KB model, combined with the intrinsic and extrinsic parameters of the rendering camera corresponding to the rendered image, and uses the nonlinear optimization method to calibrate the intrinsic and extrinsic parameters of the real-scene camera, realizing the self-calibration of the real-scene camera. It eliminates the need for a calibration board for camera calibration, improving the convenience of video virtual-real fusion. Furthermore, it adopts a GPU rendering pipeline to fuse video images and rendered images, meeting the performance requirements of real-time rendering and improving the efficiency of virtual-real fusion video generation.
[0133] Please see Figure 8 , Figure 8 This is a schematic diagram of a video virtual-real fusion system in one embodiment of this application. This application also provides a video virtual-real fusion system, including:
[0134] The rendering image acquisition module 11 is used to acquire video images captured by a real-world camera and to acquire a rendering image in a preset 3D map whose difference from the viewpoint parameters of the video images is within a preset range.
[0135] The feature matching module 12 is used to obtain the feature matching relationship between the video image and the rendered image based on a preset neural network matching method.
[0136] The rendering camera intrinsic and extrinsic parameter acquisition module 13 is used to acquire the field of view, width information and height information of the rendered image, and calculate the rendering camera intrinsic and extrinsic parameters;
[0137] The back projection module 14 is used to back project the rendered image according to the intrinsic and extrinsic parameters of the rendering camera to obtain the 3D point coordinates corresponding to the pixels of the rendered image.
[0138] The initial intrinsic and extrinsic parameter acquisition module 15 is used to acquire the initial intrinsic and extrinsic parameters of the real-scene camera;
[0139] The real-scene camera intrinsic and extrinsic parameter acquisition module 16 is used to optimize the initial intrinsic and extrinsic parameters according to the 3D point coordinates and the feature matching relationship, and to obtain the real-scene camera intrinsic and extrinsic parameters by nonlinear optimization.
[0140] The video fusion module 17 is used to fuse the video image and the rendered image in the GPU rendering pipeline based on the intrinsic and extrinsic parameters of the real-scene camera to obtain a virtual-real fused image.
[0141] It should be noted that the video virtual-real fusion system provided in the above embodiments is only illustrated by the division of the above functional modules when executing the video virtual-real fusion method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The video virtual-real fusion system provided in the above embodiments is used to execute the video virtual-real fusion method described in the above embodiments. Its operation method and principle are the same as the video virtual-real fusion method described above. That is, the video virtual-real fusion system and the video virtual-real fusion method provided in the above embodiments belong to the same concept. The implementation process is detailed in the above method embodiments and will not be repeated here.
[0142] Please see Figure 9 , Figure 9 This is a schematic diagram of a computer device for a video virtual-real fusion method in an embodiment of this application. The computer device 21 includes: a control device 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the control device 211, such as a video virtual-real fusion program; the control device 211 executes the computer program 213 to implement the video virtual-real fusion method described in the above embodiment.
[0143] The control device 211 includes a processor, which may include one or more processing cores. The processor connects to various parts within the computer device 21 using various interfaces and lines. It executes various functions of the computer device 21 and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 212, and by calling data stored in memory 212. Optionally, the processor can be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor can integrate one or more of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required to be displayed on the touch screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.
[0144] The memory 212 may include random access memory (RAM) or read-only memory. Optionally, the memory 212 may include non-transitory computer-readable storage medium. The memory 212 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 212 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch instructions), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory 212 may also be at least one storage device located remotely from the aforementioned processor.
[0145] This application also provides a readable storage medium that can store multiple instructions. These instructions are applicable to being loaded and executed by a control device using the method steps described in the above embodiments. For details of the execution process, please refer to the specific description of the above embodiments, which will not be repeated here.
[0146] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.
Claims
1. A video virtual-real fusion method, characterized in that, Includes the following steps: Acquire video images captured by a real-world camera, and acquire a rendered image from a preset 3D map whose viewpoint parameters differ from those of the video images within a preset range; Based on a preset neural network matching method, the feature matching relationship between the video image and the rendered image is obtained; Obtain the field of view, width, and height information of the rendered image, and calculate the intrinsic and extrinsic parameters of the rendering camera; Based on the intrinsic and extrinsic parameters of the rendering camera, the rendered image is back-projected to obtain the 3D point coordinates corresponding to the pixels of the rendered image; Obtain the initial intrinsic and extrinsic parameters of the real-view camera; Based on the 3D point coordinates and the feature matching relationship, the initial intrinsic and extrinsic parameters are optimized using a nonlinear optimization method to obtain the intrinsic and extrinsic parameters of the real-scene camera; Based on the intrinsic and extrinsic parameters of the real-world camera, the video image and the rendered image are fused in the GPU rendering pipeline to obtain a virtual-real fused image.
2. The video virtual-real fusion method according to claim 1, characterized in that, The process of optimizing the initial intrinsic and extrinsic parameters based on the 3D point coordinates and the feature matching relationship using a nonlinear optimization method to obtain the intrinsic and extrinsic parameters of the real-scene camera includes the following steps: Based on the 3D point coordinates, construct the fisheye camera KB model; The fisheye camera KB model is combined with the feature matching relationship and the 3D point coordinates to construct a reprojection error function; The initial intrinsic and extrinsic parameters are input into the reprojection error function to calculate the reprojection error; The initial intrinsic and extrinsic parameters are input into the reprojection error function using the nonlinear optimization method to perform iterative optimization calculation to obtain the reprojection error. When the reprojection error meets the preset threshold, the initial intrinsic and extrinsic parameters are used as the intrinsic and extrinsic parameters of the real-scene camera. The initial intrinsic and extrinsic parameters are obtained through a preset sampling method and include multiple sets of initial intrinsic and extrinsic parameters.
3. The video virtual-real fusion method according to claim 2, characterized in that, The step of using the nonlinear optimization to input the initial intrinsic and extrinsic parameters into the reprojection error function for iterative optimization calculation to obtain the reprojection error further includes the following steps: If the reprojection error calculated by the reprojection error function based on the initial intrinsic and extrinsic parameters does not meet the preset threshold, then the difference between the video image viewpoint parameter and the reprojection error is within a preset range, and the rendering viewpoint is reselected, and the rendered image in the 3D map is re-acquired based on the reselected rendering viewpoint.
4. The video virtual-real fusion method according to claim 1, characterized in that, Based on the field of view, width, and height information, the intrinsic and extrinsic parameters of the rendering camera are determined, including the following steps: The focal length parameters of the rendering camera are calculated based on the field of view and the height information of the rendered image. The principal point coordinates of the rendering camera are calculated based on the width and height information. The focal length parameter and the principal point coordinates are used as intrinsic and extrinsic parameters of the rendering camera.
5. The video virtual-real fusion method according to claim 1, characterized in that, The process of back-projecting the rendered image to obtain 3D point coordinates includes the following steps: Obtain the 2D image pixel coordinates of the rendered image; Obtain the depth map of the rendered image, and obtain the depth information corresponding to the pixel coordinates of the 2D image based on the depth map; The coordinates of the 3D point are calculated based on the pixel coordinates of the 2D image, the depth information, and the intrinsic and extrinsic parameters of the rendering camera.
6. The video virtual-real fusion method according to claim 1, characterized in that, Obtaining the initial intrinsic and extrinsic parameters of the real-view camera includes the following steps: Obtain the pose of the rendering camera corresponding to the rendered image in the 3D map, and set the pose as the extrinsic parameter of the real-world camera; Obtain the width and height information of the video image, and calculate the initial principal point coordinates; Within the preset camera focal length parameter range, sampling is performed with a preset sampling length to obtain the initial focal length parameter; The extrinsic parameters of the real-view camera, the initial principal point coordinates, and the initial focal length parameters are used as the preset extrinsic and extrinsic parameters of the real-view camera. The initial intrinsic and extrinsic parameters of the real-view camera are obtained by adjusting the preset intrinsic and extrinsic parameters of the real-view camera through the PNP algorithm.
7. The video virtual-real fusion method according to claim 1, characterized in that, Based on the intrinsic and extrinsic parameters of the real-scene camera, the video image and the rendered image are fused in the GPU rendering pipeline to obtain a virtual-real fused image, including the following steps: Based on the intrinsic and extrinsic parameters of the real-scene camera, the coordinates of the 3D point are projected onto the video image to obtain the projection position information; Obtain the depth map corresponding to the rendered image, and filter out projection points that are not within the field of view or have negative depth values in the projection position information based on the depth map to obtain the target projection points for video fusion. Based on the target projection point, obtain the RGB color information of the 3D point coordinates in the video image, obtain the pixel position of the 3D point coordinates in the rendered image, and replace the color information of the coordinate pixel position of the 3D point in the rendered image with the RGB color information of the 3D point coordinates. After processing all 3D points, the video image and the rendered image are fused together to obtain a virtual-real fused video.
8. A video virtual-real fusion system, characterized in that, include: The rendering image acquisition module is used to acquire video images captured by a real-world camera and to acquire a rendering image in a preset 3D map whose difference from the viewpoint parameters of the video images is within a preset range. The feature matching module is used to obtain the feature matching relationship between the video image and the rendered image based on a preset neural network matching method. The rendering camera intrinsic and extrinsic parameter acquisition module is used to acquire the field of view, width information, and height information of the rendered image, and calculate the rendering camera intrinsic and extrinsic parameters. The back projection module is used to back project the rendered image based on the intrinsic and extrinsic parameters of the rendering camera to obtain the 3D point coordinates corresponding to the pixels of the rendered image. An initial intrinsic and extrinsic parameter acquisition module is used to acquire the initial intrinsic and extrinsic parameters of the real-view camera; The real-scene camera intrinsic and extrinsic parameter acquisition module is used to optimize the initial intrinsic and extrinsic parameters based on the 3D point coordinates and the feature matching relationship using a nonlinear optimization method to obtain the real-scene camera intrinsic and extrinsic parameters; The video fusion module is used to fuse the video image and the rendered image in the GPU rendering pipeline based on the intrinsic and extrinsic parameters of the real-scene camera to obtain a virtual-real fused image.
9. A computer device, comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the video virtual-real fusion method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the video virtual-real fusion method as described in any one of claims 1 to 7.