Method for fusing monitoring video and 3D virtual scene

Abstract

The invention relates to a method for fusing a monitoring video and a 3D virtual scene, and belongs to the technical field of image processing and computer vision. The method comprises the following steps: firstly, constructing a three-dimensional digital scene and completing space calibration of a camera; then, camera offset is detected and corrected through real-time video analysis, and meanwhile, a target in a picture is recognized and tracked; secondly, the 2D pixel coordinates of the target are converted into space coordinates in a 3D scene through inverse projection calculation, and the positioning precision is improved through the multi-view intersection technology; and finally, based on the converted 3D coordinate data, performing fusion rendering on a video picture in a rendering engine, and outputting an interactive three-dimensional scene which can be used for situation awareness and behavior analysis. According to the method, the problems of inaccurate virtual and real space registration, dynamic error accumulation and unnatural rendering in the prior art are effectively solved, and high-precision and dynamic fusion of the monitoring video and the virtual scene is realized.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular to a method for fusing surveillance video and 3D virtual scenes. Background Technology

[0002] Video surveillance technology originated in the public safety field in the 1970s, initially centered on analog closed-circuit television (CCTV) systems, using cameras, video recorders (VCRs), and other equipment to achieve basic image acquisition and storage. At this stage, systems relied on coaxial cables to transmit analog signals, had limited functionality, were susceptible to interference, and could only meet basic security needs.

[0003] In the 1990s, digital technology propelled surveillance into its second generation—the era of digital video surveillance. The iconic device, the hard disk video recorder (DVR), integrated recording, storage, and remote control functions, replacing the cumbersome equipment of traditional analog systems and supporting long-term, high-quality recording and multi-channel management. The maturity of digital encoding and decoding technologies (such as MPEG) significantly improved image compression efficiency, while the application of network transmission protocols made remote access possible, initially breaking down geographical limitations.

[0004] In the 21st century, networking and intelligence have become core trends. The third-generation all-digital network video surveillance system, based on IP cameras, directly transmits digital signals via Ethernet or wireless networks, and combines embedded web server technology to achieve plug-and-play functionality and large-scale networking. Intelligent analysis technology incorporates computer vision algorithms, supporting target detection, behavior recognition (such as boundary crossing and trajectory tracking), and abnormal event early warning, driving surveillance from "passive recording" to "active defense."

[0005] Currently, video surveillance is further integrating AI, IoT, and big data technologies, possessing characteristics such as high definition (4K / 8K), low latency, and multi-system integration (e.g., access control, alarms). It is widely used in scenarios such as logistics loading and unloading, and industrial production, optimizing operational safety and efficiency through real-time monitoring and data analysis. In the future, with the popularization of 5G and edge computing, distributed intelligent monitoring will further enhance its real-time performance and collaborative capabilities.

[0006] Simultaneously, the technology of integrating video surveillance into virtual 3D scenes (commonly referred to as "video fusion" or "real-scene 3D fusion") is an important development direction in fields such as smart cities, security monitoring, and emergency management. This technology achieves a combined virtual and real visualization by spatially aligning and overlaying real-time video streams with 3D geographic information models (such as BIM, CIM, oblique photography, etc.). However, current technologies for integrating video surveillance into virtual 3D scenes suffer from spatial registration and alignment accuracy issues. I. Coordinate System - One Difficulty: 1. Video surveillance equipment (especially ordinary cameras) usually lacks high-precision spatial positioning information (such as accurate latitude and longitude, pitch angle, and yaw angle), making it difficult to accurately map their field of view into a three-dimensional scene.

[0007] 2. Different data sources (such as GPS, RTK, laser point cloud, oblique photogrammetry) have inconsistent coordinate systems, requiring complex transformations and calibrations during fusion.

[0008] II. Dynamic error accumulation: After installation, the camera may experience slight displacement due to environmental factors (such as wind, vibration, and temperature difference), causing the originally calibrated parameters to become invalid and affecting the stability of long-term use.

[0009] III. Distortion under non-ideal imaging conditions: Wide-angle and fisheye lenses suffer from severe image distortion, which, if not effectively corrected, can cause the video footage to fail to accurately match the 3D model. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the prior art and propose a method for fusing surveillance video and 3D virtual scenes. This method aims to streamline the entire process from camera calibration, dynamic target tracking, 2D / 3D coordinate transformation to final fusion rendering, effectively solving the problems of inaccurate spatial registration, dynamic error accumulation, and unnatural rendering effects.

[0011] The technical problem solved by this invention is achieved through the following technical solution: A method for fusing surveillance video and 3D virtual scenes includes the following steps: Step S1: Construct a 3D digital scene: Obtain or create a 3D scene file for the target area; Step S2, Camera Spatial Calibration: In the 3D scene constructed in step S1, determine the position and pose parameters of the camera; Step S3, Real-time Video Analysis and Target Tracking: Process the real-time video stream, detect and correct any camera shifts, identify and track dynamic targets in the scene, and output their 2D pixel coordinates and unique IDs. Step S4, 2D to 3D coordinate transformation: Based on the calibration parameters in step S2, the 2D target coordinates obtained in step S3 are transformed into spatial coordinates in the 3D scene through inverse projection mapping, and coordinate optimization can be performed by combining multi-view data; Step S5, Virtual-Real Fusion Rendering: Combine the 3D coordinate data obtained in step S4 with the 3D scene in step S1 in the rendering engine to render the video image or target and generate the fused 3D scene. Step S6, Interaction and Application Output: Output the fused scene generated in step S5 to support users in 3D situational awareness, behavior analysis and human-computer interaction.

[0012] Furthermore, in step S1, a 3D scene file containing geometric meshes and texture maps is constructed by means of LiDAR scanning, oblique photogrammetry, BIM model import or manual modeling, and the file is imported into the rendering engine.

[0013] Furthermore, step S2 includes either a manual calibration mode or an automatic calibration mode: The manual calibration mode is as follows: select at least 4 sets of corresponding feature points in the 3D scene and the real video image respectively, and solve the external and internal parameters of the camera through the PnP algorithm; The automatic calibration mode is as follows: using computer vision algorithms to automatically match feature points of 3D scenes and video images to calculate parameters, or using a positioning and orientation module to directly obtain the spatial pose of the camera during installation.

[0014] Furthermore, step S3 includes the following sub-steps: Step S3.1, Camera Offset Correction: By analyzing the static scene features in the real-time video and comparing them with the features during calibration, the camera offset is detected and corrected, and the calibration parameters are updated. Step S3.2, Object Detection: Use a deep learning model to identify targets in the video frame and output their 2D pixel coordinates and bounding boxes; Step S3.3, Target Tracking: Use a multi-target tracking algorithm to assign a unique ID to the detected target and form a motion trajectory across frames.

[0015] Furthermore, step S4 includes the following sub-steps: Step S4.1, Single-view inverse projection mapping: For a single camera, the 2D pixel coordinates of the target within its field of view are converted into a ray in the 3D scene through inverse projection calculation, and the initial coordinates of the target in 3D space are obtained by calculating the intersection of the ray with the geometric model of the 3D scene. Step S4.2, Multi-view intersection optimization: When the same target is captured by two or more calibrated cameras, the intersection calculation of rays from different viewpoints is performed by triangulation to obtain a more accurate 3D spatial coordinate of the target.

[0016] Furthermore, in step S4.1, the core calculation process of inverse projection mapping and ray-triangle intersection includes: Define a ray that originates from the camera's optical center and passes through a pixel; Calculate the intersection point of the ray with a specific triangular face in the 3D scene; By calculating the centroid coordinates and ray parameters, the precise position P_w of the intersection point in the 3D world coordinate system is determined.

[0017] Moreover, in step S5, the rendering engine receives the real-time 3D coordinate data stream output in step S4 through the application programming interface, accurately draws and merges the dynamic target (or its visual representation) in the video into the 3D scene of step S1 according to its 3D coordinates, and realizes the synchronous rendering of virtual and real images.

[0018] Moreover, in step S6, the output fused scene is an interactive 3D visualization interface in which users can perform operations such as viewpoint switching, target query, trajectory playback, and alarm management.

[0019] The advantages and positive effects of this invention are: 1. This invention has high fusion accuracy: Through a rigorous calibration process and inverse projection calculation, combined with multi-view intersection optimization, it achieves high-precision mapping from 2D pixels to 3D space.

[0020] 2. The system of this invention has strong robustness: the built-in camera offset correction mechanism can adaptively compensate for the error caused by physical displacement, ensuring the stability of the system in long-term operation.

[0021] 3. The invention produces natural visual effects: By performing blending rendering in an advanced rendering engine, it can better handle occlusion, lighting and perspective relationships, making the combination of video content and virtual scene more natural and harmonious.

[0022] 4. The invention has outstanding application value: the output interactive 3D fusion scene greatly enhances the intuitiveness and operability of monitoring data, and provides a powerful visualization tool for smart city management, security emergency response and other applications. Detailed Implementation

[0023] A method for fusing surveillance video and 3D virtual scenes includes the following steps: Step 1: Construct a 3D digital scene by obtaining or creating a 3D scene file for the target area.

[0024] The specific implementation method of step 1 is as follows: construct a 3D scene file containing geometric meshes and texture maps by means of LiDAR scanning, oblique photogrammetry, BIM model import or manual modeling, and import the 3D scene file into the rendering engine.

[0025] Step 2: In the 3D scene constructed in step S1, determine the position and orientation parameters of the camera.

[0026] Step 2 includes both manual and automatic tagging. Manual marking includes the following steps: Step 2.1.1: In the 3D scene file from Step 1, find at least 4 marker points that correspond to the real world. Step 2.1.2: In the camera video view, click on the pixel positions corresponding to the marked points in sequence; in the video view, mark the positions of the corners of the room and other vertices. This is an important data link between the real 2D pixel world and the virtual 3D geometric world.

[0027] Step 2.1.3: Solve the external and internal parameters of the camera using the PnP algorithm. The external parameters include position and rotation parameters, and the internal parameters include focal length and distortion parameters. The specific implementation method of automatic tagging is as follows: Computer vision algorithms are used to automatically identify feature points in a scene and match them with the positions of vertices such as the corners of the room calibrated by the 3D model, and parameters are automatically calculated; or RTK-GPS and inertial measurement units are installed in the camera to directly record its spatial pose.

[0028] The relationship between a camera and a 3D scene is that of an "observer" and a "world model." The core relationship is one of mapping and being mapped. Their interaction aims to bridge two different coordinate systems. Simply put, it involves mapping the camera view onto the 3D environment.

[0029] Step 3: Process the real-time video stream, detect and correct any camera offset, identify and track dynamic targets in the scene, and output their 2D pixel coordinates and unique IDs.

[0030] Step 3 includes the following steps: By combining the video footage from step 2 with information such as buildings and corners in the real-time video, and through depth analysis, when the camera shifts, the calibration coordinates and bounding box information are corrected and output.

[0031] Step 3.1: Use a deep learning model to detect targets in the image in real time and output their 2D pixel coordinates and bounding boxes; Step 3.2: Use the DeepSORT or ByteTrack algorithm to assign a unique ID to each detected target to achieve cross-frame tracking and form a motion trajectory.

[0032] Step 4: Based on the calibration parameters of step S2, convert the 2D target coordinates obtained in step S3 into spatial coordinates in the 3D scene through inverse projection mapping, and can combine multi-view data for coordinate optimization.

[0033] Step 4 includes the following steps: Step 4.1: Perform inverse projection mapping from 2D to 3D rays: Regarding the calibration of fbx spatial coordinate information, for the ray emitted from pixel p... Where O is the origin of the ray, which is [0,0,0] in the camera coordinate system, D is the direction, vector d, and a vector originating from the vertex. Find the intersection of the triangles formed. The local coordinate system of the constructed triangle (using...) With the origin as the point, Using the basis vectors (o, v), and combining the ray origin O and direction D, the centroid coordinates (u, v) and depth parameters are efficiently calculated using the cross product and dot product of the vectors. Finally, the world coordinates Pw of the intersection point are reconstructed.

[0034] Step 4.1.1: Calculate the edge vectors: Bind the coordinate edges of the 2D image and the 3D model. In the 3D model, with vertex V0 as the origin, two edge vectors e1 and e2 are constructed.

[0035] e1 is the vector of the first side of the triangle. It is defined as... , indicating from vertex point to The vector is used to span a triangular plane as a directional basis.

[0036] e2 is the vector of the second side of the triangle. It is defined as... , indicating from vertex point to The vector and e1 together span the plane containing the triangle.

[0037] It is the first vertex (reference vertex) of the triangle. It is one of the three vertices that make up the target triangle. (), which serves as the origin of the local coordinate system, has its coordinates already given in the world coordinate system.

[0038] Vector group and linear combination calculation: a=[e1,e2,e3] represents a basis consisting of three vectors. Any point or vector can be represented as a linear combination of these three basis vectors.

[0039] e3 is the vector of the third side of the triangle. It is defined as... , indicating from vertex point to The vectors e1 and e2 together span the plane containing the triangle.

[0040] 'a' is a three-dimensional vector (or point) representing a position or direction in space. Let x, y, z be the components of vector a in Cartesian coordinates.

[0041] Step 4.1.2: Calculate the determinant: This function calculates the cross product of two 3D vectors a and b, representing the cross product operation as a linear operation of multiplying a matrix by a vector. It serves as a bridge between 3D geometry and algebraic computation. It can efficiently determine whether a ray intersects a triangle and calculate the intersection parameters. P is used as an auxiliary vector for subsequent calculations, and the result is a vector. P is perpendicular to the plane spanned by the ray direction D and the triangle side e2.

[0042] D is the ray direction vector.

[0043] `det` is the key determinant value, which is [value] in the standard algorithm. Geometrically, it relates to the angle between the ray and the triangle's normal vector. It represents the scalar triple product formed by the ray direction D and the two basis vectors e1 and e2 of the triangle plane, the absolute value of which is equal to the volume of the parallelepiped spanned by these three vectors. If det≈0, it means the ray is parallel to the triangle plane (no intersection or coplanar).

[0044] Matrix calculation of the crossproduct of three-dimensional vectors.

[0045] b is another vector multiplied by a.

[0046] i, j, k are orthonormal basis vectors, corresponding to x, y, z respectively. The unit vectors along the axes are: (1, 0, 0), (0, 1, 0), (0, 0, 1).

[0047] Step 4.1.3: Calculate the center coordinates and distance λ. The center coordinates of the image correspond to the center coordinates of the 3D model camera. Matrix transformation calculation, e1, e2, e3: three basis vectors in the original coordinate system. The basis vectors in the new coordinate system are linearly transformed from the new coordinate system to the old coordinate system through a 3×3 transformation matrix.

[0048] T is the calculation point. The displacement vector to point O. O is the center coordinate of the image plane and a reference point in 3D space. It is the initial position or reference point of the camera in 3D space.

[0049] O-ray origin. The ray origin that is projected from the camera's optical center or image pixel into 3D space.

[0050] u is the first barycentric coordinate component. Calculated in step 4.1.3, it represents the intersection point along... Direction relative to The weights. When u∈[0,1] and satisfies the triangle's internal constraints, it indicates that the intersection point is within the valid region.

[0051] If u is not in the range [0,1], the intersection point is not inside the triangle.

[0052] Q is a temporary vector, calculated in the previous steps, and is usually defined as follows: (in , Q is T and The cross product of the ray and the triangle is perpendicular to the plane spanned by the ray and the first side of the triangle, and its magnitude reflects the area of ​​the parallelogram formed by the two sides. This vector is used to extract the projection information of the ray onto the normal direction of the triangle.

[0053] v represents the second centroid coordinate component. It is calculated from steps 4.1.3 or 4.1.4, and indicates the intersection point along... The weight of the direction. Usually, v≥0 and u+v≤1 must be satisfied to ensure that the intersection point is inside the triangle.

[0054] If v < 0 or u + v > 1, the intersection point is not inside the triangle.

[0055] Step 4.1.4: Calculate the ray parameter λ (depth): Calculate the depth parameter λ and perform ray triangle intersection calculations.

[0056] λ is the ray parameter, also known as the depth value. It represents the proportion of the distance from the camera center (or the ray origin) along the ray direction to the intersection of the triangles. If the ray is represented as R(t) = O + tD, then t = λ is the parameter corresponding to the intersection point.

[0057] Step 4.1.5: Calculate the world coordinates of the intersection point: Achieve image-model alignment, depth estimation, or interactive localization, and calculate the precise spatial position Pw of the intersection point of a ray and a 3D triangle surface in the world coordinate system.

[0058] Pw: World coordinates of the intersection point. Represents the three-dimensional coordinates of the actual intersection point of a ray originating from the camera's optical center, passing through the image center (or a specified pixel), and a triangular surface in the 3D model, in the global coordinate system.

[0059] Step 4.2, Multi-view intersection (from ray to 3D point); If a target is simultaneously observed by two or more calibrated cameras, triangulation, similar to binocular vision, can be used to calculate the target's precise 3D coordinates (including height Y). Inverse projection mapping and multi-view intersection are core technologies for mapping 2D images to 3D spatial coordinates. Their logical relationship is closely coupled and progressive.

[0060] Logical chain: inverse projection → ray generation → multi-view intersection → triangulation → 3D positioning.

[0061] The same physical target point Pw is observed simultaneously by ≥2 cameras, and the corresponding pixel points are (u1,v1) and (u2,v2) respectively, which is a more accurate positioning mapping relationship.

[0062] Step 5: Combine the 3D coordinate data obtained in step S4 with the 3D scene in step S1 in the rendering engine to render the video or target and generate the merged 3D scene.

[0063] Step 5.1: Import the constructed 3D scene into the engine; Step 5.2: The engine receives real-time data streams from the video analytics server via API, including target ID, 3D coordinates, and status.

[0064] Step 6: Output the fused scene generated in step S5 to support users in 3D situational awareness, behavior analysis and human-computer interaction.

[0065] It should be emphasized that the embodiments described in this invention are illustrative rather than limiting. Therefore, this invention includes, but is not limited to, the embodiments described in the specific implementation. Any other implementations derived by those skilled in the art based on the technical solutions of this invention are also within the scope of protection of this invention.

Claims

1. A method for fusing surveillance video and 3D virtual scenes, characterized in that, Includes the following steps: Step S1: Construct a 3D digital scene: Obtain or create a 3D scene file for the target area; Step S2, Camera Spatial Calibration: In the 3D scene constructed in step S1, determine the position and pose parameters of the camera; Step S3, Real-time Video Analysis and Target Tracking: Process the real-time video stream, detect and correct any camera shifts, identify and track dynamic targets in the scene, and output their 2D pixel coordinates and unique IDs. Step S4, 2D to 3D coordinate transformation: Based on the calibration parameters in step S2, the 2D target coordinates obtained in step S3 are transformed into spatial coordinates in the 3D scene through inverse projection mapping, and coordinate optimization can be performed by combining multi-view data; Step S5, Virtual-Real Fusion Rendering: Combine the 3D coordinate data obtained in step S4 with the 3D scene in step S1 in the rendering engine to render the video image or target and generate the fused 3D scene. Step S6, Interaction and Application Output: Output the fused scene generated in step S5 to support users in 3D situational awareness, behavior analysis and human-computer interaction.

2. The method according to claim 1, characterized in that, In step S1, a 3D scene file containing geometric meshes and texture maps is constructed by means of LiDAR scanning, oblique photogrammetry, BIM model import or manual modeling, and the file is imported into the rendering engine.

3. The method according to claim 1, characterized in that, Step S2 includes manual calibration mode or automatic calibration mode: The manual calibration mode is as follows: select at least 4 sets of corresponding feature points in the 3D scene and the real video image respectively, and solve the external and internal parameters of the camera through the PnP algorithm; The automatic calibration mode is as follows: using computer vision algorithms to automatically match feature points of 3D scenes and video images to calculate parameters, or using a positioning and orientation module to directly obtain the spatial pose of the camera during installation.

4. The method according to claim 1, characterized in that, Step S3 includes the following sub-steps: Step S3.1, Camera Offset Correction: By analyzing the static scene features in the real-time video and comparing them with the features during calibration, the camera offset is detected and corrected, and the calibration parameters are updated. Step S3.2, Object Detection: Use a deep learning model to identify targets in the video frame and output their 2D pixel coordinates and bounding boxes; Step S3.3, Target Tracking: Use a multi-target tracking algorithm to assign a unique ID to the detected target and form a motion trajectory across frames.

5. The method according to claim 1, characterized in that, Step S4 includes the following sub-steps: Step S4.1, Single-view inverse projection mapping: For a single camera, the 2D pixel coordinates of the target within its field of view are converted into a ray in the 3D scene through inverse projection calculation, and the initial coordinates of the target in 3D space are obtained by calculating the intersection of the ray with the geometric model of the 3D scene. Step S4.2, Multi-view intersection optimization: When the same target is captured by two or more calibrated cameras, the intersection calculation of rays from different viewpoints is performed by triangulation to obtain a more accurate 3D spatial coordinate of the target.

6. The method according to claim 5, characterized in that, In step S4.1, the core calculation process of inverse projection mapping and ray-triangle intersection includes: Define a ray that originates from the camera's optical center and passes through a pixel; Calculate the intersection point of the ray with a specific triangular face in the 3D scene; By calculating the centroid coordinates and ray parameters, the precise position P_w of the intersection point in the 3D world coordinate system is determined.

7. The method according to claim 1, characterized in that, In step S5, the rendering engine receives the real-time 3D coordinate data stream output in step S4 through the application programming interface, accurately draws the dynamic targets in the video according to their 3D coordinates and integrates them into the 3D scene in step S1, thereby achieving synchronous rendering of virtual and real images.

8. The method according to claim 1, characterized in that, In step S6, the output fused scene is an interactive 3D visualization interface in which users can perform operations such as viewpoint switching, target query, trajectory playback and alarm management.

Images