A 4D video generation method and system fusing monocular vision and terrain elevation data

Abstract

The application discloses a 4D video generation method fusing monocular vision and terrain elevation data, comprising the following steps: acquiring video frame images and corresponding camera pose data collected by a monocular camera; acquiring terrain elevation data of a target area, and constructing a three-dimensional elevation network model; based on the camera pose data, establishing a ray projection model for pixels in the video frame images, calculating the intersection of the rays and the three-dimensional grid model, and obtaining the reference distance from the monocular camera to the ground surface corresponding to the pixels; generating an initial relative depth map of the video frame images by using a depth estimation model; selecting high-confidence ground pixels from the video frame images as reference points, taking the reference distance corresponding to the reference points as the true value, and performing absolute scale correction on the initial relative depth map to obtain an absolute depth map; performing back projection calculation on the absolute depth map, the camera pose data and pixel color information to generate a three-dimensional point cloud, mapping the three-dimensional point cloud to a real geographic coordinate system, and generating 4D video data in time sequence.

Description

Technical Field

[0001] This invention relates to the fields of monocular vision and geographic information system technology, specifically to a 4D video generation method and system that integrates monocular vision and terrain elevation data. Background Technology

[0002] 4D video generation technology aims to construct dynamic scene representations that include three-dimensional spatial geometry and temporal dimensions, and has broad application prospects in fields such as autonomous driving simulation, robot navigation, virtual reality, and digital twins. Currently, monocular vision-based 4D video generation methods are receiving significant attention due to their advantages such as low sensor cost and flexible deployment.

[0003] In the prior art, for example, Chinese invention patent application CN121567935A discloses a method, apparatus, and device for generating 4D video with joint controllable camera and illumination. This method generates dynamic point clouds and sparse, heavily illuminated point clouds based on input video, processes the point clouds through the target camera trajectory to generate geometric and illumination prior information, and finally outputs the target video using a diffusion model. This method has made progress in the joint control of camera trajectory and illumination conditions, but its depth information acquisition mainly relies on a monocular depth estimation model to generate a relative depth map, lacking absolute scale information, resulting in a deviation between the generated 3D point cloud and the real geographic scale. At the same time, this method does not involve the fusion of terrain elevation data, and cannot guarantee that the generated point cloud is accurately aligned with the real surface geometry, limiting its applicability in application scenarios that require high-precision geographic coordinate information.

[0004] Furthermore, existing monocular vision-based depth estimation methods generally suffer from scale ambiguity, meaning that the absolute depth scale of a scene cannot be recovered from a single frame image; only relative depth relationships can be obtained. Although multi-view geometry or inertial measurement units can be used to assist in solving the scale, absolute scale correction of depth maps remains a technical challenge in scenarios lacking multi-view constraints or sensors. On the other hand, terrain elevation data can provide absolute surface depth information, but existing technologies lack solutions for effectively fusing monocular vision depth estimation with terrain elevation data to generate 4D videos with realistic geographic scales. Summary of the Invention

[0005] In order to overcome the technical defects of existing monocular vision-based 4D video generation methods, such as the inability to obtain absolute depth scale and the difficulty in aligning with real terrain elevation data, this invention provides a 4D video generation method and system that integrates monocular vision and terrain elevation data.

[0006] To solve the above problems, the present invention is implemented according to the following technical solution:

[0007] In a first aspect, the present invention provides a 4D video generation method that integrates monocular vision and terrain elevation data, comprising the following steps:

[0008] Acquire video frame images captured by a monocular camera and the camera pose data corresponding to each video frame image;

[0009] Acquire topographic and elevation data of the target area and construct a three-dimensional elevation network model;

[0010] For each video frame, based on the camera pose data, a ray projection model is established for the pixels in the video frame, and the intersection of the ray and the three-dimensional mesh model is calculated to obtain the reference distance from the monocular camera to the ground surface corresponding to the pixel.

[0011] An initial relative depth map of the video frame image is generated using a depth estimation model;

[0012] High-confidence ground pixels are selected from the video frame images as reference points, and the reference distance corresponding to the reference points is used as the ground value to perform absolute scale correction on the initial relative depth map to obtain an absolute depth map.

[0013] Based on the absolute depth map, camera pose data, and pixel color information, a three-dimensional point cloud is generated through back projection calculation. The three-dimensional point cloud is then mapped to the real geographic coordinate system, and 4D video data is generated by organizing it in time sequence.

[0014] In conjunction with the first aspect, the present invention provides a first specific implementation of the first aspect, specifically, the step of acquiring topographic elevation data of the target area and constructing a three-dimensional elevation network model includes:

[0015] The terrain elevation data is preprocessed to generate a continuous digital elevation surface;

[0016] The digital elevation surface is processed using a triangulation algorithm to generate a three-dimensional mesh model composed of multiple triangular facets.

[0017] A KD-Tree index structure is constructed for the three-dimensional mesh model to support fast intersection calculations between rays and triangular patches.

[0018] In conjunction with the first aspect, the present invention provides a second specific implementation of the first aspect. Specifically, the step of establishing a ray projection model for pixels in the video frame image based on the camera pose data, calculating the intersection points of the rays and the three-dimensional mesh model, and obtaining the reference distance from the monocular camera to the ground surface corresponding to the pixel specifically includes:

[0019] Obtain the pre-calibrated camera intrinsic parameter matrix and the extrinsic parameter matrix corresponding to the current frame. Based on the intrinsic parameter matrix, convert the image pixel coordinates into a normalized direction vector in the camera coordinate system. Then, use the rotation matrix in the extrinsic parameter matrix to convert the normalized direction vector to the world coordinate system to obtain the ray direction.

[0020] Using the translation vector in the extrinsic parameter matrix as the origin of the camera optical center in the world coordinate system, construct the ray parameter equation along the ray direction;

[0021] Candidate triangular facets are retrieved using the KD-Tree index structure. By solving the intersection equations of rays and triangles, the intersection points that satisfy the centroid coordinate constraints and have the minimum ray parameters are selected as the surface intersection points.

[0022] Calculate the Euclidean distance between the intersection point on the ground and the optical center of the camera to obtain the reference distance from the monocular camera to the ground corresponding to the image pixel.

[0023] In conjunction with the first aspect, the present invention provides a third specific implementation of the first aspect, specifically, the step of selecting high-confidence ground pixels as reference points from the video frame image specifically includes:

[0024] The video frame images are processed using a semantic segmentation model to identify and extract ground category regions, and an initial ground mask is generated.

[0025] A geometric consistency constraint is applied to the pixels within the initial ground mask, and pixels that simultaneously satisfy the initial ground mask and the geometric consistency constraint are retained to form a high-confidence ground reference pixel set.

[0026] In conjunction with the first aspect, the present invention provides a fourth specific implementation of the first aspect. Specifically, the step of using the reference distance corresponding to the reference point as the true value to perform absolute scale correction on the initial relative depth map to obtain an absolute depth map specifically includes:

[0027] Extract the relative depth value corresponding to the high-confidence ground reference cell in the initial relative depth map, and obtain the reference distance corresponding to the cell as the absolute depth true value;

[0028] A linear correction model is established from relative depth to absolute depth, and the linear correction model includes two parameters to be solved: global scale factor and offset.

[0029] Using the relative depth value as input and the absolute depth true value as constraint, the global scale factor and the offset are solved by least squares fitting method by utilizing the correspondence between the relative depth values ​​and reference distances of multiple sets of high-confidence ground reference pixels.

[0030] The global scale factor and offset obtained by solving are applied to all pixels in the initial relative depth map, and the absolute depth value is calculated pixel by pixel to obtain the absolute depth map.

[0031] In conjunction with the first aspect, the present invention provides a fifth specific implementation of the first aspect, specifically, the step of organizing and generating the 4D video data in a time sequence includes:

[0032] Based on the absolute depth map and the camera intrinsic parameter matrix, the pixel coordinates are back-projected onto the camera coordinate system to generate a 3D point cloud in the camera coordinate system.

[0033] Using the rotation matrix and translation vector in the camera pose data, the 3D point cloud in the camera coordinate system is transformed into a 3D point cloud in the world coordinate system;

[0034] Based on the preset projection transformation relationship, the three-dimensional point cloud in the world coordinate system is converted into a three-dimensional point cloud containing real geographic coordinates including latitude, longitude and elevation.

[0035] The 3D point cloud generated from each frame, with geographic coordinates, is organized into a point cloud sequence according to the timestamp order to form the 4D video data.

[0036] In conjunction with the first aspect, the present invention provides a sixth specific implementation of the first aspect, specifically, for non-ground pixels in the video frame image, they are not used as reference points in the absolute scale correction process;

[0037] The correction parameters obtained by performing absolute scale correction using high-confidence ground pixels are directly applied to the relative depth values ​​of the non-ground pixels to obtain the approximate absolute depth of the non-ground region.

[0038] Secondly, the present invention also provides a 4D video generation system that integrates monocular vision and terrain elevation data, comprising:

[0039] The data acquisition module is configured to acquire video frame images captured by a monocular camera and the camera pose data corresponding to each video frame image;

[0040] The terrain elevation modeling module is configured to acquire terrain elevation data of the target area and construct a three-dimensional elevation network model.

[0041] The reference distance calculation module is configured to, for each video frame image, establish a ray projection model for the pixels in the video frame image based on the camera pose data, calculate the intersection point of the ray and the three-dimensional mesh model, and obtain the reference distance from the monocular camera to the ground surface corresponding to the pixel.

[0042] The depth correction module is configured to generate an initial relative depth map of the video frame image using a depth estimation model; select high-confidence ground pixels from the video frame image as reference points, use the reference distance corresponding to the reference points as ground values, and perform absolute scale correction on the initial relative depth map to obtain an absolute depth map;

[0043] The 4D video generation module is configured to perform back projection calculations to generate a three-dimensional point cloud based on the absolute depth map, camera pose data, and pixel color information, and to map the three-dimensional point cloud to the real geographic coordinate system, and to organize and generate 4D video data in a time sequence.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] This invention utilizes only video frames and corresponding pose data acquired by a monocular camera, combined with pre-acquired publicly available terrain elevation data, to calculate the absolute distance from the camera to a ground reference point through ray projection, and uses high-confidence ground pixels as the ground truth to perform scale correction on the relative depth map output by deep learning. It can obtain an absolute depth map with real physical scale without additional hardware such as LiDAR, binocular cameras, or inertial measurement units.

[0046] This invention uses the absolute distance of elevation data to the ground as a global scale benchmark, and uses monocular depth estimation to obtain the relative structural details of each pixel. The generated absolute depth map after correction maintains the consistency of real geographic coordinates and preserves the geometric texture of non-ground objects.

[0047] This invention automatically selects high-confidence ground pixels from video frames as reference points and uses the distance between the ray-elevation grid intersections corresponding to these pixels as ground truth to perform global or local absolute scale correction on the initial relative depth map. This strategy avoids the errors caused by fitting all pixels of the entire image. 4D video data can be directly used for applications such as digital twins of dynamic scenes, temporal change detection, disaster monitoring, and autonomous driving simulation, solving the problems of missing time dimension or scale inconsistency in traditional 3D reconstruction. Attached Figure Description

[0048] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, wherein:

[0049] Figure 1 This is a flowchart of a 4D video generation method that integrates monocular vision and terrain elevation data according to the present invention.

[0050] Figure 2 This is a block diagram of an electronic device used to implement embodiments of the present invention;

[0051] In the picture:

[0052] 100 - Electronic device, 101 - Computing unit, 102 - ROM, 103 - RAM, 104 - Bus, 105 - I / O interface, 106 - Input unit, 107 - Output unit, 108 - Storage unit, 109 - Communication unit. Detailed Implementation

[0053] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0054] like Figures 1-2 As shown, this invention provides a 4D video generation method and system that integrates monocular vision and terrain elevation data.

[0055] In recent years, with the advancement of smart city, digital twin, and 3D reality construction in China, the country has increasingly higher requirements for the authenticity and timeliness of geospatial information acquisition. Traditional 3D reconstruction technologies often face the dilemma of "having shape but no scale" or "having scale but no geographic coordinates," especially in drone aerial videos. Although monocular vision can acquire rich texture information, it lacks true physical depth and geographic reference. At the same time, while existing digital elevation models (DEMs) can provide accurate surface elevation, they are difficult to represent protruding features such as buildings and vegetation, and cannot directly generate dynamic video stream data.

[0056] Currently, converting monocular video into 4D data with real geographic coordinates and physical scale mainly relies on expensive LiDAR or multi-view dense matching technology. The former has extremely high hardware costs, while the latter is computationally time-consuming and has stringent requirements for texture. For scenarios such as drone hovering monitoring and vehicle-mounted mobile mapping, how can we utilize low-cost monocular cameras combined with publicly available geographic elevation data to generate 4D video with geographic coordinates in real time?

[0057] To address this, this invention proposes a 4D video generation method and system based on the fusion of monocular vision and geographic elevation data, aiming to achieve the mapping of video pixels to real-world three-dimensional coordinates without the need for expensive hardware. This invention constructs a conversion process from two-dimensional pixels to four-dimensional spatiotemporal data by fusing camera pose, DEM data, and deep learning depth estimation.

[0058] Example 1

[0059] This embodiment provides a method for generating 4D video by fusing monocular vision and terrain elevation data. This method can be executed by a processing device with computing capabilities (such as a ground station server, edge computing unit, or UAV onboard computer). This method is particularly suitable for scenarios where the camera pose remains basically static or changes slowly, such as when a UAV is hovering in one place or a fixed monitoring camera. By fusing terrain elevation data, it reconstructs a time-series 3D point cloud with a true geographical scale from monocular video frames, thereby generating 4D video data.

[0060] like Figure 1 As shown, the 4D video generation method that integrates monocular vision and terrain elevation data includes the following steps:

[0061] S100: Acquire video frame images captured by a monocular camera and camera pose data corresponding to each video frame image;

[0062] S200: Acquire terrain and elevation data of the target area and construct a three-dimensional elevation network model;

[0063] S300: For each video frame image, based on the camera pose data, establish a ray projection model for the pixels in the video frame image, calculate the intersection point of the ray and the three-dimensional mesh model, and obtain the reference distance from the monocular camera to the ground surface corresponding to the pixel.

[0064] S400: Generate an initial relative depth map of the video frame image using a depth estimation model; select high-confidence ground pixels from the video frame image as reference points, use the reference distance corresponding to the reference points as ground truth, and perform absolute scale correction on the initial relative depth map to obtain an absolute depth map;

[0065] S500: Based on the absolute depth map, camera pose data and pixel color information, perform back projection calculations to generate a 3D point cloud, map the 3D point cloud to the real geographic coordinate system, and organize it in time sequence to generate 4D video data.

[0066] Specifically, the present invention provides a detailed description of each step of the 4D video generation method as follows.

[0067] S100: Acquire video frame images captured by a monocular camera and the camera pose data corresponding to each video frame image.

[0068] In this invention, video frame images are continuously acquired in a time sequence using a monocular camera. Camera pose data, including the camera intrinsic parameter matrix and the camera extrinsic parameter matrix, serves as a bridge connecting the pixels of the video frame images with real-world coordinates.

[0069] Specifically, the intrinsic parameter matrix of a monocular camera This describes the camera's own projection properties, determined by the camera's focal length. and principal point coordinates The composition was obtained through pre-calibration offline using the Zhang Zhengyou calibration method: ;in: Focal length (in pixels) of a monocular camera in the x and y directions, respectively. Principal point coordinates (unit: pixels).

[0070] The camera extrinsic parameter matrix describes the rigid body transformation from the camera coordinate system to the world coordinate system, including the rotation matrix R and the translation vector t. The translation vector t (i.e., the camera optical center world coordinate system) The coordinates are acquired in real time by the RTK-GPS module carried by the UAV, and the latitude, longitude, and altitude coordinates are transformed to the UTM projected coordinate system to obtain... The rotation matrix R is synthesized from the pitch, roll, and yaw angles acquired in real time by the IMU (Inertial Measurement Unit). To suppress jitter noise during UAV hovering, GPS and IMU data are fused using an extended Kalman filter to ensure the smoothness and real-time performance of the pose data.

[0071] S200: Acquire terrain elevation data of the target area and construct a three-dimensional mesh model.

[0072] This invention acquires digital elevation model (DEM) data of a target area in raster format, with each raster cell recording the surface elevation at that location. To facilitate ray intersection calculations, the raster DEM needs to be constructed as a continuous three-dimensional mesh model, i.e., a surface geometry representation composed of a series of interconnected triangular facets. This DEM model represents the basic elevation information of bare land or the surface, excluding the height of objects above buildings, vegetation, vehicles, etc.

[0073] In this embodiment of the invention, the specific steps for acquiring terrain elevation data of the target area and constructing a three-dimensional elevation network model are as follows:

[0074] S201: Preprocess the terrain elevation data to generate a continuous digital elevation surface;

[0075] The raw DEM data obtained in this invention is typically in a regular raster format, with each raster cell recording the surface elevation at that location. However, raw DEMs may suffer from missing data, noise, or uneven distribution of discrete sampling points. Therefore, preprocessing is required:

[0076] Invalid values ​​in the DEM data are filled using neighborhood interpolation, with options including inverse distance weighted interpolation, kriging interpolation, or bilinear interpolation. Secondly, to eliminate acquisition noise in the elevation data, median filtering or Gaussian filtering is used to smooth the DEM raster. After these processes, a continuous and smooth digital elevation surface is generated. ,in Z represents the plane coordinates in the UTM projection coordinate system, and Z represents the ground elevation at these coordinates.

[0077] S202: The digital elevation surface is processed using a triangulation algorithm to generate a three-dimensional mesh model composed of multiple triangular facets.

[0078] Specifically, while regular raster DEMs are easy to store, each of their raster cells is rectangular, which can lead to ambiguity and low computational efficiency when directly used for ray intersection calculations. This invention employs the Delaunay triangulation algorithm to convert the digital elevation surface into a triangular mesh model, specifically:

[0079] All grid points of the DEM raster are treated as a two-dimensional plane point set. A Delaunay triangulation is constructed to ensure that the circumcircle of any triangle does not contain any other points, thereby maximizing the minimum angle and avoiding the generation of overly elongated triangles. Each triangle vertex is defined by its planar coordinates. The triangular patch formed by the corresponding elevation value Z in three-dimensional space All triangular facets are pieced together without overlap or gaps to form a complete, continuous three-dimensional mesh model of the Earth's surface. .

[0080] A KD-Tree index structure is constructed for the three-dimensional mesh model to support fast intersection calculations between rays and triangular patches.

[0081] Specifically, rays are emitted from a large number of pixels in each frame of the image and their intersections with the triangular mesh are calculated. If all triangular faces in M ​​are traversed and tested one by one, the time complexity is O(n log n). The computational overhead is enormous and cannot meet the real-time requirements of video.

[0082] Therefore, this invention provides a triangular mesh model. Construct the KD-Tree index structure. The construction process is as follows:

[0083] Using the bounding boxes of triangular faces as basic units, the space is recursively divided. Each time, the longest axis in the current space is selected as the dividing direction, and the median of the center coordinates of all triangular bounding boxes along that direction is taken as the dividing point, dividing the space into left and right subspaces. This division is performed recursively until the number of triangular faces contained in each leaf node is less than a preset threshold (e.g., 5). Finally, all triangular faces are organized in a balanced KD-Tree.

[0084] In actual ray intersection calculations, the ray first intersects with the bounding box of the root node of the KD-Tree, quickly and recursively skipping subtrees with no intersection, and only performing precise ray-triangle intersection checks on a small number of candidate triangles in the leaf nodes. With KD-Tree acceleration, the intersection complexity of a single ray can be reduced to... This significantly improves computational efficiency.

[0085] S300: For each video frame image, based on the camera pose data, establish a ray projection model for the pixels in the video frame image, calculate the intersection point of the ray and the three-dimensional mesh model, and obtain the reference distance from the monocular camera to the ground surface corresponding to the pixel.

[0086] Specifically, in order to calculate the real physical distance corresponding to each pixel in an image, it is necessary to establish a mapping relationship from two-dimensional pixels to the three-dimensional physical world.

[0087] In a specific implementation of the present invention, the step of establishing a ray projection model for pixels in the video frame image based on the camera pose data, calculating the intersection points of the rays and the three-dimensional mesh model, and obtaining the reference distance from the monocular camera to the ground surface corresponding to the pixel specifically includes:

[0088] Obtain the pre-calibrated camera intrinsic parameter matrix and the extrinsic parameter matrix corresponding to the current frame. Based on the intrinsic parameter matrix, convert the image pixel coordinates into a normalized direction vector in the camera coordinate system. Then, use the rotation matrix in the extrinsic parameter matrix to convert the normalized direction vector to the world coordinate system to obtain the ray direction.

[0089] Specifically, obtain the pre-calibrated camera intrinsic parameter matrix K and the extrinsic parameter matrix corresponding to the current frame (including the rotation matrix R and the translation vector t).

[0090] For any pixel coordinate to be projected in the image Based on the intrinsic parameter matrix K, the image pixel coordinates are converted into normalized direction vectors in the camera coordinate system. The calculation is as follows: This vector represents the three-dimensional direction from the camera optical center to the pixel on the normalized image plane, and its magnitude is... The direction is in front of the camera (positive Z-axis direction).

[0091] The normalized direction vector is obtained by rotating the rotation matrix R in the extrinsic parameter matrix. Transform to the world coordinate system to obtain the final direction of the ray. .

[0092] At the same time, the translation vector t in the extrinsic parameter matrix directly determines the coordinates of the camera optical center in the world coordinate system. : ;in, The latitude, longitude, and altitude data collected in real time by RTK-GPS are converted to the UTM coordinate system.

[0093] Using the translation vector in the extrinsic parameter matrix as the origin of the camera optical center in the world coordinate system, construct the ray parameter equation along the ray direction;

[0094] Specifically, based on the aforementioned ray origin and ray direction Constructing ray parameter equations ( (For depth parameters): .in, The distance traveled from the optical center along the ray direction (unit: meters). Let be the world coordinates of the corresponding point on the ray.

[0095] Candidate triangular facets are retrieved using the KD-Tree index structure. By solving the intersection equations of rays and triangles, the intersection points that satisfy the centroid coordinate constraints and have the minimum ray parameters are selected as the surface intersection points.

[0096] Specifically, the KD-Tree index structure is used to perform accurate ray-triangle intersection detection on candidate triangular faces.

[0097] Starting from the root node of the KD-Tree, recursively check if the ray intersects with the AABB bounding box of the node. If they do not intersect, prune and skip the ray; if they intersect, continue to the child nodes until all leaf nodes are reached. All triangular faces contained in the leaf nodes are used as a candidate set.

[0098] For each triangle facet in the candidate set By solving the equations of the intersection of the ray and the triangle: ;in The coordinates are the centroid coordinates. The equation can be rewritten in matrix form: The solution can be obtained quickly using Cramer's rule or the Möller-Trumbore algorithm. .

[0099] Select the intersection points from all valid solutions that satisfy the following conditions:

[0100] (The intersection point is in front of the ray);

[0101] (The intersection point is inside the triangle or on its side).

[0102] Among all intersections that satisfy the conditions, take The smallest value is denoted as Its corresponding world coordinates This is the physical point where the ray first intersects the Earth's surface.

[0103] Calculate the Euclidean distance between the intersection point on the ground and the optical center of the camera to obtain the reference distance from the monocular camera to the ground corresponding to the image pixel.

[0104] Specifically, obtain the intersection point on the ground. Calculate the intersection of the ground surface and the camera optical center. The Euclidean distance between them is the actual physical reference distance from the monocular camera to the Earth's surface corresponding to that image pixel. : .because Since it is a unit direction vector, the intersection parameters are... Numerically, it is equal to the reference distance (unit: meters).

[0105] S400: Generate an initial relative depth map of the video frame image using a depth estimation model; select high-confidence ground pixels from the video frame image as reference points, use the reference distance corresponding to the reference points as ground truth, and perform absolute scale correction on the initial relative depth map to obtain an absolute depth map;

[0106] In this invention, a pre-trained depth estimation network (such as DPT, MiDaS, etc.) is used to process a single frame image. This network outputs a grayscale image with the same resolution as the input image. Its numerical range is The relative depth value represents the relative distance between pixels in a scene. Typically, this relative depth value is inversely proportional to the actual physical depth; that is, the larger the relative depth value (the whiter it is), the closer the physical distance to that point. Since a DEM can only represent terrain elevation and cannot accurately describe objects such as buildings, vehicles, and trees that are above or below the ground surface, pixels belonging only to the ground are selected as the correction benchmark.

[0107] In a specific implementation of the present invention, selecting high-confidence ground pixels as reference points from the video frame image specifically includes:

[0108] The video frame images are processed using a semantic segmentation model to identify and extract ground category regions, and an initial ground mask is generated.

[0109] Specifically, models such as DeepLabV3+, SegFormer, or OCRNet perform pixel-by-pixel classification on the input single-frame RGB image. This model, trained on datasets containing diverse scenes (such as Cityscapes and ADE20K), can identify typical ground categories such as "roads," "sidewalks," "bare ground," "grass," "squares," and "farmland." For each pixel... The model outputs its category label. Generate an initial ground mask based on the label category. : Pixels with a value of 1 in the mask are initially considered to be ground pixels.

[0110] A geometric consistency constraint is applied to the pixels within the initial ground mask, and pixels that simultaneously satisfy the initial ground mask and the geometric consistency constraint are retained to form a high-confidence ground reference pixel set.

[0111] Specifically, the ground regions obtained from semantic segmentation may contain two types of errors: non-ground objects are incorrectly labeled as ground, or the ground regions contain outliers located above (e.g., a pedestrian's feet) or below (e.g., the edge of a pothole). To address this, this invention introduces a geometric consistency constraint for candidate pixel selection. This constraint includes the following criteria:

[0112] 1. Smoothness of normal vector change: Calculate the local normal vectors of the intersection points of several pixel rays in the neighborhood of the candidate pixel with the DEM. If the change amplitude of the normal vector exceeds the preset threshold (e.g., 15∘), the pixel is considered to be located in the edge of a non-ground object with drastic terrain undulations and is removed.

[0113] 2. Neighborhood Depth Continuity: Examine the reference distance between the candidate cell and its spatial neighboring cells. The absolute value of the difference, if a certain pixel's If the difference between the pixel and the median of the surrounding pixels is greater than a threshold (e.g., 0.5 meters), it is considered an outlier and is removed.

[0114] 3. High consistency: The height of the intersection point between the pixel ray and the DEM. The height of the pixel is compared with the height of the intersection point in the surrounding 3×3 area. If the difference exceeds the threshold (e.g., 0.3 meters), it indicates that the pixel may correspond to a tree, building, or moving object and is therefore removed.

[0115] 4. Non-dynamic target constraint: Pixels identified as dynamic targets are excluded by using optical flow information of adjacent frames in the video stream or by directly using dynamic object categories (such as vehicles and pedestrians) in semantic segmentation.

[0116] Pixels that simultaneously satisfy all the above geometric consistency constraints and the initial ground mask conditions are ultimately identified as high-confidence ground reference pixels, forming a set of reference points. Each cell in this set simultaneously possesses:

[0117] Image coordinates ;

[0118] Ground attributes guaranteed by semantic segmentation;

[0119] The precise reference distance obtained from calculation (That is, the actual distance from the camera optical center to the intersection of the DEM and the ground surface).

[0120] In a specific implementation of the present invention, the step of using the reference distance corresponding to the reference point as the true value to perform absolute scale correction on the initial relative depth map to obtain an absolute depth map specifically includes:

[0121] Extract the relative depth value corresponding to the high-confidence ground reference cell in the initial relative depth map, and obtain the reference distance corresponding to that cell as the absolute depth ground truth; specifically, for each high-confidence ground reference cell... Its image coordinates are The initial relative depth map output from the monocular depth estimation network. The relative depth value of the pixel is read from the image. Simultaneously, obtain the reference distance corresponding to that pixel. This distance is the true physical distance (in meters) obtained by intersecting the ray-DEM, and is used as the true value of absolute depth.

[0122] A linear correction model is established from relative depth to absolute depth, and the linear correction model includes two parameters to be solved: global scale factor and offset.

[0123] Because the monocular depth estimation network outputs the relative depth value Typically within the range of (0,1], and roughly inversely proportional to the true depth (near objects have larger relative depth values, and distant objects have smaller relative depth values), this invention uses the following affine transformation model to map relative depth to absolute depth:

[0124] Specifically, absolute depth From relative depth value After linear transformation, we obtain: ; As a global scale factor, This is the offset. To prevent the use of tiny constants for division by zero, the model can be interpreted as linearly scaling and translating the reciprocal of the relative depth (which approximates the unscaled absolute depth) to fit the true absolute depth. Two parameters... and The entire frame of the image is shared globally.

[0125] Using the relative depth value as input and the absolute depth true value as constraint, the global scale factor and the offset are solved by least squares fitting method by utilizing the correspondence between the relative depth values ​​and reference distances of multiple sets of high-confidence ground reference pixels.

[0126] Specifically, suppose N high-confidence ground reference pixels have been selected, where N is an integer greater than or equal to 3. For the i-th reference point, we have According to the correction model, theoretically, the following should be satisfied: Since the actual data contains noise and residuals, the optimal values ​​of s and b are solved by minimizing the sum of squared residuals. This problem has a closed-form solution. Let: ;but: To ensure that the solution results are not affected by abnormal reference points, this invention optionally employs a random sampling consensus algorithm for robust fitting: randomly selecting a minimum subset (preferably 3) for multiple iterations, and selecting the model parameter with the most interior points as the final result. and .

[0127] The global scale factor and offset obtained by solving are applied to all pixels in the initial relative depth map, and the absolute depth value is calculated pixel by pixel to obtain the absolute depth map.

[0128] Specifically, the solved values ​​s and b are applied to all pixels (including non-ground pixels) in the initial relative depth map, and the absolute depth value is calculated pixel by pixel: .

[0129] in, That is, pixels The absolute depth value (in meters). Performing this calculation on each pixel of the image yields an absolute depth map with the same resolution as the original image. This depth map has the following characteristics:

[0130] For the ground area, the absolute depth value achieves high-precision alignment with the DEM reference distance;

[0131] For non-ground objects such as buildings, vehicles, and trees, the absolute depth value is obtained by propagating from the relative depth through a global scale factor. Although it is not as accurate as ground points, it maintains the consistency of relative spatial relationships and physical scale.

[0132] The output depth values ​​are continuous and smooth, and consistent with the physical distance dimensions of the real world, making them directly usable for 3D point cloud generation. After completing the absolute depth correction process, a usable absolute depth map is obtained. .

[0133] S500: Based on the absolute depth map, camera pose data and pixel color information, perform back projection calculations to generate a 3D point cloud, map the 3D point cloud to the real geographic coordinate system, and organize it in time sequence to generate 4D video data.

[0134] In this invention, the corrected absolute depth map contains only depth information from a two-dimensional image plane. To generate a 4D video with a true geographic reference, each pixel needs to be back-projected into a three-dimensional physical space and assigned realistic world coordinates.

[0135] In a specific implementation of the present invention, the step of organizing and generating the 4D video data in a time sequence specifically includes:

[0136] S501: Based on the absolute depth map and the camera intrinsic parameter matrix, back-project the pixel coordinates to the camera coordinate system to generate a 3D point cloud in the camera coordinate system.

[0137] Specifically, for any pixel coordinate in a video frame image Absolute depth value has been passed Using the pre-calibrated camera intrinsic parameter matrix K, the pixel can be back-projected to a 3D point in the camera coordinate system. The depth value is directly used as the Z-coordinate in the camera coordinate system. Calculate the X and Y coordinates based on the inverse transformation of the pinhole camera projection model: ;in, , Camera focal length (unit: pixels). The coordinates of the principal point are in pixels. The above transformations can also be uniformly expressed in matrix form: Perform the above back-projection operation on each pixel in the video frame image to obtain the 3D point cloud of the frame image in camera coordinate system. ,in These represent the image's height and width (in pixels). In addition to their 3D coordinates, each point also retains the color information of its corresponding pixel. .

[0138] S502: Using the rotation matrix and translation vector in the camera pose data, transform the 3D point cloud in the camera coordinate system to the 3D point cloud in the world coordinate system.

[0139] Specifically, the camera extrinsic parameters of the current frame, namely the rotation matrix RR and the translation vector, are obtained in real time through the RTK-GPS / IMU fusion system. (Coordinates of the camera's optical center in the world coordinate system). Point cloud in the camera coordinate system. Point cloud in world coordinate system The transformation relationship is as follows: ,in, It is the transpose of the rotation matrix, which transforms the direction vector of a point in the camera coordinate system to the world coordinate system;

[0140] = The world coordinates of the camera's optical center (provided by RTK-GPS and converted to the UTM coordinate system).

[0141] The camera coordinate system point cloud generated in step one Applying the above transformation to each point in the image yields the corresponding 3D point cloud in the world coordinate system. At this point, the spatial scale of the point cloud is completely consistent with the real physical world, and the relative positional relationships between the points accurately reflect the geometric structure of the scene.

[0142] S503: Based on the preset projection transformation relationship, convert the 3D point cloud in the world coordinate system into a 3D point cloud containing real geographic coordinates including latitude, longitude and elevation.

[0143] Specifically, the coordinates located in the UTM projection coordinate system Its coordinates The unit is meters, and the origin is located at a certain reference latitude and longitude. To facilitate application and sharing in Geographic Information Systems (GIS), the UTM coordinates need to be converted back to latitude and longitude coordinates under the WGS84 ellipsoid model. The standard UTM back-projection algorithm is preferred for this conversion. The specific process is as follows:

[0144] Determine the UTM projection zone: The current UTM projection zone number is determined based on the longitude of the camera's optical center. The UTM projection divides the globe into 60 longitude zones, each with a bandwidth of 6 degrees.

[0145] For each point in the point cloud The geodetic latitude and longitude were calculated using the UTM back projection formula. The back projection process involves the following key steps:

[0146] Calculate the relevant parameters of the equatorial radius 'a' and the reciprocal of the flattening;

[0147] Calculate the longitude of the central meridian;

[0148] The normalized dimensions can be solved using an iterative method or by directly applying a closed-form transformation formula.

[0149] Calculate the longitude difference to obtain the final latitude and longitude.

[0150] This is the ellipsoidal height (or the orthographic height corrected using DEM). Since the absolute depth correction of this invention incorporates DEM information... It can be directly used as the altitude Alt (unit: meters) of that point.

[0151] Ultimately, each point Converted into a quadruple containing latitude, longitude, and elevation. This process preserves the original RGB color information, resulting in a 3D point cloud with accurate geographic coordinates. .

[0152] S504: Organize the 3D point cloud with geographic coordinates generated in each frame into a point cloud sequence according to the timestamp order to form the 4D video data.

[0153] Specifically, for each frame of the video stream Repeat steps S501 to S503 to generate a 3D point cloud with geographic coordinates corresponding to the frame. Each point cloud The following metadata is attached:

[0154] Timestamp (Accurate to the millisecond level, provided by drones or data collection equipment);

[0155] Camera pose information (used for trajectory recording);

[0156] Frame number .

[0157] Organize the point clouds of all frames into a point cloud sequence in ascending order of timestamps: This sequence is 4D video data (3D spatial coordinates + time dimension). This data has the following characteristics:

[0158] Each point has WGS84 latitude and longitude coordinates, which can be directly overlaid onto existing GIS platforms or 3D earth software (such as Google Earth, Cesium, ArcGIS) for visualization and analysis.

[0159] Depth correction uses the true distance of the DEM as the ground truth to ensure that the three-dimensional scale of the point cloud scene is consistent with the actual physical world.

[0160] Since all frames are generated based on the same DEM model and a unified pose estimation framework, the point clouds between adjacent frames are naturally aligned in space.

[0161] By displaying the point cloud sequence frame by frame in chronological order, a dynamic 3D scene playback effect can be created. Users can freely switch perspectives to observe changes in the scene over time, including changes in land cover, construction progress of buildings, and the movement trajectories of vehicles and pedestrians.

[0162] The output 4D video data can be stored and transmitted using standard point cloud sequence formats (such as LAS / LAZ file sequences, timestamped PCD files, or video-encoded compressed depth + color dual-stream formats).

[0163] In a preferred embodiment, non-ground pixels in the video frame image are not used as reference points in the absolute scale correction process;

[0164] The correction parameters obtained by performing absolute scale correction using high-confidence ground pixels are directly applied to the relative depth values ​​of the non-ground pixels to obtain the approximate absolute depth of the non-ground region.

[0165] Specifically, through semantic segmentation and geometric consistency constraints, the set of pixels belonging to the ground is accurately identified and extracted. Correspondingly, any pixel that does not meet the ground category conditions (i.e., the semantic label does not belong to categories such as "road", "bare ground", "grass", "square") or does not meet the geometric consistency constraints (such as drastic changes in normal, discontinuous depth, or dynamic characteristics) is classified as a non-ground pixel.

[0166] Non-terrestrial pixels mainly include, but are not limited to, the following categories:

[0167] Buildings: walls, roofs, windows, etc.;

[0168] Vegetation: Trees, shrubs, tall grasslands, etc.;

[0169] Man-made facilities: streetlights, traffic signs, railings, etc.;

[0170] Dynamic objects: vehicles, pedestrians, animals, etc.;

[0171] Natural protrusions: rocks, mounds of earth, etc.

[0172] In this preferred embodiment, the aforementioned non-terrestrial pixels are not included in the reference point set. This means it does not participate in the solution process for the absolute scale correction parameters (global scale factor s and offset b). The fundamental reason for this design is:

[0173] A DEM (Digital Elevation Model) only describes the basic elevation information of bare ground or the ground surface and cannot accurately reflect the true spatial location of non-ground objects such as building height, tree canopy height, or vehicle rooftops. Substituting the distance from the intersection of a ray of a non-ground pixel and the DEM to the camera as the ground truth into the calibration model introduces systematic errors, leading to inaccurate scale factor estimation and consequently reducing the accuracy of full-map depth recovery. The constraint points used to solve for s and b are all derived from high-confidence ground reference pixels, ensuring that the parameters of the calibration model are entirely driven by stable and reliable ground geometry information.

[0174] Non-ground pixels do not participate in parameter solving, but after the global scale factor s and offset b are calculated, the same set of correction parameters are directly applied to the relative depth values ​​of non-ground pixels to obtain the approximate absolute depth of the non-ground region.

[0175] Specifically, for any non-ground pixel Its initial relative depth is The absolute depth of the pixel Calculated by the following formula: .

[0176] The relative depth relationships between ground and non-ground objects in the same scene (such as "vehicles are higher than the ground" and "buildings are higher than the ground") are implicitly learned by the network during training and usually have good relative measurement relationships.

[0177] By performing absolute scale correction on ground points, the relative depth map of the entire scene is effectively anchored to the true physical scale. Since ground pixels are distributed across multiple locations at different distances in the image, the estimated s and b reflect the systematic scale transformation of the entire scene, a transformation that also applies to non-ground objects.

[0178] Although there is a discrepancy between the true depth of non-ground objects (such as trees and vehicles) and the distance between their ray-DEM intersection points, the s and b estimated by ground points can still provide a scale-correct approximate absolute depth for non-ground objects. The error is usually within an acceptable range (relative error of about 5% to 15%), which is sufficient to meet the application requirements of 3D reconstruction, environmental perception and other applications.

[0179] Example 2

[0180] A 4D video generation system integrating monocular vision and terrain elevation data includes:

[0181] The data acquisition module is configured to acquire video frame images captured by a monocular camera and the camera pose data corresponding to each video frame image;

[0182] The terrain elevation modeling module is configured to acquire terrain elevation data of the target area and construct a three-dimensional elevation network model.

[0183] The reference distance calculation module is configured to, for each video frame image, establish a ray projection model for the pixels in the video frame image based on the camera pose data, calculate the intersection point of the ray and the three-dimensional mesh model, and obtain the reference distance from the monocular camera to the ground surface corresponding to the pixel.

[0184] The depth correction module is configured to generate an initial relative depth map of the video frame image using a depth estimation model; select high-confidence ground pixels from the video frame image as reference points, use the reference distance corresponding to the reference points as ground values, and perform absolute scale correction on the initial relative depth map to obtain an absolute depth map;

[0185] The 4D video generation module is configured to perform back projection calculations to generate a three-dimensional point cloud based on the absolute depth map, camera pose data, and pixel color information, and to map the three-dimensional point cloud to the real geographic coordinate system, and to organize and generate 4D video data in a time sequence.

[0186] Specifically, by fusing monocular visual and geographic elevation data, and using ray-DEM intersection to provide a true scale constraint for monocular depth estimation, this method effectively solves the problem of the lack of absolute scale in monocular visual depth estimation. It can recover 3D point clouds with true physical scale without relying on LiDAR or binocular cameras. Simultaneously, semantic segmentation and geometric consistency constraints are used to screen high-confidence ground reference points, avoiding interference from non-ground objects such as buildings and vehicles on scale correction, thus improving the robustness and accuracy of depth recovery. Furthermore, based on a unified geographic reference frame and DEM model, the 3D point clouds with geographic coordinates (WGS84) generated from adjacent video frames naturally possess spatiotemporal consistency, forming 4D video data without additional inter-frame registration or optical flow optimization, significantly reducing computational complexity. In addition, this method is applicable to monocular camera scenarios such as UAV hovering and fixed monitoring, is low-cost, flexible in deployment, and the generated point cloud sequences can be directly used in GIS applications such as environmental monitoring, digital twins, and urban management.

[0187] Example 3

[0188] According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0189] Figure 2 A schematic block diagram of an example electronic device 100 that can be used to implement embodiments of the present invention is shown. Electronic device 100 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 100 may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their links and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0190] like Figure 2 As shown, the electronic device 100 includes a computing unit 101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 102 or a computer program loaded from a storage unit 108 into a random access memory (RAM) 103. The RAM 103 may also store various programs and data required for the operation of the device 100. The computing unit 101, ROM 102, and RAM 103 are interconnected via a bus 104. An input / output (I / O) interface 105 is also linked to the bus 104.

[0191] Multiple components in electronic device 100 are linked to I / O interface 105, including: input unit 106, such as keyboard, mouse, etc.; output unit 107, such as various types of displays, speakers, etc.; storage unit 108, such as disk, optical disk, etc.; and communication unit 109, such as network card, modem, wireless transceiver, etc. Communication unit 109 allows electronic device 100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0192] The computing unit 101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 101 performs the various methods and processes described above, such as a 4D video generation method that fuses monocular vision and terrain elevation data. For example, in some embodiments, a 4D video generation method that fuses monocular vision and terrain elevation data can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 108. In some embodiments, part or all of the computer program can be loaded and / or installed on device 100 via ROM 102 and / or communication unit 109. When the computer program is loaded into RAM 103 and executed by the computing unit 101, one or more steps of the 4D video generation method that fuses monocular vision and terrain elevation data described above can be performed. Alternatively, in other embodiments, computing unit 101 may be configured by any other suitable means (e.g., by means of firmware) to perform a 4D video generation method that fuses monocular vision and terrain elevation data.

[0193] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0194] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0195] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical links based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0196] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0197] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0198] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0199] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0200] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Therefore, any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for generating 4D video by fusing monocular vision and terrain elevation data, characterized in that, Includes the following steps: Acquire video frame images captured by a monocular camera and the camera pose data corresponding to each video frame image; Acquire topographic and elevation data of the target area and construct a three-dimensional elevation network model; For each video frame, based on the camera pose data, a ray projection model is established for the pixels in the video frame, and the intersection of the ray and the three-dimensional mesh model is calculated to obtain the reference distance from the monocular camera to the ground surface corresponding to the pixel. An initial relative depth map of the video frame image is generated using a depth estimation model; High-confidence ground pixels are selected from the video frame images as reference points, and the reference distance corresponding to the reference points is used as the ground value to perform absolute scale correction on the initial relative depth map to obtain an absolute depth map. Based on the absolute depth map, camera pose data, and pixel color information, a three-dimensional point cloud is generated through back projection calculation. The three-dimensional point cloud is then mapped to the real geographic coordinate system, and 4D video data is generated by organizing it in time sequence.

2. The 4D video generation method fusing monocular vision and terrain elevation data according to claim 1, characterized in that, The acquisition of terrain elevation data of the target area and the construction of a three-dimensional elevation network model specifically include: The terrain elevation data is preprocessed to generate a continuous digital elevation surface; The digital elevation surface is processed using a triangulation algorithm to generate a three-dimensional mesh model composed of multiple triangular facets. A KD-Tree index structure is constructed for the three-dimensional mesh model to support fast intersection calculations between rays and triangular patches.

3. The 4D video generation method fusing monocular vision and terrain elevation data according to claim 2, characterized in that, The step of establishing a ray projection model for pixels in the video frame image based on the camera pose data, calculating the intersection points of the rays with the three-dimensional mesh model, and obtaining the reference distance from the monocular camera to the ground surface corresponding to the pixel specifically includes: Obtain the pre-calibrated camera intrinsic parameter matrix and the extrinsic parameter matrix corresponding to the current frame. Based on the intrinsic parameter matrix, convert the image pixel coordinates into a normalized direction vector in the camera coordinate system. Then, use the rotation matrix in the extrinsic parameter matrix to convert the normalized direction vector to the world coordinate system to obtain the ray direction. Using the translation vector in the extrinsic parameter matrix as the origin of the camera optical center in the world coordinate system, construct the ray parameter equation along the ray direction; Candidate triangular facets are retrieved using the KD-Tree index structure. By solving the intersection equations of rays and triangles, the intersection points that satisfy the centroid coordinate constraints and have the minimum ray parameters are selected as the surface intersection points. Calculate the Euclidean distance between the intersection point on the ground and the optical center of the camera to obtain the reference distance from the monocular camera to the ground corresponding to the image pixel.

4. The 4D video generation method fusing monocular vision and terrain elevation data according to claim 1, characterized in that, The step of selecting high-confidence ground pixels as reference points from the video frame image specifically includes: The video frame images are processed using a semantic segmentation model to identify and extract ground category regions, and an initial ground mask is generated. A geometric consistency constraint is applied to the pixels within the initial ground mask, and pixels that simultaneously satisfy the initial ground mask and the geometric consistency constraint are retained to form a high-confidence ground reference pixel set.

5. The 4D video generation method fusing monocular vision and terrain elevation data according to claim 1, characterized in that, The step of using the reference distance corresponding to the reference point as the true value to perform absolute scale correction on the initial relative depth map to obtain an absolute depth map specifically includes: Extract the relative depth value corresponding to the high-confidence ground reference cell in the initial relative depth map, and obtain the reference distance corresponding to the cell as the absolute depth true value; A linear correction model is established from relative depth to absolute depth, and the linear correction model includes two parameters to be solved: global scale factor and offset. Using the relative depth value as input and the absolute depth true value as constraint, the global scale factor and the offset are solved by least squares fitting method by utilizing the correspondence between the relative depth values ​​and reference distances of multiple sets of high-confidence ground reference pixels. The global scale factor and offset obtained by solving are applied to all pixels in the initial relative depth map, and the absolute depth value is calculated pixel by pixel to obtain the absolute depth map.

6. The 4D video generation method fusing monocular vision and terrain elevation data according to claim 1, characterized in that, The generation of the 4D video data in a time-series manner specifically includes: Based on the absolute depth map and the camera intrinsic parameter matrix, the pixel coordinates are back-projected onto the camera coordinate system to generate a 3D point cloud in the camera coordinate system. Using the rotation matrix and translation vector in the camera pose data, the 3D point cloud in the camera coordinate system is transformed into a 3D point cloud in the world coordinate system; Based on the preset projection transformation relationship, the three-dimensional point cloud in the world coordinate system is converted into a three-dimensional point cloud containing real geographic coordinates including latitude, longitude and elevation. The 3D point cloud generated from each frame, with geographic coordinates, is organized into a point cloud sequence according to the timestamp order to form the 4D video data.

7. The 4D video generation method fusing monocular vision and terrain elevation data according to claim 1, characterized in that, Also includes: For non-ground pixels in the video frame image, they are not used as reference points in the absolute scale correction process; The correction parameters obtained by performing absolute scale correction using high-confidence ground pixels are directly applied to the relative depth values ​​of the non-ground pixels to obtain the approximate absolute depth of the non-ground region.

8. A 4D video generation system integrating monocular vision and terrain elevation data, characterized in that, include: The data acquisition module is configured to acquire video frame images captured by a monocular camera and the camera pose data corresponding to each video frame image; The terrain elevation modeling module is configured to acquire terrain elevation data of the target area and construct a three-dimensional elevation network model. The reference distance calculation module is configured to, for each video frame image, establish a ray projection model for the pixels in the video frame image based on the camera pose data, calculate the intersection point of the ray and the three-dimensional mesh model, and obtain the reference distance from the monocular camera to the ground surface corresponding to the pixel. The depth correction module is configured to generate an initial relative depth map of the video frame image using a depth estimation model; select high-confidence ground pixels from the video frame image as reference points, use the reference distance corresponding to the reference points as ground values, and perform absolute scale correction on the initial relative depth map to obtain an absolute depth map; The 4D video generation module is configured to perform back projection calculations to generate a three-dimensional point cloud based on the absolute depth map, camera pose data, and pixel color information, and to map the three-dimensional point cloud to the real geographic coordinate system, and to organize and generate 4D video data in a time sequence.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the 4D video generation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the 4D video generation method according to any one of claims 1-7.

Images