Real-time intelligent deduction method for three-dimensional geographic environment based on multi-modal large model

Abstract

The application discloses a three-dimensional geographic environment real-time intelligent deduction method based on a multi-modal large model, and relates to the technical field of computer vision. The method is used to solve the technical problem of unified modeling and real-time deduction of dynamic objects and static environment in a three-dimensional geographic environment. First, a camera pose is calculated by a motion recovery structure algorithm to generate a sparse point cloud, and a dynamic foreground object in a monitoring video is separated to extract motion features. Then, a three-dimensional Gaussian distribution set is initialized based on the sparse point cloud, semantic features are extracted by a visual encoder and are mapped to corresponding Gaussian distributions. Next, a topological graph structure of the Gaussian distributions is constructed, and the motion features are used as initial excitation to iteratively update the coordinate offset and appearance change of each distribution by message passing calculation. Finally, the Gaussian distribution attributes are dynamically updated, and a continuous deduction image sequence is rendered by differentiable rasterization to realize high-fidelity real-time simulation of the dynamic evolution of the three-dimensional geographic environment.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method for real-time intelligent inference of three-dimensional geographic environment based on multimodal large models. Background Technology

[0002] With the frequent occurrence of sudden natural disasters such as urban flooding, forest fires, landslides, and mudslides, emergency command, fire rescue, and urban flood control departments urgently need a high-fidelity, visualized real-time simulation technology capable of projecting the evolution of a large-scale three-dimensional geographic scene over several minutes after a disaster, within seconds to tens of seconds of its occurrence. This technology is crucial for rapidly determining evacuation routes, deploying rescue forces, and assessing secondary disaster risks. However, existing technologies still suffer from substantial shortcomings in practical applications.

[0003] Existing methods mainly fall into the following categories: Traditional numerical simulation methods based on meshes or voxels have huge computational overhead, and a single complete simulation usually takes tens of minutes to several hours, which cannot meet the real-time decision-making needs of disaster sites; Methods based on neural radiation fields and their variants can generate photo-realistic rendering quality, but each frame of rendering requires hundreds of network samplings for each ray, resulting in extremely low frame rates, and the geometric structure is implicitly represented, making it difficult to explicitly edit and physically drive; Dynamic 3D Gaussian sputtering technology, which has emerged in recent years, has significantly improved training and rendering speed, but its geometric and appearance updates mainly rely on pixel-by-pixel density control or simple affine transformations. When facing non-rigid, large-deformation, and drastically changing disaster scenarios such as flood spread, fire jumps, and violent smoke plume diffusion, it is very easy to produce serious visual artifacts such as Gaussian drift, distribution breakage, clipping, and voids, making the simulation results unreliable. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a real-time intelligent simulation method for three-dimensional geographic environments based on multimodal large models, thus solving the problems mentioned in the background.

[0005] To achieve the above objectives, this invention employs the following technical solution: a real-time intelligent inference method for three-dimensional geographic environment based on a multimodal large model, comprising the following steps: S1. Acquiring oblique photographic images and surveillance videos of the target scene; calculating the intrinsic and extrinsic poses of the image camera using a structure-of-motion reconstruction algorithm to generate sparse point cloud data; separating the dynamic foreground objects from the background in the surveillance video; and extracting the motion optical flow feature vectors of the foreground objects; S2. Initializing a three-dimensional Gaussian distribution set with the sparse point cloud as the centroid; defining the attributes of each three-dimensional Gaussian distribution, including spatial mean coordinates, covariance matrix, opacity, and spherical harmonic coefficients; inputting the oblique photographic images into a visual encoder to generate semantic feature maps; and combining the projection transformation matrix corresponding to the camera pose to map the semantic vectors in the semantic feature maps to the corresponding... S3. Construct a K-nearest neighbor topology graph of the three-dimensional Gaussian distribution set based on the spatial mean coordinates. Treat each three-dimensional Gaussian distribution as a graph node. Generate a diffusion coefficient matrix by analyzing the texture changes of the monitoring video stream. Use the motion optical flow feature vector as the initial excitation of the graph node. Perform message passing calculation in the topology graph. Iteratively update the hidden state of each graph node. Calculate the spatial coordinate offset and spherical harmonic coefficient change of each three-dimensional Gaussian distribution within the future preset time step. S4. Apply the calculated spatial coordinate offset and spherical harmonic coefficient change to the three-dimensional Gaussian distribution set frame by frame. Dynamically update the covariance matrix and opacity. Project the updated three-dimensional Gaussian distribution set onto the two-dimensional viewport plane through the differentiable Gaussian rasterization rendering pipeline to synthesize a continuous inference image sequence.

[0006] Further, the process of acquiring oblique photographic images and surveillance videos of the target scene, and using the structure-of-motion (SOM) algorithm to solve the intrinsic and extrinsic poses of the camera to generate sparse point cloud data is as follows: Multi-scale Gaussian difference detection is performed on the oblique photographic images to detect scale-invariant feature points. Cross-image keypoint matching is performed using the nearest neighbor to second nearest neighbor distance ratio, and mismatches are eliminated. The image pair with the most matching point pairs is selected as the initial image pair. The fundamental matrix is ​​solved using epipolar geometry constraints and triangulated to generate the initial 3D point cloud and initial camera pose. Perspective n-point registration is performed frame by frame on the remaining images to obtain the rotation and translation matrix for each frame. A global reprojection error objective function containing all camera intrinsic and extrinsic poses and 3D point coordinates is constructed. The objective function is iteratively minimized using the nonlinear least squares method until the reprojection error converges, outputting globally consistent sparse point cloud data.

[0007] Furthermore, the specific process of separating dynamic foreground objects from the background in the surveillance video and extracting the motion optical flow feature vector of the foreground objects is as follows: Temporal sampling of the surveillance video is performed, and a static background model of the scene is constructed using a Gaussian mixture model. The pixel difference between the current frame and the background model is calculated, and a binary foreground mask is generated using an adaptive threshold. Morphological closing and opening operations are performed on the binary foreground mask to fill the holes inside the foreground region and filter out isolated noise. The complete outline of the foreground object is extracted through connected component analysis. The pyramid optical flow algorithm is used only in the region of interest covered by the foreground object outline to calculate the pixel displacement vector between adjacent frames, and a dense optical flow field containing horizontal and vertical components is constructed. The average velocity vector and acceleration vector in the region of the foreground object are calculated and combined with the curl and divergence features of the optical flow field to form a motion optical flow feature vector describing the instantaneous motion state of the dynamic foreground object.

[0008] Furthermore, a set of three-dimensional Gaussian distributions is initialized with the sparse point cloud as the centroid. The attributes of each three-dimensional Gaussian distribution are defined, including spatial mean coordinates, covariance matrix, opacity, and spherical harmonic coefficients. The specific process of inputting the oblique photogrammetric image into the visual encoder to generate a semantic feature map is as follows: traverse each three-dimensional point in the sparse point cloud, directly assign its coordinates to the spatial mean coordinates of the corresponding three-dimensional Gaussian distribution, calculate the mean Euclidean distance between the point and its nearest neighbor, and construct an initial isotropic covariance matrix accordingly; initialize the opacity to a preset constant, initialize the spherical harmonic coefficients of each order to zero, input all oblique photogrammetric images into the pre-trained visual converter model and remove the encoder of the classification head, extract the last layer of feature map, and restore it to the original image resolution through bilinear upsampling to obtain the pixel-level semantic feature map corresponding to the pixels of the input image.

[0009] Furthermore, combining the projection transformation matrix corresponding to the camera pose, the specific process of mapping the semantic vectors in the semantic feature map to the three-dimensional Gaussian distribution at the corresponding projection position is as follows: For each three-dimensional Gaussian distribution, it is determined whether the Gaussian distribution is within the camera's view frustum based on the solved camera extrinsic pose and intrinsic parameter matrices, and all effective viewing angles are selected. The spatial mean coordinates are projected onto the two-dimensional pixel plane of each effective viewing angle according to the corresponding projection matrix to obtain the pixel index coordinates. A depth buffer test is performed to remove occluded projection points. The semantic feature vectors sampled at the corresponding pixel index of all effective viewing angles are weighted and averaged according to the reciprocal of the projection distance. The weighted average semantic vectors are then accumulated to the spherical harmonic coefficients and opacity properties of the three-dimensional Gaussian distribution.

[0010] Furthermore, based on the spatial mean coordinates, a K-nearest neighbor topological graph of a three-dimensional Gaussian distribution set is constructed. Each three-dimensional Gaussian distribution is used as a graph node. The specific process of generating the diffusion coefficient matrix by analyzing the texture changes of the surveillance video stream is as follows: Calculate the Euclidean distance between the spatial mean coordinates of any two points in the three-dimensional Gaussian distribution set. Select the K nearest neighbors of each Gaussian distribution to establish undirected edges and generate a spatial adjacency matrix. Project the spatial mean coordinates of the Gaussian distributions at both ends of each edge onto the image plane of the current frame of the surveillance video using the camera extrinsic pose and intrinsic parameter matrix. Extract the local image gradient magnitude of the projection point and calculate the gradient difference between the two points. Use a Gaussian kernel function to convert the gradient difference into edge weights. The larger the weight, the stronger the physical conductivity. Traverse all edges to obtain the complete diffusion coefficient matrix.

[0011] Furthermore, the motion optical flow feature vector is used as the initial excitation of the graph nodes. Message passing computation is performed in the topology graph to iteratively update the hidden states of each graph node. The specific process of calculating the spatial coordinate offset and spherical harmonic coefficient change of each three-dimensional Gaussian distribution within a future preset time step is as follows: The motion optical flow feature vector is mapped to the initial hidden state through a multilayer perceptron. Within each prediction time step, the K-neighborhood hidden states are weighted and aggregated according to the diffusion coefficient matrix to obtain the neighborhood message. The neighborhood message is concatenated with the hidden state of the current node at the previous time step and input into a gated recurrent unit for state update. The updated hidden states are sent to two independent fully connected regression heads, which output the three-dimensional spatial displacement vector and the increment of each order of spherical harmonic coefficient for that time step, respectively, as the spatial coordinate offset and spherical harmonic coefficient change for that time step.

[0012] Furthermore, the calculated spatial coordinate offset and spherical harmonic coefficient change are applied frame by frame to the three-dimensional Gaussian distribution set. The specific process of dynamically updating the covariance matrix and opacity is as follows: The spatial coordinate offset is directly accumulated to the spatial mean coordinate of the corresponding three-dimensional Gaussian distribution, and the spherical harmonic coefficient change is accumulated to the original spherical harmonic coefficient; for each Gaussian distribution, the relative displacement change with its K nearest neighbors before and after the update is calculated, a local Jacobian matrix is ​​constructed, and an affine transformation of the Jacobian matrix is ​​applied to the original covariance matrix to achieve shape stretching and rotation. At the same time, the rate of change of the number of Gaussian distributions in the local K nearest neighbor sphere is statistically analyzed. The opacity is reduced proportionally in areas where the number decreases and increased proportionally in areas where the number increases.

[0013] Furthermore, the specific process of projecting the updated 3D Gaussian distribution set onto the 2D viewport plane through the differentiable Gaussian rasterization rendering pipeline to synthesize a continuous inference image sequence is as follows: the updated 3D Gaussian distribution set is transformed to the current viewport camera coordinate system, and after frustum clipping, the 2D covariance matrix of each Gaussian distribution on the imaging plane is calculated. All 2D Gaussians are cardinally sorted by depth, a list of contributing Gaussians is established for each pixel, and the color and opacity contributions are accumulated sequentially by the alpha mixing formula in depth order to obtain a single-frame rendered image. The above rendering process is continuously executed according to the time step to generate a continuous dynamic inference video sequence.

[0014] The present invention has the following beneficial effects:

[0015] (1) A real-time intelligent inference method for 3D geographic environment based on a multimodal large model achieves synchronous reconstruction and explicit binding of the geometric structure and semantic features of 3D scenes through multimodal visual fusion of oblique photographic images and surveillance videos. The semantic vectors generated by the multimodal large model are directly assigned 3D Gaussian distributed semantic attributes. Compared with traditional projection methods based on texture or single-view features, this invention can maintain higher projection consistency and semantic stability in complex geographic environments. This process significantly improves the accuracy and editability of 3D scene representation and enhances the mapping ability of visual information to 3D parameter space.

[0016] (2) A real-time intelligent inference method for 3D geographic environment based on a multimodal large model is proposed. This method constructs a graph structure based on a 3D Gaussian distribution and uses video texture changes and optical flow as dynamic driving factors. It calculates the future geometric and appearance change trajectory through a message passing mechanism, and then uses a differentiable Gaussian rasterization rendering pipeline for continuous image synthesis. This framework effectively overcomes the problems of computational redundancy, high rendering latency, and uncontrollable geometric updates in dynamic scenes caused by traditional implicit field or voxel-based rendering methods. It enables complex scenes to maintain stable 3D structures and high frame rate rendering performance under significant deformation. This optimization significantly improves the real-time performance and visual consistency of 3D dynamic rendering.

[0017] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0018] Figure 1 This is a flowchart of the real-time intelligent simulation method for three-dimensional geographic environment based on a multimodal large model according to the present invention. Detailed Implementation

[0019] This application's embodiments solve the problems of existing 3D dynamic scene inference methods, such as difficulty in stable updating of geometric structures, inconsistent semantic mapping, easy artifacts in rendering non-rigid large deformation scenes, and low overall inference frame rate, by using a real-time intelligent inference method based on multimodal large models for 3D geographic environments.

[0020] The overall approach of the scheme in this application is as follows: A multimodal input source is constructed using oblique photographic images and surveillance video. The initial sparse geometry of the scene is obtained using the structure-of-motion (SOMO) algorithm, and dynamic foreground information is extracted through optical flow and texture changes. Based on this, a three-dimensional Gaussian distribution set is established with sparse point clouds as the core, and the semantic features generated by the multimodal large model are accurately projected into the three-dimensional structure to form an explicit and updatable three-dimensional semantic representation. Subsequently, a topological graph is constructed based on the spatial neighborhood of the three-dimensional Gaussian distribution. Optical flow features are used as dynamic driving signals, and geometric displacement and appearance changes in future time steps are calculated through a message passing mechanism. Finally, using a differentiable rasterization rendering method, the updated three-dimensional Gaussian distribution set is projected in real time into a continuous sequence of inferred images, achieving rapid, stable, and high-fidelity dynamic inference of complex three-dimensional geographical environments.

[0021] Please see Figure 1 This invention provides a technical solution: a real-time intelligent inference method for three-dimensional geographic environment based on a multimodal large model, comprising the following steps: S1. Acquiring oblique photographic images and surveillance videos of the target scene, calculating the intrinsic and extrinsic poses of the image camera using the structure-of-motion reconstruction algorithm, generating sparse point cloud data, separating the dynamic foreground objects from the background in the surveillance video, and extracting the motion optical flow feature vectors of the foreground objects; S2. Initializing a three-dimensional Gaussian distribution set with the sparse point cloud as the centroid, defining the attributes of each three-dimensional Gaussian distribution including spatial mean coordinates, covariance matrix, opacity, and spherical harmonic coefficients, inputting the oblique photographic images into a visual encoder to generate semantic feature maps, and combining the projection transformation matrix corresponding to the camera pose to map the semantic vectors in the semantic feature maps to the corresponding projection positions. S3. Construct a K-nearest neighbor topology graph of the three-dimensional Gaussian distribution set based on the spatial mean coordinates. Treat each three-dimensional Gaussian distribution as a graph node. Generate a diffusion coefficient matrix by analyzing the texture changes of the monitoring video stream. Use the motion optical flow feature vector as the initial excitation of the graph node. Perform message passing calculation in the topology graph, iteratively update the hidden state of each graph node, and calculate the spatial coordinate offset and spherical harmonic coefficient change of each three-dimensional Gaussian distribution within the future preset time step. S4. Apply the calculated spatial coordinate offset and spherical harmonic coefficient change to the three-dimensional Gaussian distribution set frame by frame, dynamically update the covariance matrix and opacity, and project the updated three-dimensional Gaussian distribution set onto the two-dimensional viewport plane through the differentiable Gaussian rasterization rendering pipeline to synthesize a continuous inference image sequence.

[0022] In this implementation scheme, step S1 acquires and preprocesses multimodal inputs to construct initial geometric and motion representations. Specifically, this includes using the Structure for Motion Reconstruction (SfM) algorithm to solve for camera parameters and pose (the camera intrinsic parameter matrix describes inherent parameters such as focal length, principal point, and distortion; the extrinsic parameter pose describes the camera's rotation and translation in the world coordinate system) on the oblique photographic image, and obtaining a sparse point cloud through triangulation (a sparse point cloud is a set of points composed of several matched and triangulated 3D sampling points, used to approximate the main geometric skeleton of the scene); simultaneously, background modeling and foreground segmentation are performed on the surveillance video, and motion optical flow feature vectors are extracted in the foreground region using methods such as pyramid optical flow (optical flow describes the displacement field of pixels in the time dimension; optical flow feature vectors typically contain velocity, acceleration, and curl / divergence information). By separating and normalizing the geometric skeleton (sparse point cloud) and foreground motion (optical flow vector), a reliable geometric and motion basis is established for accurately mapping semantic information and dynamic driving onto the 3D structure, thereby improving the accuracy and robustness of subsequent modeling and inference. Step S2 initializes the explicit 3D representation with sparse point clouds and completes semantic binding. Specifically, this involves constructing a set of 3D Gaussian distributions with the sparse point cloud points as centroids (each 3D Gaussian distribution is described by parameters such as spatial mean, covariance matrix, opacity, and spherical harmonic coefficients; the covariance matrix controls the shape and scale of the Gaussian in 3D space; opacity represents the transparency / occlusion properties of voxel-based contributions; and the spherical harmonic coefficients are used to represent the directional distribution of appearance with low-dimensional expansion). The oblique photographic image is then input into a visual encoder (a convolutional or visual transformer-based feature extraction network used to generate pixel-level high-dimensional semantic feature maps). The resulting semantic vectors are mapped onto the corresponding 3D Gaussian distributions using the camera's projection transformation matrix. The projection transformation matrix maps 3D points to a 2D pixel plane, and a depth buffer is used to remove occluded projection sampling points. Explicitly associating pixel-level semantics with 3D Gaussian parameters preserves image semantic details and enables the 3D representation to have updatable semantic properties, thereby resolving semantic inconsistencies across multiple viewpoints and providing semantic constraints for controllable editing and physical driving. Step S3: Establish the neighborhood topology of a 3D Gaussian and perform temporal dynamics reasoning based on video-driven processing. Specifically, this includes constructing a K-nearest neighbor topology graph (KNN topology graph: a directed or undirected graph structure defined by spatial nearest neighbor relationships as edges) based on the spatial mean coordinates of the 3D Gaussian. Local texture gradients are extracted by projecting each Gaussian onto the video frame and a diffusion coefficient matrix is ​​constructed using Gaussian kernel mapping (the diffusion coefficient matrix characterizes the strength of interactions between neighborhoods, such as mapping texture differences to conduction / coupling strength). Foreground optical flow features are used as the initial excitation of graph nodes. The hidden state of the nodes is iteratively updated in a graph neural network-style message passing computation (hidden state: the internal representation of a node at a time step, carrying neighborhood information and historical dynamics). The spatial coordinate offset and spherical harmonic coefficient change of each Gaussian in the future time step are obtained by decoder regression.Message passing computation refers to the process of feature aggregation, updating, and propagation on a topological graph according to edge weights; the diffusion coefficient is used to regulate the transmission intensity of neighborhood information. Through explicit topology and a learnable propagation mechanism, the transmission and fusion of local dynamics in the 3D structure are realized, enabling non-rigid, locally coupled spatiotemporal changes to be predicted in a structured and interpretable manner in the 3D representation, significantly improving the dynamic consistency and physical perception capabilities in complex topological change scenarios. Step S4: The predicted geometric and appearance changes are applied frame-by-frame and differentiable rendered to generate a continuous inference visualization. Specifically, this involves frame-by-frame superimposing of the calculated spatial offset and spherical harmonic coefficient increments onto a 3D Gaussian ensemble, updating the covariance matrix and opacity based on positional changes (using Jacobi transform to approximate the effect of deformation on covariance, and adjusting opacity based on local node density), then projecting the 3D Gaussian ensemble into 2D pixels through a differentiable Gaussian rasterization rendering pipeline (differentiable rasterization: the rendering process is mathematically differentiable, facilitating end-to-end training or gradient propagation), sorting the pixels based on depth, and generating single-frame images by alpha mixing to accumulate color and opacity. Multiple consecutive frames are then synthesized into a deduced video sequence. Alpha mixing is a standard volume rendering strategy that accumulates opacity and color contributions along the line of sight; differentiable rendering emphasizes that the rendering operator supports gradient backpropagation. While maintaining the continuity of the 3D structure and rendering efficiency, this approach achieves high-fidelity visualization and differentiable optimization of the prediction results. It can be used for real-time decision display and also facilitates end-to-end fine-tuning of the prediction module using supervisory signals or loss functions, thereby reducing rendering artifacts and improving the interpretability and credibility of the deduced results.

[0023] Specifically, the process of acquiring oblique photographic images and surveillance videos of the target scene, and using the structure-of-motion (SOM) algorithm to solve for the intrinsic and extrinsic poses of the camera images to generate sparse point cloud data is as follows: Multi-scale Gaussian difference detection is performed on the oblique photographic images to detect scale-invariant feature points. Cross-image keypoint matching is performed based on the nearest neighbor to second nearest neighbor distance ratio, and mismatches are eliminated. The image pair with the most matching point pairs is selected as the initial image pair. The fundamental matrix is ​​solved using epipolar geometric constraints and triangulated to generate the initial 3D point cloud and initial camera pose. Perspective n-point registration is performed frame by frame on the remaining images to obtain the rotation and translation matrix for each frame. A global reprojection error objective function containing all camera intrinsic and extrinsic poses and 3D point coordinates is constructed. The objective function is iteratively minimized using the nonlinear least squares method until the reprojection error converges, outputting globally consistent sparse point cloud data.

[0024] In this implementation scheme, the Structure for Motion Restoration (SfM) algorithm is used to achieve accurate spatial registration and 3D geometric reconstruction of multi-source images. Its core lies in robust feature matching, pose solving, and global optimization. First, multi-scale Gaussian difference is used to detect scale-invariant feature points, and cross-image keypoint matching is performed based on the ratio of nearest neighbor to second nearest neighbor distances. This step, through ratio constraints, eliminates a large number of erroneous matches caused by repeated textures or lighting variations, significantly improving matching accuracy and ensuring the basic quality of subsequent triangulation. The image pair with the most matched point pairs is selected as the initial image pair, and the fundamental matrix is ​​solved through epipolar geometric constraints. And triangulate the three-dimensional points according to the following formula: ;in: :No. Estimated coordinates of a point in three-dimensional space; :No. a: candidate coordinates of a 3D spatial point; b: image index in the image pair, taking 1 or 2; The image corresponding to the b-th image Two-dimensional pixel coordinates; The projection matrix of the b-th image is composed of the camera's intrinsic and extrinsic parameters. : Represents the reprojection distance from the 2D point to the projection line. This step establishes the first local coordinate system and generates the initial 3D point cloud. Subsequently, perspective n-point (PnP) registration is performed frame by frame on the remaining images to obtain the rotation and translation matrix for each frame. This approach incorporates all camera poses into a global coordinate system. To eliminate local error accumulation and obtain a 3D model with the highest geometric consistency, this scheme constructs a global reprojection error objective function, which is optimized using a nonlinear least squares method for bundle adjustment. The goal is to minimize the difference between the observed 2D points and their reprojected 3D positions. ;in: : Contains the intrinsic and extrinsic pose parameter set for all cameras; :The set of coordinates of all three-dimensional points; The collection of all image frames; :image A set of three-dimensional points visible in the image; Robust loss function, used to reduce the impact of outliers (such as isolated points) on optimization; :No. The first camera observed the The reprojection error vector of a 3D point , Let be the projection function. This global optimization step ensures the geometric accuracy and consistency of all sparse point clouds in a unified coordinate system, guaranteeing the quality of subsequent 3D Gaussian distribution initialization. Iteratively minimize until the reprojection error is below the convergence threshold. Determine the threshold A combination of empirical values ​​and median absolute deviation of residuals (MAD) is used to ensure global consistency of the results.

[0025] Specifically, the process of separating dynamic foreground objects from the background in surveillance video and extracting the motion optical flow feature vector of the foreground objects is as follows: Temporal sampling of the surveillance video is performed, and a static background model of the scene is constructed using a Gaussian mixture model. The pixel difference between the current frame and the background model is calculated, and a binary foreground mask is generated using an adaptive threshold. Morphological closing and opening operations are performed on the binary foreground mask to fill the holes inside the foreground region and filter out isolated noise. The complete outline of the foreground object is extracted through connected component analysis. The pyramid optical flow algorithm is used only in the region of interest covered by the foreground object outline to calculate the pixel displacement vector between adjacent frames, and a dense optical flow field containing horizontal and vertical components is constructed. The average velocity vector and acceleration vector in the foreground object region are calculated and combined with the curl and divergence features of the optical flow field to form a motion optical flow feature vector describing the instantaneous motion state of the dynamic foreground object.

[0026] In this implementation scheme, dynamic foreground objects in the real-time monitoring video stream are accurately separated, and their high-dimensional motion features are extracted for subsequent dynamic activation of the graph neural network. First, the monitoring video is temporally sampled, and a static background model of the scene is constructed using a Gaussian mixture model. This step, through statistical learning of pixel color and variance, effectively handles slow changes in illumination and slight background jitter, thereby accurately distinguishing moving pixels. The current frame is then calculated. With background model pixel difference And through adaptive threshold Generate a foreground binary mask Threshold The determination method combines the local region entropy value of the image with the Otsu algorithm, which enables the accurate differentiation of moving targets in local regions of varying complexity. ;in: At any moment Pixel coordinates Foreground mask value at the location; The difference between the current frame and the background model at this pixel; Adaptive segmentation threshold. Mask. The dynamic region requiring depth motion analysis was fixed. Morphological closing and opening operations were then performed on the binarized foreground mask. This step effectively eliminated small holes and scattered noise, ensuring the integrity and coherence of the moving object's contour. Only the region of interest covered by the foreground object's contour was analyzed. Within this framework, the pyramid optical flow algorithm is used to calculate the pixel displacement vector between adjacent frames. Constructing a dense optical flow field Finally, in order to compress the optical flow field into high-dimensional, physically meaningful features, this scheme calculates the foreground object region. Average velocity vector within acceleration vector and the curl of the flow field with divergence Combined to form motion optical flow feature vectors : ;in: The mean of the velocity vectors of all pixels within the foreground region, representing the overall motion trend; The rate of change of the average velocity over time within the foreground region represents the state of acceleration or deceleration of the motion. The curl of the optical flow field represents the rotational and eddy current characteristics of fluid or group motion, reflecting the complexity of the motion; The divergence of the optical flow field represents the local convergence or divergence characteristics of the motion, reflecting the state of congestion or diffusion; Concat( ): Vector concatenation operation. This feature vector The compression of motion speed, rate of change, and fluid properties becomes the initial stimulus driving the graph neural network (GNN) to perform real-time inference.

[0027] Specifically, a set of three-dimensional Gaussian distributions is initialized with the sparse point cloud as the centroid. The attributes of each three-dimensional Gaussian distribution are defined, including spatial mean coordinates, covariance matrix, opacity, and spherical harmonic coefficients. The specific process of inputting the oblique photogrammetric image into the visual encoder to generate a semantic feature map is as follows: traverse each three-dimensional point in the sparse point cloud, directly assign its coordinates to the spatial mean coordinates of the corresponding three-dimensional Gaussian distribution, calculate the mean Euclidean distance between the point and its nearest neighbor, and construct an initial isotropic covariance matrix accordingly; initialize the opacity to a preset constant, initialize the spherical harmonic coefficients of each order to zero, input all oblique photogrammetric images into the pre-trained visual converter model and remove the encoder of the classification head, extract the last layer of feature map, and restore it to the original image resolution through bilinear upsampling to obtain the pixel-level semantic feature map corresponding to the pixels of the input image.

[0028] In this implementation scheme, the initialization of the 3D Gaussian distribution set and the generation of the semantic feature map aim to provide a multimodal benchmark of geometry, appearance, and semantics for real-time inference. First, each 3D point in the sparse point cloud... The coordinates are directly assigned to the spatial mean coordinates of the corresponding three-dimensional Gaussian distribution. This achieves the geometric transformation from discrete points to a continuum. Subsequently, through calculation... With The mean Euclidean distance between the nearest neighbors Construct the initial isotropic covariance matrix : ;in: Initial covariance matrix; : Scale factor, used to control the size of the initial Gaussian distribution; : No. The mean Euclidean distance between the nearest neighbors; : A third-order identity matrix. This construction method guarantees that the initial Gaussian distribution is an isotropic sphere, maximizing its deformation potential in subsequent derivations. Simultaneously, the opacity is initialized. Initialize the spherical harmonic coefficients of each order as preset constants. The initial value is zero, providing a neutral starting point for subsequent color and transparency optimization. Subsequently, all oblique photogrammetric images are fed into a pre-trained Visual Transformer (ViT) model, the encoder of the classification head is removed, and the final layer feature maps are extracted. By using bilinear upsampling to restore the original image resolution, pixel-level semantic feature maps corresponding one-to-one with the input image pixels are obtained. : ;in: Pixel coordinates The semantic feature vector at the location; : Bilinear interpolation function; : Low-resolution feature map output by the encoder; Normalized feature map coordinates. This step, through a deep learning model, extracts and spatially aligns high-dimensional semantic features of the scene image, which is crucial for achieving multimodal information fusion.

[0029] Specifically, the process of mapping the semantic vectors in the semantic feature map to the corresponding 3D Gaussian distribution at the projection position, based on the projection transformation matrix corresponding to the camera pose, is as follows: For each 3D Gaussian distribution, it is determined whether the Gaussian distribution is within the camera's view frustum based on the solved camera extrinsic pose and intrinsic parameter matrices, and all effective viewing angles are selected. The spatial mean coordinates are projected onto the 2D pixel plane of each effective viewing angle according to the corresponding projection matrix to obtain the pixel index coordinates. A depth buffer test is performed to remove occluded projection points. The semantic feature vectors sampled at the corresponding pixel index of all effective viewing angles are weighted and averaged using the reciprocal of the projection distance as the weight. The weighted average semantic vectors are then accumulated to the spherical harmonic coefficients and opacity properties of the 3D Gaussian distribution.

[0030] In this implementation scheme, the semantic vectors in the semantic feature map are efficiently mapped and accumulated onto the attributes of a three-dimensional Gaussian distribution through projection transformation, realizing knowledge distillation from two-dimensional high-dimensional semantic information to three-dimensional spatial attributes. First, for each three-dimensional Gaussian distribution, based on the solved camera extrinsic pose parameters... With intrinsic parameter matrix Determine whether it is within the camera's field of view and filter out all valid viewing angles. Subsequently, using the projection matrix The spatial mean coordinates of the Gaussian distribution Pixel index coordinates are obtained by projecting the image onto a two-dimensional pixel plane at each effective viewing angle. : ;in: Projected onto camera Pixel index coordinates on the image; No. Projection matrix of each camera; Spatial mean coordinates of a three-dimensional Gaussian distribution; Equivalence relations in homogeneous coordinates. This step establishes the geometric correspondence between the 3D Gaussian distribution and the 2D semantic feature map. Next, a depth buffer test is performed to remove occluded projection points, ensuring that the sampled semantic features come from the visible surface closest to the 3D Gaussian distribution. Based on projection distance... The reciprocal of the weight is used as the weight. The semantic feature vector obtained by sampling at the corresponding pixel index of all valid viewpoints Perform weighted average : ;in: The final semantic vector after aggregation; The set of cameras that can observe this Gaussian distribution; Weighting coefficients are determined by the reciprocal of the projected distance. ; :No. The semantic feature vectors are sampled from each camera image. This weighted aggregation step effectively fuses multi-view information and suppresses noise from distant or oblique views. Finally, the weighted average semantic vectors are... Spherical harmonic coefficients accumulated in this three-dimensional Gaussian distribution With opacity property: , ;in: :Update the spherical harmonic coefficient vector before and after; Opacity values ​​before and after update; The learning weight coefficients are determined by the proportion of gradient contributions of the training loss function in the initial optimization phase to semantic features, spherical harmonics, and opacity. This step uses high-dimensional semantic information as the driving force for initialization, so that the spherical harmonics and opacity have a clear semantic bias in the early stages of model training, accelerating scene convergence and attribute optimization.

[0031] Specifically, the process of constructing a K-nearest neighbor topology graph of a three-dimensional Gaussian distribution set based on spatial mean coordinates, with each three-dimensional Gaussian distribution as a graph node, and generating a diffusion coefficient matrix by analyzing the texture changes of the surveillance video stream is as follows: Calculate the Euclidean distance between the spatial mean coordinates of any two points within the three-dimensional Gaussian distribution set; select the K nearest neighbors for each Gaussian distribution to establish undirected edges and generate a spatial adjacency matrix; project the spatial mean coordinates of the Gaussian distributions at both ends of each edge onto the image plane of the current frame of the surveillance video using the camera's extrinsic pose and intrinsic parameter matrix; extract the local image gradient magnitude at the projection point and calculate the gradient difference between the two points; convert the gradient difference into edge weights using a Gaussian kernel function, where a larger weight indicates stronger physical conductivity; and traverse all edges to obtain the complete diffusion coefficient matrix.

[0032] In this implementation scheme, a topological graph is constructed using geometric proximity, and a physically meaningful diffusion coefficient matrix is ​​generated by utilizing texture differences in real-time video. First, the Euclidean distance between the spatial mean coordinates of any two points within the 3D Gaussian distribution set is calculated, and the nearest point is selected for each Gaussian distribution. Undirected edges are established between each neighbor, thereby generating a spatial adjacency matrix. : ;in: : The Euclidean distance between the coordinates of the q-th and e-th Gaussian distribution spatial mean; : Spatial mean coordinates of the q-th and e-th Gaussian distributions; The Euclidean distance norm is used to establish a sparse, local topological structure, ensuring that the GNN interacts only within physically adjacent regions. Subsequently, the Gaussian spatial mean coordinates at both ends of each edge are used to project the data onto the camera matrix of the current frame of the monitored video. Project onto the image plane of this frame and extract the projection points. Local image gradient magnitude Calculate the gradient difference between the two projection points. The gradient difference is converted into edge weights using a Gaussian kernel function. : ;in: The diffusion weight of the edge connecting the q-th and e-th Gaussian distributions; Projection point The local image gradient magnitude at that location; The bandwidth parameter of the Gaussian kernel function controls the sensitivity of texture differences to diffusion; its value is determined by empirical variance of the gradient statistics of the scene image. The absolute difference in gradient magnitudes. This weight. This constitutes the complete diffusion coefficient matrix. The smaller the weight (the greater the gradient difference, such as at the edge of an object), the greater the obstacle to message transmission. This simulates the inhibitory effect of physical boundaries on information diffusion, ensuring the physical rationality of the deduction.

[0033] Specifically, the process of using the motion optical flow feature vector as the initial excitation of the graph nodes, performing message passing calculations in the topology graph, iteratively updating the hidden states of each graph node, and calculating the spatial coordinate offset and spherical harmonic coefficient change of each three-dimensional Gaussian distribution within a future preset time step is as follows: The motion optical flow feature vector is mapped to the initial hidden state through a multilayer perceptron. Within each prediction time step, the K-neighborhood hidden states are weighted and aggregated according to the diffusion coefficient matrix to obtain the neighborhood message. The neighborhood message is concatenated with the hidden state of the current node at the previous time step and input into the gated recurrent unit for state update. The updated hidden states are sent to two independent fully connected regression heads, which output the three-dimensional spatial displacement vector and the increment of each order of spherical harmonic coefficient for that time step, respectively, as the spatial coordinate offset and spherical harmonic coefficient change for that time step.

[0034] In this implementation scheme, the Graph Neural Network (GNN) receives motion optical flow features as initial stimuli and performs iterative message passing in the topological graph to calculate dynamic changes in the scene, thereby achieving intelligent prediction of the future state of complex geographical environments. First, the extracted motion optical flow feature vectors... The hidden states are mapped to a high-dimensional space using a multilayer perceptron (MLP) and serve as the initial hidden states for the graph nodes (a 3D Gaussian distribution). : ;in: :No. The initial hidden states of the Gaussian distribution (graph nodes); : Multilayer perceptron mapping function; : Corresponding to the A Gaussian-distributed motion optical flow feature vector. This step efficiently encodes the dynamic information in the 2D video stream and injects it into the 3D topological graph structure. At each prediction time step... Internally, the graph network is based on the diffusion coefficient matrix. right We perform weighted aggregation of the hidden states in the neighborhood to obtain the neighborhood messages. : ;in: :No. Each node Messages received from the neighborhood at any given time; :node of Nearest neighbor set; :node and The diffusion weights between them (from the diffusion coefficient matrix) ); Neighbor nodes exist The hidden state at any given moment. This aggregation operation ensures the mutual influence between Gaussian distributions and makes information transfer follow the physical connectivity defined by texture differences. Subsequently, neighborhood messages are... The hidden state of the current node at the previous time step After concatenation, the data is input into a gated loop unit (GRU) for state updates. ;in: :No. Each node The hidden state after constant updates; Gated recurrent unit function. The GRU structure enables the network to effectively capture and remember complex temporal dependencies, achieving long-term dynamic prediction. Finally, the updated hidden state. Two independent fully connected regression heads, RegrHead(·), are fed into the head, and each head outputs the three-dimensional spatial displacement vector for that time step. and the increments of spherical harmonic coefficients of each order : ;in: :Predicted 3D spatial coordinate offset; : The predicted change in spherical harmonic coefficients; Independent, fully connected layer for regression prediction. This dual-head decoding mechanism effectively separates the internally learned dynamic features into geometric motion and appearance changes, achieving accurate, decoupled prediction of the core properties of 3D Gaussians.

[0035] Specifically, the calculated spatial coordinate offset and spherical harmonic coefficient change are applied frame by frame to the three-dimensional Gaussian distribution set, and the covariance matrix and opacity are dynamically updated as follows: The spatial coordinate offset is directly accumulated to the spatial mean coordinate of the corresponding three-dimensional Gaussian distribution, and the spherical harmonic coefficient change is accumulated to the original spherical harmonic coefficient; for each Gaussian distribution, the relative displacement change with its K nearest neighbors before and after the update is calculated, a local Jacobian matrix is ​​constructed, and an affine transformation of the Jacobian matrix is ​​applied to the original covariance matrix to achieve shape stretching and rotation. At the same time, the rate of change of the number of Gaussian distributions in the local K nearest neighbor sphere is statistically analyzed, and the opacity is reduced proportionally in areas where the number decreases and increased proportionally in areas where the number increases.

[0036] In this implementation scheme, the spatial coordinate offset calculated by GNN is... With the change in spherical harmonic coefficient Frame-by-frame application to a 3D Gaussian distribution ensemble enables dynamic updates of geometry and appearance under the predicted state. First, the spatial coordinate offsets are directly accumulated to the spatial mean coordinates of the corresponding 3D Gaussian distribution using the principle of linear superposition. At the same time, the changes in spherical harmonic coefficients are accumulated to the original spherical harmonic coefficients. ;in: :No. and Spatial mean coordinates at time point; Spatial coordinate offset calculated by GNN; :No. and The vector of spherical harmonic coefficients at time t; The change in spherical harmonic coefficients calculated by the GNN. This step directly drives the evolution of the position and color of the 3D Gaussian distribution towards the predicted future state. Subsequently, for each Gaussian distribution, its relationship with... The relative displacement changes of nearest neighbors before and after the update are used to construct a Jacobian matrix describing the degree of local deformation. . use For the original covariance matrix Apply affine transformations to stretch and rotate the Gaussian shape: ;in: Covariance matrices before and after the update; The local Jacobian matrix, derived from the Gaussian distribution and The relative displacement change rate of nearest neighbors is constructed to capture local velocity gradients; The transpose of the local Jacobian matrix. This deformation step simulates geometric changes caused by non-rigid motions such as fluid shearing and material compression, maintaining the physical plausibility of the Gaussian representation. Simultaneously, through statistical local... Rate of change of Gaussian distribution in the nearest neighbor sphere The opacity is dynamically adjusted based on the principle of volume conservation approximation. ;in: Opacity values ​​before and after update; The scaling factor for adjusting opacity; Local The change in the number of Gaussian distributions within the nearest neighbor sphere; The initial Gaussian distribution number within the nearest neighbor spheres. This step ensures that volume compression ( When the opacity is 0), the opacity of the material increases, and when the volume expands, the opacity decreases, enhancing the visual realism of the inference results.

[0037] Specifically, the process of projecting the updated 3D Gaussian distribution set onto the 2D viewport plane through the differentiable Gaussian rasterization rendering pipeline to synthesize a continuous inference image sequence is as follows: the updated 3D Gaussian distribution set is transformed to the current viewport camera coordinate system, and after frustum clipping, the 2D covariance matrix of each Gaussian distribution on the imaging plane is calculated. All 2D Gaussians are cardinally sorted by depth, a list of contributing Gaussians is established for each pixel, and the color and opacity contributions are accumulated sequentially by the alpha mixing formula in depth order to obtain a single-frame rendered image. The above rendering process is continuously executed according to the time step to generate a continuous dynamic inference video sequence.

[0038] In this implementation, a differentiable Gaussian rasterization rendering pipeline is used to efficiently project the updated 3D Gaussian distribution set onto a 2D plane, synthesizing a continuous sequence of inferred images. First, the updated 3D Gaussian distribution set is transformed to the current viewport camera coordinate system C, and view frustum clipping is performed to remove 3D Gaussian distributions outside the viewport. Then, the 2D covariance matrix of each Gaussian distribution on the imaging plane is calculated: ;in: : A two-dimensional covariance matrix in screen space; Σ′: The Jacobian matrix projecting the 3D covariance onto the 2D plane; Σ′: The 3D covariance matrix in the camera coordinate system. This step determines the projection shape and size of each Gaussian distribution on the screen. Next, all 2D Gaussians are cardinally sorted by depth, a list of contributing Gaussians for each pixel is created, and colors are sequentially accumulated using the alpha mixing formula in depth order. With Opacity contribute: ; ;in: : The final synthesized pixel color; Glist: A list of contributing Gaussians sorted by depth; The color of the a-th Gaussian distribution (obtained by SH decoding); The opacity of the a-th Gaussian distribution; Cumulative transmittance represents the proportion of light rays that are not blocked before reaching the *a*th Gaussian ray. This rendering process approximates the volumetric rendering equation, achieving real-time, high-quality single-frame rendering of complex dynamic 3D scenes through efficient sorting and parallel computation. Finally, the above rendering process is executed continuously at a preset time step to generate a continuous dynamic inference video sequence Vseq, enabling real-time visualization and prediction of future geographical environment conditions.

[0039] In summary, this application has at least the following effects:

[0040] This method, based on a multimodal large model, enables real-time intelligent inference of 3D geographic environments, achieving a closed loop from multimodal heterogeneous data to real-time, high-fidelity dynamic 3D scene inference. This significantly improves the prediction accuracy and visualization efficiency of complex dynamic environments. First, a high-precision 3D Gaussian geometric foundation is constructed using a motion reconstruction structure and a visual encoder. Simultaneously, motion optical flow features from surveillance videos and high-dimensional semantic information from images are effectively fused and mapped onto a 3D Gaussian distribution, overcoming the shortcomings of traditional 3D reconstruction models in fusing semantic and motion information. This solves the problem of balancing real-time performance and physical plausibility in traditional inference methods. Finally, through a differentiable rasterization pipeline, the prediction results are directly rendered into a continuous video sequence, ensuring efficient and high-fidelity output of the inference results and providing powerful real-time intelligent support for geographic environment decision analysis.

[0041] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0042] This invention is described with reference to flowchart illustrations and / or block diagrams of systems, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0043] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0044] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0045] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0046] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for real-time intelligent simulation of 3D geographic environment based on multimodal large model, characterized in that, Includes the following steps: S1. Acquire oblique photographic images and surveillance videos of the target scene, solve the intrinsic and extrinsic poses of the image camera using the structure of motion reconstruction algorithm, generate sparse point cloud data, separate the dynamic foreground objects from the background in the surveillance video, and extract the motion optical flow feature vectors of the foreground objects. S2. Initialize a set of three-dimensional Gaussian distributions with sparse point clouds as centroids. Define the attributes of each three-dimensional Gaussian distribution, including spatial mean coordinates, covariance matrix, opacity and spherical harmonic coefficients. Input the oblique photogrammetric image into the visual encoder to generate a semantic feature map. Combine the projection transformation matrix corresponding to the camera pose to map the semantic vectors in the semantic feature map to the three-dimensional Gaussian distribution at the corresponding projection position. S3. Construct a K-nearest neighbor topology graph of a three-dimensional Gaussian distribution set based on spatial mean coordinates. Take each three-dimensional Gaussian distribution as a graph node. Generate a diffusion coefficient matrix by analyzing the texture changes of the monitoring video stream. Use the motion optical flow feature vector as the initial excitation of the graph node. Perform message passing calculation in the topology graph, iteratively update the hidden state of each graph node, and calculate the spatial coordinate offset and spherical harmonic coefficient change of each three-dimensional Gaussian distribution within a future preset time step. S4. The calculated spatial coordinate offset and spherical harmonic coefficient change are applied frame by frame to the three-dimensional Gaussian distribution set, the covariance matrix and opacity are dynamically updated, and the updated three-dimensional Gaussian distribution set is projected onto the two-dimensional viewport plane through the differentiable Gaussian rasterization rendering pipeline to synthesize a continuous inference image sequence.

2. The real-time intelligent inference method for three-dimensional geographic environment based on multimodal large model according to claim 1, characterized in that: The specific process of acquiring oblique photographic images and surveillance videos of the target scene, and then using the structure-of-motion (SOM) algorithm to calculate the intrinsic and extrinsic pose parameters of the camera to generate sparse point cloud data is as follows: Multi-scale Gaussian difference detection is performed on oblique photographic images to detect scale-invariant feature points. Cross-image key point matching is performed by the ratio of nearest neighbor to second nearest neighbor distance and mismatches are eliminated. The image pair with the most matching point pairs is selected as the initial image pair. The fundamental matrix is ​​solved by epipolar geometric constraints and triangulated to generate the initial 3D point cloud and the initial camera pose. For the remaining images, perform perspective n-point registration frame by frame to obtain the rotation and translation matrix for each frame. Construct a global reprojection error objective function that includes all camera intrinsic and extrinsic parameters, pose, and 3D point coordinates. Iteratively minimize the objective function using the nonlinear least squares method until the reprojection error converges, and output globally consistent sparse point cloud data.

3. The real-time intelligent inference method for three-dimensional geographic environment based on multimodal large model according to claim 2, characterized in that: The specific process of separating dynamic foreground objects from the background in surveillance video and extracting the motion optical flow feature vectors of the foreground objects is as follows: Temporal sampling is performed on the surveillance video, and a static background model of the scene is constructed using a Gaussian mixture model. The pixel difference between the current frame and the background model is calculated, and a foreground binary mask is generated using an adaptive threshold. Morphological closing and opening operations are performed on the binarized foreground mask to fill the holes inside the foreground region and filter out isolated noise. The complete foreground object contour is extracted through connected component analysis. The pyramid optical flow algorithm is used only in the region of interest covered by the foreground object contour to calculate the pixel displacement vector between adjacent frames and construct a dense optical flow field containing horizontal and vertical components. The average velocity vector and acceleration vector within the foreground object region are calculated and combined with the curl and divergence characteristics of the optical flow field to form a motion optical flow feature vector describing the instantaneous motion state of the dynamic foreground object.

4. The real-time intelligent inference method for three-dimensional geographic environment based on multimodal large model according to claim 1, characterized in that: A set of three-dimensional Gaussian distributions is initialized with sparse point clouds as centroids. The attributes of each three-dimensional Gaussian distribution are defined, including spatial mean coordinates, covariance matrix, opacity, and spherical harmonic coefficients. The specific process of inputting the oblique photogrammetric image into the visual encoder to generate semantic feature maps is as follows: Traverse each 3D point in the sparse point cloud, directly assign its coordinates to the spatial mean coordinates of the corresponding 3D Gaussian distribution, calculate the mean Euclidean distance between the point and its nearest neighbor, and construct the initial isotropic covariance matrix accordingly. The opacity is initialized to a preset constant, and the spherical harmonic coefficients of each order are initialized to zero. All oblique photogrammetric images are input into the pre-trained visual converter model and the encoder of the classification head is removed. The last layer feature map is extracted and restored to the original image resolution through bilinear upsampling to obtain the pixel-level semantic feature map corresponding to the pixels of the input image.

5. The real-time intelligent inference method for three-dimensional geographic environment based on multimodal large model according to claim 4, characterized in that: The specific process of mapping the semantic vectors in the semantic feature map to the corresponding 3D Gaussian distribution at the projection position, based on the projection transformation matrix corresponding to the camera pose, is as follows: For each 3D Gaussian distribution, determine whether the Gaussian distribution is within the camera's view frustum based on the calculated camera extrinsic pose and intrinsic matrix, filter out all effective viewing angles, and project the spatial mean coordinates onto the 2D pixel plane of each effective viewing angle according to the corresponding projection matrix to obtain the pixel index coordinates. Perform depth buffer testing to remove occluded projection points. Using the reciprocal of the projection distance as the weight, the semantic feature vectors sampled at the corresponding pixel index of all effective viewpoints are weighted and averaged. The weighted average semantic vectors are then added to the spherical harmonic coefficients and opacity properties of the three-dimensional Gaussian distribution.

6. The real-time intelligent inference method for three-dimensional geographic environment based on multimodal large model according to claim 1, characterized in that: The specific process of constructing a K-nearest neighbor topological graph of a 3D Gaussian distribution set based on spatial mean coordinates, using each 3D Gaussian distribution as a graph node, and generating a diffusion coefficient matrix by analyzing the texture changes of the surveillance video stream is as follows: Calculate the Euclidean distance between the spatial mean coordinates of any two points within a set of three-dimensional Gaussian distributions, and select the K nearest neighbors for each Gaussian distribution to establish undirected edges and generate a spatial adjacency matrix; The Gaussian distribution spatial mean coordinates at both ends of each edge are projected onto the image plane of the current frame of the surveillance video using the camera extrinsic pose and intrinsic parameter matrix. The local image gradient magnitude of the projection point is extracted and the gradient difference between the two points is calculated. The gradient difference is converted into edge weights using a Gaussian kernel function. The larger the weight, the stronger the physical conductivity. The complete diffusion coefficient matrix is ​​obtained by traversing all edges.

7. The real-time intelligent inference method for three-dimensional geographic environment based on a multimodal large model according to claim 6, characterized in that: Using the motion optical flow feature vector as the initial excitation of the graph nodes, message passing computation is performed in the topological graph to iteratively update the hidden states of each graph node. The specific process of calculating the spatial coordinate offset and spherical harmonic coefficient change of each three-dimensional Gaussian distribution within a future preset time step is as follows: The motion optical flow feature vector is mapped to the initial hidden state through a multilayer perceptron. In each prediction time step, the K-neighbor hidden state is weighted and aggregated according to the diffusion coefficient matrix to obtain the neighborhood message. The neighboring message is concatenated with the hidden state of the current node at the previous time step and then input into the gated loop unit for state update. The updated hidden state is then fed into two independent fully connected regression heads, which output the three-dimensional spatial displacement vector and the increment of each order of spherical harmonic coefficients for that time step, respectively, as the spatial coordinate offset and the change of spherical harmonic coefficients for that time step.

8. The real-time intelligent inference method for three-dimensional geographic environment based on multimodal large model according to claim 1, characterized in that: The calculated spatial coordinate offsets and spherical harmonic coefficient changes are applied frame-by-frame to a 3D Gaussian distribution set, and the specific process for dynamically updating the covariance matrix and opacity is as follows: The spatial coordinate offset is directly accumulated to the spatial mean coordinates of the corresponding three-dimensional Gaussian distribution, and the change in spherical harmonic coefficients is accumulated to the original spherical harmonic coefficients. For each Gaussian distribution, calculate the relative displacement change between it and its K nearest neighbors before and after the update, construct a local Jacobian matrix, apply an affine transformation of the Jacobian matrix to the original covariance matrix to achieve shape stretching and rotation, and simultaneously calculate the rate of change of the number of Gaussian distributions in the local K nearest neighbor sphere. In regions where the number decreases, proportionally reduce the opacity, and in regions where the number increases, proportionally increase the opacity.

9. The real-time intelligent inference method for three-dimensional geographic environment based on multimodal large model according to claim 8, characterized in that: The specific process of projecting the updated 3D Gaussian distribution set onto the 2D viewport plane using a differentiable Gaussian rasterization rendering pipeline to synthesize a continuous inference image sequence is as follows: The updated 3D Gaussian distribution set is transformed to the current viewport camera coordinate system. After frustum clipping, the 2D covariance matrix of each Gaussian distribution on the imaging plane is calculated. All 2D Gaussians are cardinally sorted by depth. A list of contributing Gaussians is established for each pixel. The color and opacity contributions are accumulated sequentially by the alpha mixing formula in depth order to obtain a single-frame rendered image. The above rendering process is continuously executed according to the time step to generate a continuous dynamic inference video sequence.

Images