Three-dimensional scene fusion monitoring method and device, equipment and storage medium

Abstract

The invention discloses a three-dimensional scene fusion monitoring method and device, equipment and a storage medium. The method comprises the following steps: acquiring a real-time frame image, and matching feature points of the real-time frame image with feature points in a preset three-dimensional real scene to obtain an initial camera pose; generating a plurality of adjacent view angle images based on the initial camera pose, and performing similarity matching on the adjacent view angle images and the real-time frame image to obtain an accurate camera pose; determining a geometric intersection area according to the accurate camera pose, and mapping the real-time frame image to the geometric intersection area to obtain a live-action three-dimensional projection; splicing and fusing the live-action three-dimensional monitoring projection of each unmanned aerial vehicle to obtain live-action three-dimensional monitoring; according to the method, the pose of the unmanned aerial vehicle is determined according to the feature matching result of the real-time frame image and the high-precision preset three-dimensional real scene instead of a sensor of the unmanned aerial vehicle, and the pose of the adjacent view angle image with the highest similarity is used as the pose of the unmanned aerial vehicle for projection, so that high-precision superposition of positions is ensured, and a better three-dimensional real scene monitoring effect is obtained.

Description

Technical Field

[0001] This invention relates to the field of computer application technology, and in particular to a three-dimensional scene fusion monitoring method, apparatus, device, and storage medium. Background Technology

[0002] In recent years, with the continuous development and application of AIoT (Artificial Intelligence of Things) technologies, the global security surveillance camera market is evolving towards large-scale, comprehensive, high-resolution, and high-rate development. This trend is mainly driven by the growing demand for digital video surveillance in fields such as smart cities and factory operations. Traditional video surveillance systems face difficulties in spatial perception and scene understanding. The key to overcoming these difficulties lies in intuitively representing spatial relationships, thereby reducing the burden of understanding.

[0003] Traditional surveillance video typically uses fixed monitoring walls for image display and output. This method results in scattered and isolated displays, making it impossible to effectively combine independent and unorganized surveillance videos into a coherent whole and to grasp the scene information of each surveillance video from a macro perspective. Currently, there is a method that integrates drone surveillance video with spatial 3D scenes. This method has high requirements for the parameters returned by the drone, and there are still problems such as inaccurate drone-returned parameter information leading to poor real-scene 3D monitoring effects. Summary of the Invention

[0004] In view of this, it is necessary to provide a three-dimensional scene fusion monitoring method, device, equipment and storage medium to solve the technical problem of poor real-scene three-dimensional monitoring effect caused by inaccurate parameter information returned by UAVs.

[0005] To address the above problems, the present invention provides a method comprising: The drone's initial camera pose is obtained by acquiring real-time frame images from the drone's real-time video stream and matching the feature points in the real-time frame images with the feature points in the preset 3D real scene. Based on the initial camera pose, multiple neighboring view images are generated, and the neighboring view images and the real-time frame images are matched for similarity to obtain the accurate camera pose. Based on the precise camera pose, the geometric intersection area between the camera's view frustum and the preset 3D real scene is determined. Based on texture mapping, the real-time frame image is mapped to the geometric intersection area to obtain a 3D projection of the real scene. The real-scene 3D monitoring projections of various drones are stitched together and merged to obtain real-scene 3D monitoring.

[0006] In one possible implementation, acquiring real-time frame images from the UAV's real-time video stream and matching feature points in the real-time frame images with feature points in a preset 3D real-world scene to obtain an initial camera pose includes: Frames are extracted from the real-time video stream information of the UAV to obtain real-time frame images, and the reference UAV pose at the time of acquiring the real-time frame images is obtained from the UAV real-time video stream information. Based on the reference UAV pose, reference feature points in the preset 3D real scene are determined, and feature point matching is performed between the reference feature points and the real-time frame image to obtain matching points; The initial camera pose of the UAV is obtained by solving the pose of the UAV based on the camera intrinsic parameters and multiple matching points.

[0007] In one possible implementation, generating multiple neighboring view images based on the initial camera pose, and performing similarity matching between the neighboring view images and the real-time frame images to obtain the accurate camera pose includes: Based on the initial camera pose, the initial viewpoint image in the preset 3D real scene is extracted; Based on the initial viewpoint image and the preset offset, generate multiple neighboring viewpoint images and the neighboring camera poses corresponding to each neighboring viewpoint image; Calculate the matching degree between each neighboring view image and the real-time frame image, and take the neighboring camera pose corresponding to the neighboring view image with the highest matching degree as the accurate camera pose.

[0008] In one possible implementation, determining the geometric intersection region between the camera's view frustum and the preset 3D real scene based on the precise camera pose, and mapping the real-time frame image to the geometric intersection region based on texture mapping to obtain a 3D projection of the real scene, includes: Based on the precise camera pose and the UAV camera intrinsic parameters, the UAV camera frustum is obtained; The geometric intersection region is obtained by performing spatial intersection calculation between the view frustum of the UAV camera and the preset 3D real scene; Based on texture mapping, the real-time frame image is mapped to the geometric intersection region to obtain a real-scene 3D projection.

[0009] In one possible implementation, the step of mapping the real-time frame image to the geometrically intersecting region based on texture mapping to obtain a real-scene 3D projection includes: The projection transformation matrix is ​​obtained based on the precise camera pose and camera intrinsic parameters; The texture coordinate mapping relationship between the real-time frame image pixels and the geometrically intersecting region is determined based on the projection transformation matrix; Based on the texture coordinate mapping relationship, the real-time frame image is mapped as a dynamic texture to the geometric intersection region to obtain a real-world 3D projection.

[0010] In one possible implementation, the step of stitching and fusing the real-scene 3D monitoring projections of various drones to obtain real-scene 3D monitoring includes: Extract the overlapping areas of the real-world 3D monitoring projections of each drone; The pixel weight distribution of the projected overlapping region is obtained based on the inverse distance weighting and the spatial geometric features of the projected overlapping region; The projected overlapping region is weighted and fused according to the pixel weight distribution to obtain the initial three-dimensional monitoring; The monitoring range of the initial 3D monitoring is optimized to obtain real-scene 3D monitoring.

[0011] In one possible implementation, optimizing the monitoring range of the initial 3D monitoring to obtain real-scene 3D monitoring includes: Calculate the overlap range between the projection surface of the initial three-dimensional monitoring and the preset observation range, and crop the projection surface of the initial three-dimensional monitoring based on the overlap range to obtain the cropped three-dimensional monitoring. Based on the bounding box of the user's current viewport, the display status of each drone is determined according to the bounding box of the viewport and the spatial coordinates of the clipped 3D monitoring. Based on the display status, the target display content in the real-scene 3D monitoring is used as the real-scene 3D monitoring content.

[0012] Furthermore, to achieve the above objectives, the present invention also proposes a three-dimensional scene fusion monitoring device, the three-dimensional scene fusion monitoring device comprising: The pose calculation module is used to acquire real-time frame images from the real-time video stream information of the UAV, and match the feature points in the real-time frame images with the feature points in the preset three-dimensional real scene to obtain the initial camera pose of the UAV. The pose calculation module is also used to generate multiple neighboring view images based on the initial camera pose, and to perform similarity matching between the neighboring view images and the real-time frame images to obtain the accurate camera pose. The real-scene monitoring module is used to determine the geometric intersection area between the camera's view frustum and the preset three-dimensional real scene based on the precise camera pose, and to map the real-time frame image to the geometric intersection area based on texture mapping to obtain a real-scene three-dimensional projection. The real-scene monitoring module is also used to stitch and merge the real-scene 3D monitoring projections of each drone to obtain real-scene 3D monitoring.

[0013] Furthermore, to achieve the above objectives, the present invention also proposes an electronic device, the electronic device comprising: a memory, a processor, a display, and a three-dimensional scene fusion monitoring program stored in the memory and executable on the processor, the three-dimensional scene fusion monitoring program being configured to implement the steps of the three-dimensional scene fusion monitoring method described above.

[0014] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a three-dimensional scene fusion monitoring program, which, when executed by a processor, implements the steps of the three-dimensional scene fusion monitoring method described above.

[0015] The beneficial effects of the above implementation method are as follows: By performing feature matching between video frames and known high-precision preset 3D real scene, the drone pose is determined based on the matching results rather than the drone's low-cost sensors, fundamentally avoiding the introduction of back-transmission parameter errors. Based on the similarity comparison of neighboring view images from multiple perspectives in the 3D real scene, the pose can be adjusted with extreme precision. The pose of the neighboring view image with the highest similarity is projected as the drone pose, ensuring that the position of the object in the video coincides with the position in the 3D model with high precision. This results in a real 3D projection of the scene with accurate spatial position, without distortion or drift, achieving a true fusion of the virtual model and real-time video, resulting in better 3D real scene monitoring effects and completely solving the problem of poor real scene 3D monitoring effects caused by inaccurate drone back-transmission parameters. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the first embodiment of the three-dimensional scene fusion monitoring method of the present invention; Figure 2 This is a schematic diagram illustrating the complete steps of real-time fusion of video stream and 3D scene in the first embodiment of the 3D scene fusion monitoring method of the present invention; Figure 3 This is a flowchart illustrating the second embodiment of the three-dimensional scene fusion monitoring method of the present invention; Figure 4 This is a schematic diagram of overlapping region fusion in the second embodiment of the three-dimensional scene fusion monitoring method of the present invention. Figure 5 This is a schematic diagram illustrating video loading load optimization in the second embodiment of the three-dimensional scene fusion monitoring method of the present invention; Figure 6This is a schematic diagram of the fused 3D real-scene monitoring after the second embodiment of the 3D scene fusion monitoring method of the present invention; Figure 7 This is a structural block diagram of the first embodiment of the three-dimensional scene fusion monitoring device of the present invention; Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0019] In the description of the embodiments of this application, unless otherwise stated, "a plurality of" means two or more.

[0020] In the embodiments of the present invention, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, apparatus, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0021] The naming or numbering of steps in the embodiments of the present invention does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] The executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or 3D scene fusion monitoring device capable of performing the above functions. The following description uses a 3D scene fusion monitoring device as an example to illustrate this embodiment and the subsequent embodiments.

[0024] This invention provides a three-dimensional scene fusion monitoring method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the three-dimensional scene fusion monitoring method of the present invention.

[0025] In this embodiment, the three-dimensional scene fusion monitoring method includes steps S10 to S40: Step S10: Obtain real-time frame images from the real-time video stream information of the UAV, and match the feature points in the real-time frame images with the feature points in the preset 3D real scene to obtain the initial camera pose of the UAV.

[0026] Understandably, real-scene 3D monitoring can be achieved by mapping each frame of a real-time video stream captured by a drone into a 3D space. Multiple drones can simultaneously capture real-time video stream information, and the regions captured by each drone's real-time video stream can be different. Therefore, the 3D regions projected by each video stream are also different. The 3D projections of each drone are then aggregated and merged to complete real-scene 3D monitoring.

[0027] It should be understood that a real-time frame image can be an image that is a 3D real-scene projection of a real-time video stream. After the real-time video stream information collected by the drone is transmitted back, the real-time video stream information is projected, and the frame image being projected is the real-time frame image.

[0028] It should be understood that feature points can be relatively obvious or distinct reference objects in real-time frame images. Preset 3D reality can be understood as a projection to be displayed to users. To give a simple example, if a basketball game is being held in Europe, a video stream of a European football stadium is captured by a drone. A physical stadium model is built in China, and the video stream monitored by the drone is projected onto the stadium model. This allows users to see the European football game in the stadium model, making it more realistic.

[0029] It should be noted that the drone pose can be the exterior orientation element of the camera, including three angular elements (Heading, Pitch, Roll) describing the spatial attitude during imaging and three line elements (X, Y, Z) representing the spatial coordinates of the camera center. The initial pose can be understood as the pose obtained based on the matching point, and further optimization is still needed.

[0030] It should be understood that the collection and acquisition of drone data consists of aerial images of the target area collected by drones in the field, real-time video stream information, and some usable aerial information parameters, which constitute the initial data for drone surveillance video real-scene fusion.

[0031] It should be noted that the high-resolution aerial imagery acquired by the drone primarily consists of visible light RGB imagery, which can be expanded to multispectral, hyperspectral, and thermal infrared imagery as needed. The drone conducts aerial photography along a pre-set flight path, typically using vertical downward-looking or oblique photography methods. Providing static, high-precision, and high-resolution surface information of the target area forms the basis for generating orthophoto maps (DOM), digital surface models (DSM), and realistic 3D models.

[0032] Furthermore, drones can collect real-time video stream information. The video stream captured in real time by the drone's onboard camera (usually visible light, but also thermal imaging or zoom camera) can be transmitted back to the ground control station (GCS) or cloud platform in real time via the drone's data link (such as Wi-Fi, 4G / 5G, dedicated image transmission radio, or even satellite relay). This can provide dynamic, continuous, and low-latency situational awareness of the target area, which is key to real-time monitoring, event response, and target tracking.

[0033] Furthermore, high-precision flight status and positioning data are generated by sensors such as the GNSS receiver, inertial measurement unit, and barometer of the UAV flight control system. This data is transmitted in real time or near real time via a data link, providing the precise spatial position and attitude of the UAV at the moment each frame of image / video stream is captured. This data serves as the core basis for georegistration, image correction, video geolocation, and spatiotemporal synchronization of multi-source data.

[0034] It should be emphasized that the spatial pose fed back by the drone is inaccurate, and 3D spatial fusion based on the spatial pose fed back by the drone will lead to misalignment.

[0035] In one feasible implementation, step S10 may include steps A11 to A13: Step A11: Extract frames from the real-time video stream in the UAV real-time video stream information to obtain real-time frame images, and obtain the reference UAV pose when acquiring the real-time frame images from the UAV real-time video stream information.

[0036] Understandably, frame extraction is the process of extracting an image from a real-time video stream within a single time unit, such as 1 millisecond or 5 milliseconds. The shorter the time unit, the more images need to be processed, resulting in higher computational demands and higher resolution.

[0037] It should be understood that the reference drone pose is obtained by the drone through its own pose sensing device when capturing real-time video streams. However, due to video capture and the drone's flight status, this pose is not very accurate.

[0038] Step A12: Determine reference feature points in the preset 3D real scene based on the reference UAV pose, and perform feature point matching between the reference feature points and the real-time frame image to obtain matching points.

[0039] It should be noted that the reference drone pose is not very accurate, but it can minimize the range of reference feature points in the preset 3D real scene, significantly reducing the number of reference feature points that need to be matched and effectively improving computational efficiency.

[0040] It should be noted that matching the reference feature points with the feature points in the real-time frame image can be achieved by calculating the similarity between the descriptor of the reference feature points and the descriptors of each feature point in the real-time frame image, and then taking two feature points with a similarity exceeding a certain threshold as a set of matching points.

[0041] Step A13: Solve the pose of the UAV based on the UAV's camera intrinsic parameters and multiple matching points to obtain the initial camera pose of the UAV.

[0042] It should be noted that solving the drone's pose based on the drone's camera intrinsic parameters and multiple matching points to obtain the drone's initial camera pose is a PnP problem.

[0043] Specifically, the RANSAC algorithm is used to randomly sample a small number of point pairs (e.g., 4 pairs) to estimate multiple possible poses. Then, the pose with the most supporting points is selected by voting, which is the initial camera pose. This can effectively eliminate mismatches and ensure the stability of the solution.

[0044] Furthermore, the camera's rotation and translation matrices can be directly calculated from the matching points to obtain the initial camera pose.

[0045] In this embodiment, by matching feature points in the real-time frame images captured by the UAV with feature points in the 3D real scene, the initial camera pose of the UAV camera in 3D space is quickly estimated, which facilitates further accurate pose estimation based on the initial camera pose, thereby providing more accurate parameters for subsequent 3D scene fusion.

[0046] The above are merely feasible implementations of step S10 provided in this embodiment. This embodiment does not specifically limit the specific implementation of step S10.

[0047] Step S20: Generate multiple neighboring view images based on the initial camera pose, and perform similarity matching between the neighboring view images and the real-time frame images to obtain the accurate camera pose.

[0048] Understandably, the neighboring view image is a series of images generated based on the initial camera pose in a preset 3D real scene, after the initial camera pose has been fine-tuned.

[0049] In practice, the images in the geographic scene of the coarse matching data are used as the initial viewpoints. A set of neighboring multi-viewpoints is constructed, and the similarity between images is used as the evaluation index. The initial viewpoint image is compared with all images in the set, and the data corresponding to the image with the highest matching degree in the set is selected as the true pose of the UAV camera, that is, the camera pose is finely calculated, and the accurate camera pose is obtained.

[0050] It should be noted that similarity matching refers to the degree of overlap between neighboring viewpoint images and real-time frame images. The higher the degree of overlap, the closer the camera pose corresponding to the neighboring viewpoint image is to the pose when the UAV acquires the real-time frame image.

[0051] It should be noted that the drone pose corresponding to the nearest similarity view image is used as the accurate camera pose.

[0052] In one feasible implementation, step S20 may include steps A21 to A23: Step A21: Extract the initial viewpoint image from the preset 3D real scene based on the initial camera pose.

[0053] Understandably, the initial camera pose is located in the preset 3D real scene, and the image of the preset 3D real scene is captured based on the initial camera pose as the initial viewpoint image.

[0054] Step A22: Generate multiple neighboring view images and the poses of neighboring cameras corresponding to each neighboring view image based on the initial view image and the preset offset.

[0055] It should be noted that the computer slightly changes the six parameters of the initial pose (X, Y, Z, Heading, Pitch, Roll). The preset offset can be moving the camera position 0.1 meters to the left or tilting it down 0.5 degrees. Then, a virtual image is generated from each finely adjusted new viewpoint as a neighboring viewpoint image. Each neighboring viewpoint image after adjustment corresponds to the pose of the neighboring camera.

[0056] Step A23: Calculate the matching degree between each neighboring view image and the real-time frame image, and take the neighboring camera pose corresponding to the neighboring view image with the highest matching degree as the accurate camera pose.

[0057] It should be noted that the matching degree between each neighboring view image and the real-time frame image can be calculated by comparing the relative position patterns of feature points such as SIFT and ORB in the two images through feature point distribution, comparing the similarity of the contours and structures of the two images through image gradient / edge, or further by comparing mutual information to measure the statistical correlation of the information content of the two images.

[0058] Understandably, the similarity score is obtained by normalizing the above scores. The higher the similarity score, the closer the camera poses of the two images are.

[0059] In this embodiment, an optimal and best-matching pose is found by fine-tuning the pose near the initially estimated pose and comparing the difference between the rendered result and the real image.

[0060] The above are merely feasible implementations of step S20 provided in this embodiment. This embodiment does not specifically limit the specific implementation of step S20.

[0061] Step S30: Determine the geometric intersection region between the camera's view frustum and the preset 3D real scene based on the precise camera pose, and map the real-time frame image to the geometric intersection region based on texture mapping to obtain a 3D projection of the real scene.

[0062] Understandably, the camera's view frustum can be determined based on the camera's intrinsic parameters and pose, and is a truncated pyramidal volume region in three-dimensional space.

[0063] It should be noted that the geometric intersection region can be understood as the part of the 3D model that can be directly seen from the current camera's perspective, obtained by performing Boolean intersection calculations between the camera's view frustum and the preset 3D real-world model in 3D space.

[0064] It should be noted that the complete steps of real-scene 3D projection can be as follows: First, spatial 3D modeling of the target area image is performed to obtain a stereo 3D model with rich texture. Second, the pose calculation of the UAV camera is performed, mainly including two steps: First, coarse matching of the camera pose is performed. By extracting corresponding points, filtering feature points, and calculating matching points between UAV aerial images and real scene photos, a coarse value of the UAV camera pose under this condition is obtained. Second, a set of neighboring viewpoints under this coarse value is constructed. Using image similarity as a metric, the image frames that are closer to the real viewpoint in the surrounding area of ​​this viewpoint are detected and used as the result for subsequent calculations. Finally, spatial projection of the scene is performed. The texture coverage area of ​​this viewpoint is calculated using the corrected result, and texture mapping is completed using the UAV real-time video stream as texture information, realizing the real-time fusion of UAV monitoring video and geographic scene model.

[0065] It is worth noting that by analyzing the spatial relationship between drone video footage and geographical scenes, solving the drone camera projection parameters, and studying the overall fusion scheme of drone surveillance video and geographical scene model, and combining texture mapping technology in WebGL, the fusion accuracy of drone real-time surveillance video and scene model can be improved.

[0066] It should be noted that the scene coordinates under the view frustum model are calculated based on the camera pose obtained from the fine matching results. The real-time monitoring video is then projected onto the geographic scene model as a texture using texture mapping technology, which is the real-scene fusion of drone monitoring video.

[0067] In one feasible implementation, step S30 may include steps A31 to A33: Step A31: Obtain the UAV camera frustum based on the precise camera pose and UAV camera intrinsic parameters.

[0068] It should be noted that accurate camera pose includes rotation matrix and translation vector, which can be used to transform between the world coordinate system and the camera coordinate system.

[0069] Understandably, the view frustum of a drone camera can be understood as a pyramid-shaped hexahedron with its top cut off in parallel. The eight vertices of the hexahedron can be determined first based on the drone camera's intrinsic parameters.

[0070] Furthermore, the coordinates of each vertex are transformed into the world coordinate system through precise camera pose. The space enclosed by the eight vertices in the 3D real scene can be understood as the view frustum of the drone camera.

[0071] Step A32: Perform spatial intersection calculation between the view frustum of the UAV camera and the preset 3D real scene to obtain the geometric intersection area.

[0072] Understandably, the calculation of the geometric intersection region can be understood as the surface area in the 3D scene that is being observed by the camera, or simply as the area of ​​the preset 3D real scene that can actually be seen under the current camera pose.

[0073] It should be noted that the geometric intersection area between the known drone camera cone and the preset 3D real scene can be directly obtained through spatial intersection calculation.

[0074] Specifically, spatial intersection can be achieved by performing a preliminary fast bounding box test on each triangular facet in the 3D reality model, followed by a precise triangular facet test. Finally, for the triangular facets that intersect with the clipping plane (i.e., part inside and part outside), the intersecting polygons can be calculated, and the part located within the view frustum can be cut out to generate new, smaller triangular facets, until the most accurate result can be obtained as the geometric intersection region.

[0075] Step A33: Based on texture mapping, the real-time frame image is mapped to the geometric intersection region to obtain a real-scene 3D projection.

[0076] It should be noted that the real-time frame image is mapped to the geometric intersection region based on texture mapping to obtain a real-scene 3D projection. This can be achieved by obtaining a projection transformation matrix based on the precise camera pose and camera intrinsic parameters; determining the texture coordinate mapping relationship between the pixels of the real-time frame image and the geometric intersection region based on the projection transformation matrix; and mapping the real-time frame image as a dynamic texture to the geometric intersection region based on the texture coordinate mapping relationship to obtain a real-scene 3D projection.

[0077] It is understandable that the projection transformation matrix can be obtained by first obtaining the view matrix based on the accurate camera pose, which is the transformation matrix between the world coordinate system and the camera coordinate system; then obtaining the projection matrix based on the camera intrinsic parameters; and finally obtaining the projection transformation matrix based on the projection matrix and the transformation matrix.

[0078] The position of each vertex on the geometrically intersecting region on the real-time frame image, and the transformation relationship between the two can be understood as a texture coordinate mapping relationship.

[0079] Specifically, based on the projection transformation matrix, the vertices of the view cone are transformed to NDC space, and the triangular facets are decomposed into individual fragments. For each rasterized fragment, i.e. a point in the 3D real-world geometric intersection region, the texture coordinates of that point on the original real-time frame image are back-interpolated. Using the texture coordinates of the three vertices of the triangular facet where that point is located, interpolation is performed based on its centroid coordinates to obtain this precise texture coordinate. This texture coordinate directly corresponds to a pixel in the real-time frame image, indicating from which position in the real-time frame image each point in the 3D real-world geometric intersection region should take color.

[0080] In this embodiment, by combining precise camera pose and texture mapping, real-world dynamic video and virtual 3D static scene are geometrically seamlessly integrated to achieve better 3D real-scene monitoring.

[0081] The above are merely feasible implementations of step S30 provided in this embodiment. This embodiment does not specifically limit the specific implementation of step S30.

[0082] Step S40: Stitch and merge the real-scene 3D monitoring projections of each drone to obtain real-scene 3D monitoring.

[0083] It should be noted that all the drones' real-scene 3D projections share the same preset 3D real-scene as a base, accurately, non-overlappingly, and seamlessly covering multiple dynamic projections from different perspectives and positions onto the preset 3D real-scene to achieve real-scene 3D monitoring.

[0084] Understandably, when each drone captures real-time video streams, differences in color, brightness, and contrast arise due to variations in cameras and lighting conditions. These differences need to be eliminated during the stitching process to blend the projections of each drone, resulting in a smoother and more natural overall 3D real-scene monitoring.

[0085] It is worth noting that the fusion of UAV surveillance video and 3D scene is formed by solving the spatial pose of UAV cameras to improve the fusion accuracy of UAV surveillance video and geographic scene model, and to provide a new idea for enhancing the spatial expression of surveillance video. According to the actual application needs, an online UAV surveillance video real scene fusion system is developed to integrate and display UAV surveillance video and geographic scene model in real time, and to build a macroscopic, all-round, integrated monitoring scene.

[0086] The fusion effect of UAV surveillance video and geographic scene model depends on the matching accuracy of their image information, specifically including the calculation of the UAV camera's spatial pose parameters and the fusion representation of UAV video and the real-world 3D model. Therefore, one of the key technical problems this paper aims to solve is the specific scheme for fusion of UAV surveillance video and geographic scene model, realizing the conversion of UAV surveillance video from a two-dimensional plane to three-dimensional space, and strengthening the connection between UAV surveillance and the geographic spatial scene.

[0087] In practical implementation, the complete steps for real-time fusion of video streams and 3D scenes can be found in [reference needed]. Figure 2 , Figure 2The process is divided into three parts. The first part is the raw data, which includes UAV video streams, spatial information parameters, and UAV images. The second part is the fusion of UAV surveillance video and geographic scene models, which includes target aerial imagery, spatial reality maps, and regional 3D models (preset 3D reality). The calculation methods include coarse camera pose calculation, fine pose correction, and spatial projection. Coarse pose calculation involves extracting corresponding points from the aerial image of the area, then filtering feature points and calculating pose to obtain a coarse matching result (i.e., the initial camera pose). Fine pose correction is based on the initial viewpoint corresponding to the coarse matching result, calculating a set of neighboring viewpoints using spatial data, performing similarity matching on the neighboring viewpoint set, and using the camera pose corresponding to the neighboring viewpoint with the highest similarity as the fine matching result. Spatial projection determines the texture coordinates based on the camera pose of the fine matching result, integrates the texture information, and performs texture mapping to achieve video-real-scene fusion. The third part is the real-world video fusion processing of the swarm drones, which specifically includes video latency processing, video texture processing, and video efficiency processing. Video latency processing involves converting the drone video streams through a streaming media server to reduce swarm latency. Video texture processing can involve obtaining the spatial range, marking spatial redundancy based on the spatial range, fusing video streams without spatial redundancy markings, and optimizing the display effect of the fused result. Video efficiency processing includes loading the swarm video and then cropping each projection surface to optimize the load and thus optimize swarm efficiency.

[0088] This embodiment provides a 3D scene fusion monitoring method. By performing feature matching between video frames and known high-precision preset 3D real scenes, the drone pose is determined based on the matching results rather than the drone's low-cost sensors. This fundamentally avoids the introduction of errors in the transmitted parameters. Based on the similarity comparison of neighboring view images from multiple perspectives in the 3D real scene, the pose can be adjusted with extreme precision. The pose of the neighboring view image with the highest similarity is projected as the drone pose, ensuring that the position of the object in the video coincides with the position in the 3D model with high precision. This results in a real 3D projection of the scene with accurate spatial position, without distortion or drift, achieving a true fusion of the virtual model and real-time video, resulting in better 3D real-scene monitoring effects and completely solving the problem of poor real-scene 3D monitoring effects caused by inaccurate drone transmitted parameters.

[0089] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Step S40, the three-dimensional scene fusion monitoring method further includes steps S41~S44: Step S41: Extract the overlapping area of ​​the projection between the real-world 3D monitoring projections of each drone.

[0090] Understandably, the real-time frame images of each drone can determine the geometric intersection area in the preset 3D real scene, that is, determine the corresponding coordinates, and the area enclosed by the coordinates that coincide between the geometric intersection areas of each drone is used as the projection overlap area.

[0091] It should be noted that when displaying real-time video fusion of a cluster of drones on the front-end 3D platform, differences in spatial position and angle lead to uneven texture brightness between the individual drone videos. This phenomenon causes two main problems: first, poor video quality results in insufficient dynamic contrast; second, different lighting conditions cause brightness differences between adjacent video images, creating noticeable stitching gaps. To address these issues and improve the visual effect of drone video fusion, it is necessary to enhance video quality and perform fusion processing on overlapping areas to eliminate stitching lines.

[0092] Specifically, the intersection volume of two view frustums in three-dimensional space can be obtained by performing three-dimensional Boolean operations on the view frustums between each UAV. This volume represents the part of the three-dimensional space that the two UAVs can see together. Then, the position in the geometric intersection region can be determined based on the intersection volume, which can be used as the projection overlap region.

[0093] Step S42: Obtain the pixel weight distribution of the projection overlap region based on the inverse distance weighting and the spatial geometric features of the projection overlap region.

[0094] Specifically, the diagonal is determined based on the spatial geometric features of the projected overlapping area. The diagonal is used as a reference line, and the vertical distance from each pixel in the overlapping area to the reference line or the shortest distance from each pixel to the reference line is calculated. Based on this distance, two weight matrices are calculated for each pixel in the overlapping part (the overlapping part belongs to two projections, so the weights of the two projections are calculated separately to ensure that the sum of the weights 1 and 2 of the same pixel is 1). These two weight matrices are the pixel weight distribution.

[0095] Furthermore, the color of the merged pixels is the pixel value of the pixel in the first projection of the overlapping projection area multiplied by weight 1, plus the pixel value of the pixel in the second projection of the overlapping projection area multiplied by weight 2.

[0096] It should be noted that this pixel weight distribution can be understood as the proportion of color contribution from the images of drone A and drone B in the final fused image. For example, a weight of 0.8 at a certain location means that after fusion, 80% of the color at that point comes from the main reference image and 20% comes from the other image.

[0097] Step S43: Perform weighted fusion processing on the projected overlapping area according to the pixel weight distribution to obtain the initial three-dimensional monitoring.

[0098] In practice, a reciprocal distance-weighted fusion approach is used to fuse adjacent images, creating a gradient effect in overlapping areas to weaken the stitching lines. A fusion diagram can be found in the provided image. Figure 4 , Figure 4 In the diagram, ① and ② represent two drone projections with overlapping regions. Weighted fusion is performed along the diagonal BD of the texture overlap region ABCD. The weight of the video image in region ① gradually decreases to the right of the diagonal BD and gradually increases to the left of BD; while the weight of region ② is the opposite, gradually decreasing to the left of BD and gradually increasing to the right of BD. The texture changes with distance.

[0099] It should be understood that the pixel weight distribution can be understood as the weight of each pixel in the projected overlapping area of ​​the current drone. Since it is the projected overlapping area of ​​two drones, the other drone also has a weight for each pixel in the overlapping area. The same pixel has two weights, and the two weights are added together to 1.

[0100] It is important to emphasize that the pixel value of the pixel in the merged overlapping region is the pixel value of the current drone multiplied by the weight of that pixel value in the pixel weight distribution, plus the pixel value of the same pixel from the other drone multiplied by the weight in another pixel weight distribution. By fusing each pixel value in the projected overlapping region, the merged overlapping region is obtained, realizing the fusion of the projections of the two drones and obtaining the initial 3D monitoring.

[0101] Step S44: Optimize the monitoring range of the initial 3D monitoring to obtain real-scene 3D monitoring.

[0102] It is worth noting that the optimization of the monitoring range for the initial 3D monitoring is based on the UAV monitoring video and geographic scene model, and studies issues such as service scheduling and data transmission in the fusion of swarm UAVs and 3D scenes. To address issues such as unsmooth display and low frame rate of the fused image during swarm UAV operations, a high-performance multi-scene synchronous rendering and fusion scheme is designed and proposed to achieve collaborative operation and management of swarm UAV video-real-scene fusion.

[0103] In one feasible implementation, step S44 may include steps A441 to A443: Step A441: Calculate the overlap range between the projection surface of the initial three-dimensional monitoring and the preset observation range, and crop the projection surface of the initial three-dimensional monitoring based on the overlap range to obtain the cropped three-dimensional monitoring.

[0104] Understandably, the preset observation range can be understood as the area that the user is concerned about, and this area can be changed continuously according to the user's needs.

[0105] It should be understood that the preset observation range can be determined on the preset three-dimensional projection. The overlapping range can be obtained by performing spatial intersection calculation based on the three-dimensional spatial range of the preset observation range and the projection surface of the initial three-dimensional monitoring.

[0106] It should be emphasized that the area where the overlap range is located in the initial 3D monitoring is retained. The area outside the overlap range can be filled with the background color to reduce the amount of data displayed. On the originally huge projection surface, only the area within the "overlap range" is visible, and the rest is "cropped" and becomes transparent or invisible.

[0107] It should be noted that one of the key technologies in drone video-scene fusion is the simulation of video projection from a camera onto a 3D geographic scene, that is, mapping video images onto a 3D model to form a projection surface. The view frustum is the field of view of the simulated camera, and the part that intersects with the 3D model is the location of the projection surface.

[0108] Furthermore, when the tilt angle of the drone video projection is too small, that is, the angle between the view frustum and the scene model is too small, the field of view becomes too large, and the distortion is obvious in the remote areas of the view, resulting in a fragmented projection surface and a complex information in the scene.

[0109] It should be emphasized that, in order to improve the effect of multi-channel drone video real-scene fusion, it is necessary to crop some of the video projection surfaces to remove parts that are not within the observation range.

[0110] One common clipping method is to use polygons for clipping. That is, a polygonal region is given as the clipping plane, the projection plane and the clipping plane are superimposed and analyzed, the projection plane outside the clipping plane is clipped, and the projection plane inside the clipping plane is retained.

[0111] Step A442: Based on the bounding box of the user's current viewport, determine the display status of each drone based on the bounding box of the viewport and the spatial coordinates of the clipped 3D monitoring.

[0112] Understandably, the bounding box of the user's current viewport is the area in the preset 3D reality scene that the user is focusing on. The area to be displayed can be determined based on the bounding box of the viewport. Furthermore, the spatial coordinates of the area to be displayed can be used to determine which drone image coordinates overlap with. Drones with overlapping coordinates are marked as displayed, and drones without overlapping coordinates are marked as not displayed.

[0113] It should be understood that the display status can include both displaying and not displaying.

[0114] It should be emphasized that in the practical application of drone surveillance video real-scene fusion, the monitoring scenarios are complex and diverse, such as mountains, villages, schools, etc. In such scenarios, multiple drones are usually distributed in different locations. At one time, the system is under great pressure and the effect is not good when all the connected drone surveillance videos are merged and drawn into the same scene.

[0115] Furthermore, the scene display effect of video fusion from swarm drones needs to be optimized. A simple and efficient method for optimizing swarm drone video fusion is to use view culling. That is, in a 3D scene, the video fusion of the user's current area is loaded based on the scene observation needs, while other video fusions outside the observed scene will not be displayed.

[0116] It should be understood that only video streaming and projection are performed on the video fusion scene within the observation area, which can effectively reduce PC load and improve the overall smoothness of the fusion scene.

[0117] Step A443: Based on the display status, the target display content in the real-scene 3D monitoring is used as the real-scene 3D monitoring content.

[0118] It should be noted that points within the viewport are selected using a spatial point location detection method. First, the spatial geographic coordinates of four points at the screen edge are calculated, transforming the two-dimensional pixel coordinates on the screen into three-dimensional spatial geographic coordinates [P1, P2, P3, P4], typically converted to latitude, longitude, and altitude. This process can be implemented using the Cartesian class on a Cesium-based platform, defining and transforming the coordinate system. After obtaining the spatial coordinate values ​​of the four screen points, the drone's spatial position (x1, y1, h1) is passed in. If the drone's spatial position is within the range [P1, P2, P3, P4], the corresponding video fusion scene is displayed; otherwise, it is not displayed. A diagram illustrating video loading load optimization can be found here. Figure 5 Drones numbered 1, 2, and 4 are located within the corresponding spatial positions [P1, P2, P3, P4] around the screen, so their video fusion status is "displayed." Drones numbered 3, 5, and 6 are not within the screen's range, so their video fusion status is "not displayed." The fused 3D reality monitoring diagram can be found in the reference diagram. Figure 6 .

[0119] In this embodiment, by eliminating the field of view and cropping the projection surface, only the drone video stream within the user's current field of view is processed and rendered. At the same time, irrelevant and distorted edge areas are discarded, which greatly reduces the load on the computer's GPU and CPU. This solves the huge computational pressure brought about by the simultaneous fusion of multiple high-definition video streams and ensures that even if multiple drones are connected, the entire 3D monitoring scene can maintain a high-definition and smooth running frame rate, providing users with a smooth interactive experience.

[0120] The above are merely feasible implementations of step S44 provided in this embodiment. This embodiment does not specifically limit the specific implementation of step S44.

[0121] It is worth noting that optimizing the fusion scenario of clustered drone surveillance videos improves the processing efficiency of the monitoring system, enhances the comprehensive expressive effect of drone surveillance videos, and effectively broadens the information breadth of drone surveillance videos in geospatial space and the application depth in various fields.

[0122] It is important to emphasize that the real-time fusion and collaborative processing of surveillance video from swarm drones addresses the issues of high latency and display optimization during collaborative drone operations. For video latency, local servers are set up and data formats are converted to reduce the latency of the swarm drone video streams. Regarding the collaborative issues when swarm drones are operating simultaneously, space trimming is used to eliminate data conflicts when multiple videos are overlaid, and simultaneous loading of multiple video streams is optimized to reduce client load, thereby improving the overall display effect and enhancing the expression of the fused scene.

[0123] This embodiment provides a three-dimensional scene fusion monitoring method. By optimizing the fusion of scenes using swarm drones, it can effectively improve system processing efficiency and enhance user experience. It proposes optimization methods to address issues such as stuttering and high memory consumption when fusion of real-scene monitoring videos from swarm drones. This effectively solves problems such as high computational load and multiple service thread scheduling when fusion of multiple videos, achieving effective collaboration in the fusion of real-scene monitoring videos from drones and further enhancing the expressive effect of real-scene fusion of drone videos.

[0124] To better implement the 3D scene fusion monitoring method in this embodiment of the invention, based on the 3D scene fusion monitoring method, correspondingly, as follows: Figure 7 As shown, this embodiment of the invention also provides a three-dimensional scene fusion monitoring device, the three-dimensional scene fusion monitoring device 700 including: The pose calculation module 701 is used to extract real-time frame images from real-time video stream information acquired by the UAV, and match the feature points in the real-time frame images with the feature points in the preset three-dimensional real scene to obtain the initial camera pose. The pose calculation module 701 is further configured to generate multiple neighboring view images based on the initial camera pose, and perform similarity matching between the neighboring view images and the real-time frame images to obtain the accurate camera pose. The real-scene monitoring module 702 is used to determine the geometric intersection area between the camera's view frustum and the preset three-dimensional real scene based on the precise camera pose, and to map the real-time frame image to the geometric intersection area based on texture mapping to obtain a real-scene three-dimensional projection. The real-scene monitoring module 702 is also used to stitch and fuse the real-scene 3D monitoring projections of each drone to obtain real-scene 3D monitoring.

[0125] The three-dimensional scene fusion monitoring device 700 provided in the above embodiments can realize the technical solutions described in the above three-dimensional scene fusion monitoring method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above three-dimensional scene fusion monitoring method embodiments, and will not be repeated here.

[0126] like Figure 8 As shown, the present invention also provides an electronic device 800. The electronic device 800 includes a processor 801, a memory 802, and a display 803. Figure 8 Only some components of the electronic device 800 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0127] In some embodiments, memory 802 may be an internal storage unit of electronic device 800, such as a hard disk or memory of electronic device 800. In other embodiments, memory 802 may also be an external storage device of electronic device 800, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 800.

[0128] Furthermore, the memory 802 may include both internal storage units of the electronic device 800 and external storage devices. The memory 802 is used to store application software and various types of data installed on the electronic device 800.

[0129] In some embodiments, processor 801 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 802 or process data, such as the three-dimensional scene fusion monitoring method of the present invention.

[0130] In some embodiments, display 803 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 803 is used to display information from electronic device 800 and to display a visual user interface. Components 1001-1003 of electronic device 800 communicate with each other via a system bus.

[0131] In some embodiments of the present invention, when the processor 801 executes the 3D scene fusion monitoring program in the memory 802, the following steps can be implemented: Real-time frame images from the drone's real-time video stream are acquired. Feature points in the real-time frame images are matched with feature points in a preset 3D real-world scene to obtain the drone's initial camera pose. Multiple neighboring view images are generated based on the initial camera pose. The neighboring view images are matched with the real-time frame images to obtain the precise camera pose. The geometric intersection region between the camera's view frustum and the preset 3D real-world scene is determined based on the precise camera pose. The real-time frame images are mapped to the geometric intersection region based on texture mapping to obtain a real-world 3D projection. The real-world 3D monitoring projections of each drone are stitched and fused to obtain a real-world 3D monitoring image.

[0132] It should be understood that when the processor 801 executes the 3D scene fusion monitoring program in the memory 802, in addition to the functions mentioned above, it can also perform other functions, as detailed in the description of the corresponding method embodiments above.

[0133] Furthermore, the embodiments of the present invention do not specifically limit the type of electronic device 800 mentioned. Electronic device 800 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, electronic device 800 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0134] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional scene fusion monitoring method provided by the methods described above, the method comprising: Real-time frame images from the drone's real-time video stream are acquired. Feature points in the real-time frame images are matched with feature points in a preset 3D real-world scene to obtain the drone's initial camera pose. Multiple neighboring view images are generated based on the initial camera pose. The neighboring view images are matched with the real-time frame images to obtain the precise camera pose. The geometric intersection region between the camera's view frustum and the preset 3D real-world scene is determined based on the precise camera pose. The real-time frame images are mapped to the geometric intersection region based on texture mapping to obtain a real-world 3D projection. The real-world 3D monitoring projections of each drone are stitched and fused to obtain a real-world 3D monitoring image.

[0135] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0136] The three-dimensional scene fusion monitoring method provided by the present invention has been described in detail above. Specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the idea of ​​the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A three-dimensional scene fusion monitoring method, characterized in that, include: The drone's initial camera pose is obtained by acquiring real-time frame images from the drone's real-time video stream and matching the feature points in the real-time frame images with the feature points in the preset 3D real scene. Based on the initial camera pose, multiple neighboring view images are generated, and the neighboring view images and the real-time frame images are matched for similarity to obtain the accurate camera pose. Based on the precise camera pose, the geometric intersection area between the camera's view frustum and the preset 3D real scene is determined. Based on texture mapping, the real-time frame image is mapped to the geometric intersection area to obtain a 3D projection of the real scene. The real-scene 3D monitoring projections of various drones are stitched together and merged to obtain real-scene 3D monitoring.

2. The three-dimensional scene fusion monitoring method as described in claim 1, characterized in that, The step of acquiring real-time frame images from the drone's real-time video stream and matching feature points in the real-time frame images with feature points in a preset 3D real-world scene to obtain the initial camera pose includes: Frames are extracted from the real-time video stream information of the UAV to obtain real-time frame images, and the reference UAV pose at the time of acquiring the real-time frame images is obtained from the UAV real-time video stream information. Based on the reference UAV pose, reference feature points in the preset 3D real scene are determined, and feature point matching is performed between the reference feature points and the real-time frame image to obtain matching points; The initial camera pose of the UAV is obtained by solving the pose of the UAV based on the camera intrinsic parameters and multiple matching points.

3. The three-dimensional scene fusion monitoring method as described in claim 1, characterized in that, The process of generating multiple neighboring view images based on the initial camera pose, and performing similarity matching between the neighboring view images and the real-time frame images to obtain the accurate camera pose includes: Based on the initial camera pose, the initial viewpoint image in the preset 3D real scene is extracted; Based on the initial viewpoint image and the preset offset, generate multiple neighboring viewpoint images and the neighboring camera poses corresponding to each neighboring viewpoint image; Calculate the matching degree between each neighboring view image and the real-time frame image, and take the neighboring camera pose corresponding to the neighboring view image with the highest matching degree as the accurate camera pose.

4. The three-dimensional scene fusion monitoring method as described in claim 1, characterized in that, The step of determining the geometric intersection region between the camera's view frustum and the preset 3D real scene based on the precise camera pose, and mapping the real-time frame image onto the geometric intersection region based on texture mapping to obtain a 3D projection of the real scene includes: Based on the precise camera pose and the UAV camera intrinsic parameters, the UAV camera frustum is obtained; The geometric intersection region is obtained by performing spatial intersection calculation between the view frustum of the UAV camera and the preset 3D real scene; Based on texture mapping, the real-time frame image is mapped to the geometric intersection region to obtain a real-scene 3D projection.

5. The three-dimensional scene fusion monitoring method as described in claim 4, characterized in that, The process of mapping the real-time frame image onto the geometrically intersecting region based on texture mapping to obtain a real-world 3D projection includes: The projection transformation matrix is ​​obtained based on the precise camera pose and camera intrinsic parameters; The texture coordinate mapping relationship between the real-time frame image pixels and the geometrically intersecting region is determined based on the projection transformation matrix; Based on the texture coordinate mapping relationship, the real-time frame image is mapped as a dynamic texture to the geometric intersection region to obtain a real-world 3D projection.

6. The three-dimensional scene fusion monitoring method as described in claim 1, characterized in that, The process of stitching and fusing the real-scene 3D monitoring projections of various drones to obtain real-scene 3D monitoring includes: Extract the overlapping areas of the real-world 3D monitoring projections of each drone; The pixel weight distribution of the projected overlapping region is obtained based on the inverse distance weighting and the spatial geometric features of the projected overlapping region; The projected overlapping region is weighted and fused according to the pixel weight distribution to obtain the initial three-dimensional monitoring; The monitoring range of the initial 3D monitoring is optimized to obtain real-scene 3D monitoring.

7. The three-dimensional scene fusion monitoring method as described in claim 6, characterized in that, The process of optimizing the monitoring range of the initial 3D monitoring to obtain real-scene 3D monitoring includes: Calculate the overlap range between the projection surface of the initial three-dimensional monitoring and the preset observation range, and crop the projection surface of the initial three-dimensional monitoring based on the overlap range to obtain the cropped three-dimensional monitoring. Based on the bounding box of the user's current viewport, the display status of each drone is determined according to the bounding box of the viewport and the spatial coordinates of the clipped 3D monitoring. Based on the display status, the target display content in the real-scene 3D monitoring is used as the real-scene 3D monitoring content.

8. A three-dimensional scene fusion monitoring device, characterized in that, The 3D scene fusion monitoring device includes: The pose calculation module is used to acquire real-time frame images from the real-time video stream information of the UAV, and match the feature points in the real-time frame images with the feature points in the preset three-dimensional real scene to obtain the initial camera pose of the UAV. The pose calculation module is also used to generate multiple neighboring view images based on the initial camera pose, and to perform similarity matching between the neighboring view images and the real-time frame images to obtain the accurate camera pose. The real-scene monitoring module is used to determine the geometric intersection area between the camera's view frustum and the preset three-dimensional real scene based on the precise camera pose, and to map the real-time frame image to the geometric intersection area based on texture mapping to obtain a real-scene three-dimensional projection. The real-scene monitoring module is also used to stitch and merge the real-scene 3D monitoring projections of each drone to obtain real-scene 3D monitoring.

9. An electronic device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps of the three-dimensional scene fusion monitoring method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the three-dimensional scene fusion monitoring method as described in any one of claims 1 to 7.

Images