Intelligent view analysis method and system based on unmanned aerial vehicle

Abstract

This application relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to an intelligent view analysis method and system based on UAVs. The method includes: acquiring multi-source data, which includes at least video data and motion perception data; processing the video data to obtain video frame data; determining the spatial pose information of the UAV; performing target detection based on the video frame data to obtain the pixel coordinate information of the target, and converting the pixel coordinate information into real spatial coordinates by combining the spatial pose information and a 3D semantic map; performing target tracking and behavior analysis based on the video frame data and real spatial coordinates, and generating structured event data by combining scene semantic information; performing semantic verification based on the video frame data and structured event data to obtain event analysis results; and, if the event analysis results are preset events, controlling the UAV to perform multi-view data acquisition based on the spatial coordinates in the structured event data to generate a 3D scene model of the target area.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to an intelligent view analysis method and system based on UAVs. Background Technology

[0002] In applications such as smart inspection, security monitoring, and energy facility operation and maintenance, drones have been widely used for video acquisition and status monitoring of target areas. Current technologies typically involve acquiring video data via drones and performing target detection, tracking, and basic behavior analysis based on video frames to achieve target identification and event assessment.

[0003] However, most existing UAV view analysis methods are based solely on two-dimensional image information, lacking effective utilization of the UAV's spatial pose and the three-dimensional structure of the scene. This results in target localization typically remaining at the pixel level, failing to obtain the target's true spatial coordinates, and thus limiting the application value of the analysis results in real-world scenarios. Furthermore, while some technologies incorporate pose estimation or 3D modeling, these are usually independent of target detection and behavior analysis processes, lacking a unified data fusion mechanism and making it difficult to achieve accurate mapping from image information to spatial information.

[0004] Furthermore, existing technologies in event analysis often rely on single detection results or simple rule-based judgments, lacking a structured expression that combines spatial location and scene semantic information. This results in incomplete event semantics and low reliability of analysis results. In addition, after identifying a target or abnormal event, the relevant technologies typically cannot adaptively control the drone to collect data from multiple perspectives based on the analysis results, making it difficult to obtain supplementary information for key areas in a timely manner, and also unable to generate high-precision 3D scene models.

[0005] Clearly, existing UAV view analysis solutions suffer from problems such as missing spatial information, insufficient fusion of multi-source data, and weak event response capabilities, making it difficult to meet the needs for accurate target positioning, behavior understanding, and dynamic data acquisition in complex scenarios. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] In view of the above-mentioned shortcomings and deficiencies of the prior art, this application provides an intelligent view analysis method and system based on UAVs, which solves the technical problem of inaccurate target spatial positioning caused by the lack of accurate mapping capability from image pixel coordinates to real space coordinates in the prior art.

[0008] (II) Technical Solution

[0009] To achieve the above objectives, the main technical solutions adopted in this application include:

[0010] In a first aspect, embodiments of this application provide an intelligent view analysis method based on a drone, comprising: acquiring multi-source data collected by the drone during flight, the multi-source data including at least video data and motion perception data; processing the video data to obtain video frame data; determining the spatial pose information of the drone based on the video frame data and the motion perception data, combined with a pre-constructed three-dimensional semantic map; performing target detection based on the video frame data to obtain the pixel coordinate information of the target, and converting the pixel coordinate information into the target's real spatial coordinates by combining the spatial pose information and the pre-constructed three-dimensional semantic map; performing target tracking and behavior analysis based on the video frame data and the target's real spatial coordinates, and generating structured event data containing spatial coordinates by combining scene semantic information provided by the three-dimensional semantic map; performing semantic verification based on the video frame data and the structured event data to obtain event analysis results; and, if the event analysis results are preset events, controlling the drone to perform multi-view data acquisition based on the spatial coordinates in the structured event data, and generating a three-dimensional scene model of the target area.

[0011] Optionally, in some embodiments of this application, the process of constructing a three-dimensional semantic map includes: performing time synchronization and alignment processing on multi-source data and performing image distortion correction; on this basis, generating a sparse point cloud through feature point extraction and matching, and generating a dense point cloud and a three-dimensional mesh model through multi-view stereo matching, while combining a semantic segmentation algorithm to generate a three-dimensional semantic map and a feature point database.

[0012] Optionally, in some embodiments of this application, the process of generating a three-dimensional semantic map and feature point database includes: extracting image feature points from the corrected video frame data, calculating corresponding feature descriptors while extracting image feature points, and matching image feature points between adjacent video frames; constructing a camera motion trajectory based on the matched image feature points and the motion perception data, and jointly optimizing the camera motion trajectory and the corresponding three-dimensional point coordinates through an optimization method to obtain optimized camera pose and three-dimensional spatial points, thereby generating a sparse point cloud; using the sparse point cloud as a basis, performing pixel-level depth estimation on the video frame data using a multi-view stereo matching algorithm to generate a dense point cloud, and constructing a three-dimensional mesh model based on the dense point cloud; performing semantic classification on the three-dimensional mesh model based on a semantic segmentation algorithm to generate a three-dimensional semantic map; establishing association relationships based on the image feature points, the feature descriptors and their corresponding three-dimensional spatial points to construct a feature point database; wherein, the image feature points in the feature point database are key scene feature points that can be stably detected under different time and different view conditions, and the key scene feature points include fixed landmark feature points such as building structure edge points, road sign points and tower structure points.

[0013] Optionally, in some embodiments of this application, the spatial pose information of the UAV is determined based on the video frame data and the motion sensing data, combined with a pre-constructed three-dimensional semantic map. Specifically, this includes: extracting image feature points from the video frame data and matching them with image feature points in the feature point database to obtain a correspondence between the image feature points in the current video frame data and the image feature points in the feature point database; obtaining the three-dimensional spatial points corresponding to the image feature points in the current video frame data based on the correspondence, and calculating the pose information of the current video frame data in three-dimensional space according to the correspondence between the image feature points and the three-dimensional spatial points; and constraining and correcting the pose information based on the motion sensing data to obtain the spatial pose information of the UAV.

[0014] Optionally, in some embodiments of this application, target detection is performed based on the video frame data to obtain the pixel coordinate information of the target, and the pixel coordinate information is converted into the real spatial coordinates of the target in combination with the spatial pose information. Specifically, this includes: detecting the target based on the video frame data to obtain the pixel coordinate information of the target in the video frame data; determining the camera extrinsic parameters corresponding to the video frame data based on the spatial pose information, and establishing a mapping relationship between the pixel coordinate system and the camera coordinate system in combination with the camera intrinsic parameters; converting the target pixel coordinates into a spatial direction vector in the camera coordinate system according to the pixel coordinate information and the mapping relationship; and performing spatial constraint calculation on the spatial direction vector in combination with the three-dimensional spatial points in the three-dimensional semantic map corresponding to the spatial direction vector to determine the three-dimensional spatial coordinates of the target in real space.

[0015] Optionally, in some embodiments of this application, target tracking and behavior analysis are performed based on the video frame data and the target's real spatial coordinates, and structured event data containing spatial coordinates is generated by combining scene semantic information. Specifically, this includes: cross-frame association of targets based on the video frame data, and consistency matching of cross-frame targets based on the target's real spatial coordinates to establish a correspondence between targets in consecutive video frames, thereby achieving multi-target tracking of targets and obtaining the target's motion trajectory; calculating the target's motion parameters based on the target's motion trajectory and corresponding time series information, and determining the target's behavioral state based on the motion parameters; mapping the target's real spatial coordinates to... The scene semantic information of the target's location is obtained from the corresponding scene region in the 3D semantic map; wherein, the scene region is a region with preset semantic attributes divided in the 3D semantic map based on semantic classification results; the scene semantic information includes the semantic category of the scene region and the corresponding attribute information; based on the target's behavioral state, real spatial coordinates and corresponding scene semantic information, association determination is performed to determine the event type corresponding to the target, and structured event data containing target identifier, spatial coordinates, behavioral state and event type is generated; the association determination is based on preset rules, the preset rules include the correspondence between target behavioral state, spatial location and scene semantic information.

[0016] Optionally, in some embodiments of this application, semantic verification is performed based on the video frame data and the structured event data to obtain event analysis results. Specifically, this includes: determining an initial judgment result and corresponding confidence information for the event based on the event type and its corresponding judgment confidence level in the structured event data; when the confidence information is lower than a preset threshold or the event type belongs to a preset high-risk event, inputting the structured event data and corresponding video frame data into a preset view analysis model for semantic verification processing; performing semantic understanding on the video frame data based on the view analysis model, and combining the target behavior state, real spatial coordinates, and scene semantic information in the structured event data to perform consistency verification and semantic reasoning on the initial judgment result to obtain a verification judgment result; and weighting and fusing the initial judgment result and the verification judgment result based on their confidence levels, or selecting the result based on preset rules to determine the event analysis result; wherein the event analysis result includes at least the event type, target identifier, real spatial coordinates, and event confidence level.

[0017] Optionally, in some embodiments of this application, when the event analysis result is a preset event, the UAV is controlled to perform multi-view data acquisition based on the spatial coordinates in the structured event data, and a three-dimensional scene model of the target area is generated. Specifically, this includes: determining the target area based on the spatial coordinates in the structured event data, calculating the center position of the target area based on the spatial coordinates, and planning the multi-view acquisition path of the UAV based on the center position; controlling the UAV to fly according to the multi-view acquisition path, performing surround or multi-angle data acquisition on the target area, and obtaining image data containing different perspectives and corresponding motion perception data; performing time synchronization and alignment processing on the acquired image data and the motion perception data, and filtering the image data based on image clarity, perspective coverage, or data integrity to obtain effective data for three-dimensional reconstruction; based on the effective data, performing pixel-level depth estimation on the image data through a multi-view stereo matching method to generate dense point cloud data of the target area, and performing noise reduction processing and dynamic target removal processing on the dense point cloud data; constructing a three-dimensional scene model of the target area based on the processed dense point cloud data, and performing texture mapping processing on the three-dimensional scene model to obtain a three-dimensional scene model with texture information.

[0018] Optionally, in some embodiments of this application, during the multi-view data acquisition process, the image data acquisition adopts an adaptive sampling strategy based on changes in image feature points, specifically including:

[0019] During the multi-view data acquisition process of the UAV, image feature points are extracted from continuously acquired video frames; based on the changes in the number and spatial distribution of image feature points in adjacent video frames, the similarity between adjacent video frames is calculated; according to the comparison result of the similarity with a preset threshold, the sampling interval of the video frames is adjusted according to a preset sampling control rule; wherein, when the similarity is higher than the preset threshold, the sampling interval of the video frames is extended according to the preset sampling control rule; when the similarity is lower than the preset threshold, the sampling interval of the video frames is shortened according to the preset sampling control rule.

[0020] Optionally, in some embodiments of this application, the method further includes: during the UAV's flight mission, acquiring currently collected video frame data and corresponding motion perception data, extracting image feature points based on the video frame data, matching the image feature points with image feature points in the feature point database to obtain feature point matching results; calculating the feature point matching rate based on the feature point matching results, and determining the corresponding scene area as a candidate change area when the feature point matching rate is lower than a preset matching threshold; generating dense point cloud data of the current area based on the video frame data corresponding to the candidate change area, and performing spatial consistency comparison between the dense point cloud data and the corresponding three-dimensional spatial points in the three-dimensional semantic map, while combining the semantic classification information in the three-dimensional semantic map... Semantic consistency detection is performed to confirm the changes in the candidate changed regions and determine the change type based on semantic differences. For the confirmed changed regions, the UAV is controlled to perform incremental data acquisition on the changed regions, and feature point extraction, matching, and 3D reconstruction processing are re-executed based on the newly added video frame data and corresponding motion perception data to generate updated sparse point clouds, dense point clouds, and corresponding 3D mesh models. The updated 3D mesh model and the original 3D semantic map are geometrically and semantically fused in the corresponding scene area. The geometric fusion is based on the 3D spatial point alignment results, and the semantic fusion is based on the semantic classification results. The fusion results are verified based on the consistency of feature points, spatial points, and semantics before and after fusion to generate an updated 3D semantic map.

[0021] Optionally, in some embodiments of this application, the method further includes: mapping the real spatial coordinates to the corresponding spatial location in the three-dimensional scene model based on the target identifier, real spatial coordinates, and event type information in the structured event data, and annotating and displaying the corresponding target and event at the spatial location to achieve spatial visualization of the event; when the event type in the event analysis result belongs to a preset alarm event, generating alarm information based on the event type, event confidence level, and real spatial coordinates in the event analysis result, and outputting the alarm information; generating event records based on the structured event data and the event analysis result, and storing and managing the event records, wherein the event records at least include target identifier, event type, real spatial coordinates, time information, and event confidence level, and the time information is obtained based on the timestamp of the video frame data.

[0022] Optionally, in some embodiments of this application, the video frame data includes: video stream data for real-time transmission obtained by decoding and format conversion of the video data; a first type of video frame data extracted from the video data based on a preset fixed time interval or fixed frame number interval; and a second type of video frame data extracted from the video data based on an adaptive frame extraction strategy for video content changes. The second type of video frame data is obtained by: calculating the content similarity between adjacent video frames based on image feature points and corresponding feature descriptors in the video frames; when the content similarity is greater than or equal to a first preset threshold, extracting frames from the video data according to a first preset frame extraction interval; when the content similarity is less than or equal to a second preset threshold, extracting frames from the video data according to a second preset frame extraction interval; wherein the first preset frame extraction interval is greater than the second preset frame extraction interval.

[0023] Secondly, embodiments of this application provide an intelligent view analysis system based on a drone, comprising: at least one drone; an intelligent view analysis platform, the intelligent view analysis platform comprising: a data acquisition module for acquiring multi-source data collected by the drone during flight, the multi-source data including at least video data and motion perception data; a view sampling module for processing the video data to obtain video frame data; a spatial perception algorithm module for determining the spatial pose information of the drone based on the video frame data and the motion perception data, combined with a pre-constructed three-dimensional semantic map; and a video analysis algorithm module for performing target detection based on the video frame data. The system acquires the pixel coordinates of the target and converts them into the target's real spatial coordinates using the spatial pose information. It also performs target tracking and behavior analysis based on the video frame data and the target's real spatial coordinates, and generates structured event data containing spatial coordinates by combining scene semantic information. A large view model module performs semantic verification based on the video frame data and the structured event data to obtain event analysis results. A 3D reconstruction module, when the event analysis result is a preset event, controls the UAV to perform multi-view data acquisition based on the spatial coordinates in the structured event data and generates a 3D scene model of the target area.

[0024] Thirdly, embodiments of this application provide an intelligent view analysis platform, including: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the UAV-based intelligent view analysis method described in the above embodiments.

[0025] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the UAV-based intelligent view analysis method described in the above embodiments.

[0026] (III) Beneficial Effects

[0027] This application provides an intelligent view analysis method based on unmanned aerial vehicles (UAVs). By acquiring multi-source data, including video data and motion perception data, collected by the UAV during flight, and processing the video data to obtain video frame data, combined with a pre-constructed 3D semantic map, the method accurately determines the UAV's spatial pose information, thus providing a reliable spatial foundation for subsequent analysis. Furthermore, by performing target detection based on the video frame data and obtaining the target's pixel coordinate information, and combining this with the spatial pose information and the 3D semantic map, the pixel coordinate information is converted into the target's true spatial coordinates. This effectively solves the problem in existing technologies where target localization only reaches the pixel level and cannot obtain true spatial coordinates, thereby improving the accuracy of target spatial localization.

[0028] Meanwhile, this application performs target tracking and behavior analysis based on video frame data and the target's real spatial coordinates. Combined with scene semantic information provided by a 3D semantic map, it generates structured event data containing spatial coordinates. This ensures that event representation not only includes target behavior information but also integrates spatial location and scene semantic information, thereby improving the completeness and accuracy of event analysis. Furthermore, by performing semantic verification based on video frame data and structured event data, the event analysis results are obtained, effectively improving the reliability of event recognition and reducing the occurrence of misjudgments.

[0029] Furthermore, when the event analysis result is a preset event, the UAV is controlled to perform multi-view data acquisition based on the spatial coordinates in the structured event data, and a three-dimensional scene model of the target area is generated. This realizes a closed-loop processing process from event detection to data acquisition and then to three-dimensional modeling, which improves the UAV's adaptive perception capability and data acquisition capability of key areas, thereby improving the overall intelligence level of the system. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating an embodiment of an intelligent view analysis method based on a drone according to this application.

[0031] Figure 2 This is a flowchart illustrating the process of generating a three-dimensional semantic map and feature point database according to an embodiment of this application;

[0032] Figure 3This is a flowchart illustrating the process of determining the three-dimensional spatial coordinates of a target in real space according to an embodiment of this application;

[0033] Figure 4 This is a schematic diagram of the structure of an unmanned aerial vehicle (UAV)-based intelligent view analysis system according to an embodiment of this application;

[0034] Figure 5 This is a schematic diagram of the structure of an intelligent view analysis platform according to an embodiment of this application;

[0035] Figure 6 This is a schematic diagram of the structure of an intelligent view analysis platform according to another embodiment of this application. Detailed Implementation

[0036] To better explain and facilitate understanding of this application, the following detailed description of the application is provided in conjunction with the accompanying drawings and specific embodiments.

[0037] In related technologies, methods for UAV view analysis can be mainly categorized into three types:

[0038] The first category is target detection and behavior analysis solutions based on single video data. This approach processes video data collected by drones to obtain video frame data, and then performs target detection, tracking, and behavior recognition to analyze targets and determine events. However, this approach mainly relies on two-dimensional image information for analysis, and target localization usually only stays at the pixel coordinate level, lacking the ability to represent the real spatial location. Under conditions such as changes in viewpoint, scale, or target occlusion, it is prone to inaccurate localization or unreliable analysis results.

[0039] The second category is analysis schemes based on pose estimation or 3D reconstruction. These schemes estimate the spatial pose information of the UAV or construct 3D scene models by introducing motion-sensing data or employing techniques such as visual SLAM to enhance environmental understanding. While this approach improves spatial perception to some extent, it typically separates pose estimation from target detection and behavior analysis processes, lacking a unified data fusion mechanism. This makes it difficult to effectively combine video frame data, motion-sensing data, and 3D semantic maps, thus failing to accurately convert the target's pixel coordinates into real spatial coordinates and limiting the accuracy of the analysis.

[0040] The third category is data acquisition and event analysis solutions triggered by rules or simple results. These solutions typically detect a target or anomaly, make event judgments based on preset rules, and control a drone to collect supplementary data according to a fixed strategy. However, this approach lacks utilization of the target's true spatial coordinates and scene semantic information, making it difficult to generate structured event data containing spatial coordinates. Furthermore, it lacks a semantic verification mechanism during event analysis, resulting in low accuracy and reliability of the analysis results. In addition, its multi-view data acquisition is usually not targeted, unable to adaptively adjust the acquisition strategy based on event results, and struggles to further generate high-precision 3D scene models.

[0041] To address this, this application provides an intelligent view analysis method based on unmanned aerial vehicles (UAVs). This method acquires multi-source data, including video data and motion perception data, collected by the UAV during flight. The video data is processed to obtain video frame data. Combined with a pre-constructed 3D semantic map, the spatial pose information of the UAV is determined. Based on this, the pixel coordinates of the target are converted into real spatial coordinates. Target tracking and behavior analysis are performed based on the video frame data and the target's real spatial coordinates, generating structured event data containing spatial coordinates. Simultaneously, event analysis results are obtained through semantic verification. If the event analysis result corresponds to a preset event, the UAV is controlled to perform multi-view data acquisition and generate a 3D scene model of the target area based on the spatial coordinates in the structured event data. This achieves integrated processing of multi-source data fusion, spatial positioning, and event analysis, improving target positioning accuracy, event analysis reliability, and data acquisition adaptability even in complex scenarios.

[0042] To better understand the above technical solutions, exemplary embodiments of this application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application can be understood more clearly and thoroughly, and that the scope of this application can be fully conveyed to those skilled in the art.

[0043] Figure 1 This is a flowchart illustrating a drone-based intelligent view analysis method according to an embodiment of this application. Figure 1 As shown, this drone-based intelligent view analysis method includes:

[0044] S1. Acquire multi-source data collected by the UAV during flight, wherein the multi-source data includes at least video data and motion perception data;

[0045] The operational sensing data described in this application embodiment includes: inertial measurement unit (IMU) data, GPS / BeiDou positioning data, flight attitude data, etc.

[0046] S2. Process the video data to obtain video frame data;

[0047] Optionally, in some embodiments of this application, the video frame data includes: video stream data for real-time transmission obtained by decoding and format conversion of the video data, so as to ensure that the UAV can continuously output continuous video information during flight, providing basic data support for subsequent target detection and spatial analysis; a first type of video frame data extracted from the video data based on a pre-set fixed time interval or fixed frame interval, for example, when the UAV is patrolling and inspecting a large area of ​​the park, it can be sampled at a fixed number of frames per second, thereby reducing the data processing pressure while ensuring basic coverage and being able to stably obtain uniformly distributed video frame data; and a second type of video frame data extracted from the video data based on an adaptive frame extraction strategy for changes in video content.

[0048] The second type of video frame data is video frame data obtained through the following methods:

[0049] Based on image feature points and corresponding feature descriptors in video frames, the content similarity between adjacent video frames is calculated to characterize the degree of scene change between consecutive frames.

[0050] When the content similarity is greater than or equal to the first preset threshold, it indicates that the current scene changes little, such as a drone flying at a constant speed above an open road. In this case, the video data is extracted at the first preset frame extraction interval to reduce redundant data.

[0051] When the content similarity is less than or equal to the second preset threshold, it indicates that the current scene has changed significantly. For example, when a drone passes through a densely populated factory area or an intersection, the video data is sampled at the second preset frame extraction interval to increase the sampling density of key changing areas, thereby preserving more detailed scene information. The first preset frame extraction interval is greater than the second preset frame extraction interval. For example, during an inspection of an industrial park, when a drone flies over an open road or grassland area, the feature point changes between adjacent video frames are small, and the content similarity is high. In this case, the frame extraction interval is automatically increased to reduce data redundancy. However, when the drone flies over a densely populated factory area or an area with concentrated equipment, the structural changes between adjacent frames are significant, and the content similarity decreases. In this case, the frame extraction interval is automatically decreased to increase the video frame sampling density of key areas.

[0052] S3. Based on the video frame data and the motion perception data, combined with the pre-constructed three-dimensional semantic map, determine the spatial pose information of the UAV;

[0053] Optionally, in some embodiments of this application, the process of pre-constructing a 3D semantic map includes: performing time synchronization and alignment processing on multi-source data, and performing image distortion correction; more specifically, performing time synchronization and alignment processing on multi-source data ensures that video frame data and motion perception data are consistent in the time dimension, and performs time calibration based on the UTC time of the UAV flight control system, thereby ensuring that each video frame data can be accurately matched with the corresponding geographic information data, IMU data, and UAV pose data. Based on this, image distortion correction is performed on the video frame data according to the intrinsic parameters of the camera on the UAV, eliminating spatial deviations caused by lens distortion, and improving the geometric consistency and accuracy of subsequent 3D reconstruction from the source.

[0054] Based on this, sparse point clouds are generated by feature point extraction and matching, and dense point clouds and 3D mesh models are generated by multi-view stereo matching. At the same time, a 3D semantic map and feature point database are generated by combining semantic segmentation algorithms.

[0055] Among them, see Figure 2 The process of generating a 3D semantic map and feature point database includes:

[0056] Image feature points are extracted from the corrected video frame data, and corresponding feature descriptors are calculated simultaneously. Feature points are then matched between adjacent video frames. Specifically, feature descriptors enhance the distinguishability of feature points from different viewpoints, and matching feature points between adjacent video frames establishes cross-frame feature point correspondences. For example, when capturing data from urban road areas, feature points can be stably extracted at building edges, road marking intersections, etc., and consistent matching can be achieved across consecutive video frames, providing stable constraints for subsequent spatial reconstruction.

[0057] Based on the matched image feature points and the motion sensing data, a camera motion trajectory is constructed. An optimization method is then used to jointly optimize the camera motion trajectory and the corresponding 3D point coordinates to obtain an optimized camera pose and 3D spatial points, thereby generating a sparse point cloud that covers the flight area. This process effectively integrates UAV pose data and visual feature information, improving the stability of spatial reconstruction and avoiding drift errors caused by a single data source.

[0058] Based on the sparse point cloud, a multi-view stereo matching algorithm is used to perform pixel-level depth estimation on the video frame data to generate a dense point cloud. A three-dimensional mesh model is then constructed based on this dense point cloud, enabling a refined representation of the spatial structure of the target area. For example, when modeling roads and building areas in a park, the dense point cloud can reconstruct road boundaries, building outlines, and local structural details, thus forming a continuous three-dimensional mesh model.

[0059] The three-dimensional mesh model is semantically classified based on a semantic segmentation algorithm to generate a three-dimensional semantic map. Different areas such as road surfaces, illegal parking areas, pedestrian crossings, and building structures in the generated three-dimensional semantic map are assigned corresponding semantic labels, so that geometric information and semantic information can be expressed in a unified manner.

[0060] A feature point database is constructed by establishing associations between the image feature points, the feature descriptors and their corresponding three-dimensional spatial points.

[0061] The image feature points in the feature point database are key scene feature points that can be stably detected under different time and viewing angle conditions. These key scene feature points include fixed landmark feature points such as building structure edge points, road marking points, and tower structure points. For example, when repeatedly flying over the same area, even if the lighting or viewing angle changes, stable matching can still be achieved based on the intersection of road markings or the corner points of buildings, thereby supporting high-precision estimation of spatial pose and map reuse.

[0062] For example, in an application scenario of drone inspection modeling for an industrial park, the drone periodically flies over the park's roads, factories, and utility tunnels along a preset route, collecting video data and corresponding motion sensing data during flight, including geographic information data, IMU data, and drone pose data. After data acquisition, the video frame data and motion sensing data are first synchronized and aligned in time, ensuring that each video frame corresponds one-to-one with its spatial location and attitude information. For instance, when the drone flies over the park's main road, the video frame at that moment can be accurately matched with the corresponding GPS position and attitude angle information, and image distortion correction is performed on the video frame data based on camera intrinsic parameters, thereby eliminating geometric deviations caused by lens imaging and ensuring the accuracy of subsequent spatial reconstruction. Subsequently, image feature points are extracted from the corrected video frame data, and corresponding feature descriptors are calculated. For example, in the park's road area, key feature points such as road marking intersections, intersection corners, and factory edge structure points can be stably extracted, and the feature descriptors enhance the ability to distinguish feature points from different perspectives. Between adjacent video frames, cross-frame correspondences are established through feature point matching, enabling stable tracking of the same ground features during continuous drone flight. Based on this, camera motion trajectories are constructed by combining matched image feature points and motion sensing data. For example, when a drone flies from the park entrance along the main road to the factory area, its spatial trajectory can be inferred based on changes in feature points across consecutive frames combined with IMU data. Joint optimization is then used to correct the camera pose and 3D spatial points, generating a sparse point cloud covering the entire industrial park, allowing for the initial reconstruction of the park's basic spatial relationships, such as road structures and building outlines. Furthermore, based on the sparse point cloud, pixel-level depth estimation is performed on the video frame data using a multi-view stereo matching algorithm. For example, when observing factory facades from multiple angles, a denser spatial point cloud can be generated. A 3D mesh model is then constructed on this basis, allowing for a refined representation of details such as factory facades, road boundaries, and utility tunnel structures, thus forming a 3D spatial model with continuous geometric structures. After constructing the 3D mesh model, a semantic segmentation algorithm is used for semantic classification. For example, road areas are marked as road surfaces, parking areas as illegally parked areas, and factory entrances and exits as structural passage areas, thereby generating a 3D semantic map. This gives the park space not only geometric structural information but also understandable semantic attributes. Simultaneously, relationships are established between image feature points, feature descriptors, and their corresponding 3D spatial points to construct a feature point database. For example, factory corners, road marking intersections, and pole structure points are stored as stable feature points for long-term preservation, ensuring stable detection under different times, lighting conditions, and viewing angles.

[0063] For example, when a drone flies back to the same park during a subsequent inspection, even with changes in lighting or partial occlusion, it can still quickly match factory edge points or road intersections through a feature point database, thereby achieving high-precision spatial pose estimation and reusing and updating the original 3D semantic map. When new equipment is added or the road structure changes within the park, the changed areas can be identified by comparing the old and new point clouds, thus completing the incremental update of the 3D semantic map.

[0064] Through the above practical application process, a complete closed loop can be achieved from multi-source data acquisition, spatial reconstruction, semantic annotation to feature point database construction. This enables the 3D semantic map to not only have accurate spatial expression capabilities, but also stable cross-time reuse capabilities and dynamic update capabilities, thereby significantly improving the spatial cognition and intelligent analysis capabilities of UAVs in complex industrial scenarios.

[0065] S4. Target detection is performed based on the video frame data to obtain the pixel coordinate information of the target, and the pixel coordinate information is converted into the real spatial coordinates of the target by combining the spatial pose information and the pre-constructed three-dimensional semantic map.

[0066] S5. Based on the video frame data and the target's real spatial coordinates, perform target tracking and behavior analysis, and combine the scene semantic information provided by the three-dimensional semantic map to generate structured event data containing spatial coordinates;

[0067] S6. Perform semantic verification based on the video frame data and the structured event data to obtain event analysis results;

[0068] S7. If the event analysis result is a preset event, control the UAV to perform multi-view data acquisition based on the spatial coordinates in the structured event data, and generate a three-dimensional scene model of the target area.

[0069] This application's embodiments acquire multi-source data and process video frame data. By combining motion perception data and a pre-constructed 3D semantic map, spatial pose information is determined, achieving accurate conversion of target pixel coordinates to real spatial coordinates and improving target spatial positioning accuracy. Target tracking and behavior analysis are performed based on video frame data and real spatial coordinates, and structured event data is generated by combining scene semantic information. This ensures that event information simultaneously includes target identification, spatial coordinates, and semantic information, improving the completeness of event representation. Semantic verification enhances the reliability of event analysis results. Under preset event conditions, the UAV is controlled to perform multi-view data acquisition and generate a 3D scene model, achieving closed-loop processing from target detection and event analysis to 3D modeling, thus improving spatial perception and intelligent analysis capabilities.

[0070] Optionally, in this embodiment, step S3, based on the video frame data and the motion perception data, combined with a pre-constructed three-dimensional semantic map, determines the spatial pose information of the UAV, specifically including:

[0071] Image feature points are extracted from the video frame data and matched with image feature points in the feature point database to obtain the correspondence between the image feature points in the current video frame data and the image feature points in the feature point database. For example, in an industrial park scenario, when a drone flies to a road intersection area, image feature points such as road marking intersections, building edge points, or tower structure points can be extracted from the video frame data and matched with corresponding fixed landmark feature points pre-stored in the feature point database to establish a stable cross-view correspondence.

[0072] Based on the correspondence, the three-dimensional spatial points corresponding to the image feature points in the current video frame data are obtained, and the pose information of the current video frame data in three-dimensional space is calculated according to the correspondence between the image feature points and the three-dimensional spatial points. Specifically, based on the correspondence between image feature points and their corresponding three-dimensional spatial points, the spatial coordinates of each image feature point in the current video frame data in the three-dimensional semantic map are obtained, and the initial pose information of the current video frame data in three-dimensional space is calculated according to the correspondence between two-dimensional image feature points and three-dimensional spatial points, thereby achieving spatial pose estimation based on visual information. For example, when a drone flies over a factory area, by matching the edge feature points of the factory with the corresponding spatial points in the three-dimensional semantic map, the spatial position and attitude information of the drone relative to the factory structure can be calculated.

[0073] Based on the motion perception data, the pose information is constrained and corrected to obtain the spatial pose information of the UAV. For example, when the UAV passes through a densely populated factory area with many obstructions, some visual features may be unstable, but the pose drift can still be corrected by the short-term motion continuity constraints provided by the IMU data, thus avoiding the accumulation of positioning errors.

[0074] For example, in an application scenario of drone inspection of a large power plant area, the drone flies along a preset route over the substation, transmission towers, and plant roads, continuously collecting video frame data and motion perception data during flight. The motion perception data includes IMU data, GPS / BeiDou positioning data, and flight pose data. When the drone reaches the transmission tower area, it first extracts image feature points from the current video frame data, such as crossarm connection points, insulator string endpoints, and tower structure edge points. These image feature points are then matched with image feature points in a pre-built feature point database to obtain the correspondence between the image feature points in the current video frame data and the corresponding image feature points in the 3D semantic map. For example, the fixed connection nodes of the tower structure have stable geometric features at different times and from different perspectives, thus enabling stable matching. After obtaining the correspondence, based on the mapping relationship between the image feature points and their corresponding 3D spatial points, the coordinates of the corresponding 3D spatial points are obtained from the 3D semantic map, and the initial pose information of the current video frame data in 3D space is calculated according to the correspondence between the 2D image feature points and the 3D spatial points. For example, when a drone is located a certain distance directly in front of a pole, its spatial position relative to the pole, as well as attitude information such as pitch and heading angles, can be calculated by using the spatial correspondence of multiple feature points on the pole structure. This yields an initial spatial pose based on visual information. Subsequently, the initial pose information is constrained and corrected based on motion perception data. For instance, angular velocity and acceleration information from IMU data are used to constrain the drone's short-term trajectory, and the visual calculation results are fused and corrected by combining absolute position references provided by GPS / BeiDou positioning data. When the drone flies over areas with obstructions or weak textures, such as above a factory pipe gallery, some visual feature points may not match consistently. However, motion continuity can still be maintained using IMU data, thus avoiding jumps or drifts in pose estimation and ensuring the stability and continuity of spatial pose information.

[0075] In practical applications of the embodiments of this application, see Figure 3 In step S4, target detection is performed based on the video frame data to obtain the pixel coordinate information of the target. This pixel coordinate information is then converted into the target's true spatial coordinates using the spatial pose information. Specifically, this includes:

[0076] The target is detected based on the video frame data to obtain the pixel coordinate information of the target in the video frame data. For example, in an industrial park inspection scenario, the drone can detect pedestrians, vehicles or equipment targets on the ground through video frame data during flight and obtain their pixel coordinate positions in the image, thereby obtaining the position expression of the target in the two-dimensional image plane.

[0077] Based on the spatial pose information, the camera extrinsic parameters corresponding to the video frame data are determined, and the mapping relationship between the pixel coordinate system and the camera coordinate system is established by combining the camera intrinsic parameters. The camera intrinsic parameters are used to describe the imaging geometry, and the camera extrinsic parameters are used to describe the position and attitude of the UAV in three-dimensional space, thereby realizing the basis for spatial transformation from the image plane to the three-dimensional camera coordinate system.

[0078] Based on the pixel coordinate information and the mapping relationship, the target pixel coordinates are converted into a spatial direction vector in the camera coordinate system. For example, when a ground vehicle target is detected, its pixel coordinates can be converted into a spatial ray in the camera coordinate system through the imaging back projection relationship. This spatial direction vector represents the target's line of sight in three-dimensional space, but its specific distance information has not yet been determined.

[0079] By combining the 3D spatial points in the region corresponding to the spatial direction vector in the 3D semantic map, spatial constraint calculations are performed on the spatial direction vector to determine the target's 3D spatial coordinates in real space. For example, in an industrial park scenario, when a drone detects a vehicle target on the road, the spatial direction vector will interact with 3D spatial points such as the road plane and parking area in the 3D semantic map to determine the target's actual 3D spatial position in the road area through spatial intersection solving or nearest neighbor constraint calculation, thereby completing the accurate mapping from 2D pixel coordinates to real spatial coordinates.

[0080] For example, in an intelligent inspection application scenario at a coastal port container terminal, a drone conducts low-altitude patrols along the operational channel between berths and the yard, monitoring container loading and unloading, vehicle scheduling, and personnel operations in real time. In this scenario, when the drone flies near berth 3, a container truck entering the loading and unloading area is detected in the video frame data. The target detection algorithm first outputs the pixel coordinates of the truck in the image, for example, at position (860, 540) in the current video frame. This coordinate only indicates its position in the image plane. Subsequently, based on the spatial pose information corresponding to this video frame data, the current camera extrinsic parameters of the drone (i.e., the drone's position and attitude in the port's three-dimensional space) can be determined, and a mapping relationship between the pixel coordinate system and the camera coordinate system can be established by combining the camera intrinsic parameters. Under this mapping relationship, back-projection calculation of the pixel coordinates (860, 540) yields a spatial direction vector originating from the center of the drone's camera and pointing towards the port ground. This vector points to the intersection area of ​​the container yard and the operational channel. However, this directional vector still cannot determine the truck's exact landing location. Therefore, spatial constraint calculations are further performed using a pre-constructed 3D semantic map. In this 3D semantic map, the port area has been modeled and divided into berth water boundaries, container yard areas, transport channels, and loading / unloading areas, with each type of area corresponding to a dense set of 3D spatial points. For example, when the spatial directional vector intersects with the "transport channel ground plane" in the 3D semantic map, the truck's specific 3D spatial coordinates on the channel can be calculated, such as (x=128.5m, y=76.2m, z=0m). If the directional vector also has a proximity constraint with the "loading / unloading area boundary," the position is further corrected using the nearest 3D spatial point, so that it ultimately lands near the center of the loading / unloading waiting area, thus obtaining the truck's true spatial coordinates. For example, in actual inspections, when a truck moves from the yard to the quay crane lifting area, the pixel position of the target detected by the UAV in consecutive frames changes continuously. However, through the mapping process of pixel coordinates, spatial direction vectors, and 3D semantic map constraints, its actual movement trajectory in the port road can be continuously calculated without positioning drift due to changes in viewpoint or scale. This practical application demonstrates that this application can accurately map the pixel coordinates of targets in 2D images to 3D spatial coordinates in complex scenarios such as ports with numerous obstructions, dense targets, and frequent viewpoint changes. Combined with spatial constraints from a 3D semantic map, this achieves precise target positioning in the real working environment, providing a reliable spatial foundation for subsequent vehicle scheduling analysis, work path optimization, and safety incident identification.

[0081] In practical applications of this application, step S5 involves target tracking and behavior analysis based on the video frame data and the target's real spatial coordinates, and generating structured event data containing spatial coordinates by combining scene semantic information. Specifically, this includes:

[0082] Based on the video frame data, cross-frame association of targets is performed. For example, when a drone inspects an industrial park, the same vehicle traveling on the road is continuously detected in the video frames. Preliminary association is made based on changes in the target's appearance features and location information, and consistent matching of cross-frame targets is performed by combining the target's real spatial coordinates. For example, when multiple vehicle targets are detected in adjacent video frames, they can be filtered and matched based on the trend of their corresponding three-dimensional spatial coordinate changes, thereby eliminating possible mismatches that may occur if only two-dimensional images are relied upon. This establishes the correspondence between targets in consecutive video frames, enabling multi-target tracking and obtaining the target's motion trajectory.

[0083] Based on the target's trajectory and corresponding time series information, the target's motion parameters are calculated, and the target's behavioral state is determined according to these parameters. For example, the target's speed is calculated through the spatial distance and time interval between trajectory points, the turning angle is calculated through changes in trajectory direction, and the stability of the target's motion is judged through changes in acceleration, thereby achieving a quantitative analysis of the target's behavioral state. For example, a vehicle traveling at a constant speed on the main road of the park can be determined as "normal passage," while a vehicle lingering at low speed for a long time at the entrance of the warehouse area can be determined as "abnormal stop," thus transforming the target's motion trajectory into analyzable behavioral state information.

[0084] The target's real spatial coordinates are mapped to the corresponding scene area in the 3D semantic map to obtain the scene semantic information of the target's location. The scene area is a region with preset semantic attributes in the 3D semantic map, which is divided based on semantic classification results. The scene semantic information includes the semantic category of the scene area and its corresponding attribute information. For example, when a vehicle is located near the loading and unloading port in the park's storage area, its spatial coordinates can be mapped to the scene area "loading and unloading operation area" in the 3D semantic map. This area has preset semantic attributes, such as "frequent operation", "vehicle speed limit", and "dense personnel", thereby establishing a correspondence between the target's spatial location and the scene semantic information.

[0085] Based on the target's behavioral state, real spatial coordinates, and corresponding scene semantic information, an association determination is performed to identify the event type corresponding to the target and generate structured event data containing target identifier, spatial coordinates, behavioral state, and event type. For example, when a vehicle is detected to be "lingering" in the "loading and unloading area," it can be determined as a "suspected illegal parking event" according to preset rules; when personnel are detected moving in the "restricted area," it can be determined as an "illegal entry event"; when equipment exhibits abnormal movement in the "high-risk work area," it can be determined as an "equipment malfunction event." The association determination is based on preset rules, which clarify the correspondence between the target's behavioral state, spatial location, and scene semantic information, thereby achieving automated determination of event types.

[0086] The association determination is based on preset rules, which include the correspondence between target behavior state, spatial location and scene semantic information.

[0087] For example, in a large port container yard's intelligent drone inspection application scenario, the drone periodically patrols along the berth crane operation area, container yard roads, and vehicle access lanes to monitor vehicle scheduling and operational behavior in real time. During the inspection, when the drone flies to the intersection of Yard 2 and the main transport channel, a container truck is continuously detected in the video frame data. First, the truck is correlated across consecutive video frames. For example, if a vehicle target with pixel position (820, 510) is detected in frame t, and the same vehicle with a slightly changed position is detected in frame t+1, a preliminary correlation is established based on appearance features and position changes. However, relying solely on two-dimensional images is susceptible to occlusion and changes in viewing angle. Therefore, consistency matching is further performed by combining the target's real spatial coordinates. For example, the three-dimensional spatial coordinates corresponding to the vehicle in each frame are compared to confirm that it moves continuously in space and conforms to vehicle motion patterns. This establishes a stable correspondence between the truck and the container truck in consecutive video frames and generates its continuous motion trajectory on the port roads. After obtaining the movement trajectory, the target's motion parameters are calculated by combining the corresponding time series information, such as calculating the vehicle speed based on the spatial distance and time interval between adjacent trajectory points. When the truck is detected traveling at a constant speed of approximately 25 km / h on the main channel, its behavior can be classified as "normal passage." When its speed significantly decreases after entering the loading and unloading area of ​​the yard and it remains in the same spatial area for more than a preset time threshold, it can be classified as "abnormal stopping behavior." Subsequently, the truck's real spatial coordinates are mapped to a scene area in a 3D semantic map. For example, its spatial location falls into the semantic area of ​​"container loading and unloading operation area," which has been marked with attribute information such as "high-frequency operation area," "vehicle speed limit area," and "densely populated area" in the 3D semantic map, thereby obtaining the scene semantic information of the target's location. Based on this, a correlation judgment is made based on the target's behavior status (such as abnormal stopping), real spatial coordinates, and scene semantic information. For example, according to the preset rule "a vehicle staying in the loading and unloading operation area for a long time and not in a loading and unloading state is judged as a violation," the event type is judged as "suspected illegal parking event." Simultaneously, structured event data is generated, which includes target identifier (truck ID), spatial coordinates (three-dimensional coordinates of the stopping point), behavioral status (abnormal stopping), and event type (suspected illegal parking).

[0088] As can be seen from the above practical application process, this application can achieve stable cross-frame tracking of targets in environments such as ports where vehicles are dense, operations are complex, and obstructions are frequent. It also eliminates the influence of two-dimensional image errors by combining real spatial coordinates, and introduces scene semantic information through three-dimensional semantic maps, so that the determination of target behavior no longer depends solely on motion features, but is comprehensively judged in combination with its operating environment, thereby significantly improving the accuracy and reliability of event recognition.

[0089] In some other embodiments of this application, the specific steps for tracking a target and obtaining its motion trajectory include: for the obtained continuous video frame data, the latest decoded frame image is used as the current frame image, and the previous frame is used as the reference frame image, thereby ensuring strict temporal continuity between video frames. Based on this, the current frame image is input into a pre-trained target detection model to obtain the position and category information of each detected target in the current frame, and a target coordinate matrix for the current frame is constructed; simultaneously, the corresponding detection results in the reference frame are used to construct a target coordinate matrix for the reference frame, which is then used for subsequent cross-frame association processing. After obtaining the target detection results of the preceding and following frames, feature point detection and feature description are performed on the current frame image and the reference frame image respectively, extracting local stable feature points and their corresponding feature descriptors from the two frames. Matching is performed based on the feature descriptors to obtain cross-frame feature point matching pairs, and the homography matrix between the current frame and the reference frame is calculated based on the matching pairs to characterize the geometric transformation relationship between the two frames during the UAV's movement. Since the UAV experiences displacement, rotation, and viewpoint changes during flight, constructing the homography matrix can effectively describe the overall spatial mapping relationship between preceding and following frames. Furthermore, based on the homography matrix, coordinate mapping is performed on the target coordinate matrix of the reference frame to transform the target positions in the reference frame to the current frame coordinate system, obtaining a mapped coordinate matrix. Then, the mapped coordinate matrix is ​​matched with the target coordinate matrix of the current frame. Combining the spatial proximity and category consistency of the targets, cross-frame association is performed on the detected targets in consecutive frames to determine detection results belonging to the same physical target and establish the correspondence between targets in adjacent video frames. After completing the cross-frame association based on image coordinates, the target positions in the image coordinate system are further transformed to the real space coordinate system by combining the UAV's pose information or camera extrinsic parameters, obtaining the position representation of each target in real space. Based on the real space coordinates, consistency matching verification is performed on the cross-frame association results. By judging whether the spatial position changes of the same target in consecutive frames meet preset motion constraints, the association results are filtered and corrected, thereby effectively reducing mismatch problems caused by occlusion, viewpoint changes, or detection errors, and improving the accuracy and stability of cross-frame target matching. Based on this, a unique identifier ID is assigned to each target that has achieved consistency matching, and the target ID is continuously maintained in consecutive video frames to achieve stable tracking of multiple targets. At the same time, the position information of the same target in each consecutive frame is accumulated in chronological order to construct the target's motion trajectory, thereby obtaining the target's continuous motion path from the UAV's perspective.

[0090] In some embodiments of this application, step S6 of the embodiments of this application performs semantic verification based on the video frame data and the structured event data to obtain event analysis results, specifically including:

[0091] Based on the event types and their corresponding judgment confidence levels in the structured event data, the initial judgment result and corresponding confidence information of the event are determined. For example, during the inspection of industrial parks or port areas by drones, based on the target behavior status, real spatial coordinates and scene semantic information, it may be initially determined that the behavior of a vehicle staying in the loading and unloading area for a long time is a "suspected illegal parking event", and an initial confidence value, such as 0.72, is given to characterize the credibility of the judgment.

[0092] When the confidence level is below a preset threshold or the event type belongs to a preset high-risk event, the structured event data and corresponding video frame data are input into a preset view analysis model for semantic verification. For example, in cases involving personnel entering restricted areas, abnormal stays in hazardous work areas, or abnormal equipment behavior, the structured event data and corresponding video frame data are input into a preset view analysis model for semantic verification. This process not only utilizes the rule-based reasoning results in the structured event data but also combines the original video frame data for a deeper level of semantic understanding.

[0093] Based on the view analysis model, semantic understanding is performed on the video frame data. Combining this with the target behavior state, real-space coordinates, and scene semantic information in the structured event data, consistency verification and semantic reasoning are performed on the initial judgment result to obtain a revised judgment result. Specifically, based on the view analysis model, semantic understanding is performed on the video frame data, such as identifying fine-grained behavioral features of the target, the surrounding environment state, and target interaction relationships. This is then combined with the target behavior state, real-space coordinates, and scene semantic information in the structured event data to perform consistency verification and semantic reasoning on the initial judgment result. For example, in a "suspected illegal parking incident," the model can further identify whether the vehicle is in the process of loading and unloading goods, whether there is personnel loading and unloading goods, and whether the spatial location is a permitted temporary parking area, thereby determining whether the initial event is valid.

[0094] Based on the confidence levels of the initial judgment result and the verification judgment result, the two are weighted and fused or the result is selected based on preset rules to determine the event analysis result. For example, when the confidence level of the initial structured event data is high and the verification results are consistent, the final event confidence level can be improved. When there is a conflict between the two, the semantic verification result is preferred or the two are weighted and fused according to preset rules to obtain a more stable event analysis result.

[0095] The event analysis results include at least the event type, target identifier, real-world spatial coordinates, and event confidence level. For example, in port inspections, the final output might be "suspected illegal parking event," providing the target truck ID, three-dimensional spatial coordinates (parking location), event type, and overall confidence level (e.g., 0.89), thus forming a more reliable structured event output.

[0096] For example, in a smart monitoring scenario at a large port container terminal, drones continuously inspect the transport channel between the quay crane operation area and the container yard to identify illegal vehicle parking and operational safety risks. During one inspection, based on video frame data and structured event data, a container truck was detected to have stayed at the "Entrance Channel of Loading and Unloading Area No. 2" for more than a preset threshold (e.g., continuously staying for 90 seconds). Combining its trajectory changes (speed close to 0 and no obvious displacement) and spatial semantic information (this area is marked as a "fast passage channel" on the 3D semantic map), preliminary structured event data was generated. The event type was "suspected illegal parking event," with an initial confidence level of 0.68, lower than the preset threshold of 0.8. At this point, because this event belongs to a high-risk event type in port safety management (involving the risk of blockage in the operation channel), a semantic review process was triggered. The structured event data and the corresponding video frame data were input into a preset view analysis model for review.

[0097] In the embodiments of this application, the view analysis model can be implemented using one or more of the following models: For example, a deep learning model based on spatiotemporal feature modeling, such as a spatiotemporal video understanding model (3D CNN+Transformer structure), can be used to simultaneously extract the spatial appearance features and temporal dynamic features of the target; a video target behavior recognition model (such as a SlowFast network structure) can also be used to identify whether the vehicle is in a loading / unloading, waiting, or abnormally stopped state; in more complex implementations, a multimodal visual language model (Vision-Language Model, VLM) can also be used to jointly reason with the semantic information of video frames and structured event data (target behavior state, spatial coordinates, scene semantic information) to output a higher level of semantic judgment results.

[0098] During the review process, the view analysis model performs semantic understanding on the video frame data, such as identifying whether there are forklifts loading and unloading in front of the truck, whether there are personnel directing the operation, and whether it is in a normal queuing state. Simultaneously, it combines the real spatial coordinates in the structured event data to compare the target location with the "loading and unloading buffer zone" and "main passageway" in the 3D semantic map for consistency. For example, it further identifies that: there is indeed a quay crane loading and unloading queue 20 meters in front of the truck; multiple vehicles are linearly queued; although the current spatial area is marked as a "passageway," a portion has been temporarily adjusted to a "work waiting area." Based on the above semantic understanding and spatial consistency analysis, the view analysis model outputs the review result as "normal work waiting," with a review confidence level of 0.86. Subsequently, a weighted fusion or rule selection is performed based on the initial judgment result (suspected illegal parking, 0.68) and the review judgment result (normal waiting, 0.86). For example, if the preset rule stipulates that "when the review result conflicts with the initial event and the review confidence is higher, the review result shall prevail", then the final event analysis result shall be corrected to "normal operation waiting" and the overall confidence level shall be updated to 0.82. At the same time, the target identifier, real spatial coordinates and event type shall be recorded.

[0099] As can be seen from the above practical examples, this application performs secondary semantic verification on structured event data through a view analysis model, so that event judgment no longer depends on a single rule or behavioral characteristics at a single moment, but combines the spatiotemporal semantic information of video frames with the spatial constraints of the three-dimensional semantic map to perform consistent reasoning, thereby effectively reducing the risk of false alarms. Especially in the case of dynamic changes in port operation processes (such as temporary queuing and adjustment of operation areas), the accuracy of event analysis results is significantly improved.

[0100] Optionally, in some embodiments of this application, step S7, when the event analysis result is a preset event, involves controlling the UAV to perform multi-view data acquisition based on the spatial coordinates in the structured event data and generating a three-dimensional scene model of the target area, specifically including:

[0101] The target area is determined based on the spatial coordinates in the structured event data. For example, when an abnormal stop is detected in the No. 2 loading and unloading channel of the port, its real spatial coordinates are used to locate the point where the event occurred. The target area is determined with the spatial coordinates as the center and a preset safety range. The center position of the target area is calculated based on the spatial coordinates. The multi-view acquisition path of the UAV is planned based on the center position, so that the UAV can cover the target area from different angles.

[0102] After path planning is completed, the UAV is controlled to fly according to the multi-view acquisition path to collect data around or from multiple angles of the target area, thereby obtaining image data and corresponding motion perception data from different perspectives. For example, the UAV can fly in a circle with a radius of 10-30 meters around the point of occurrence of the event, or perform flight modes such as "Z" or "spiral ascent / descent" to collect image data and corresponding motion perception data from different heights and perspectives, thereby obtaining multi-view observation information covering the target area.

[0103] The acquired image data and motion-sensing data undergo time synchronization and alignment processing. The image data is then filtered based on image sharpness, viewpoint coverage, or data integrity to obtain effective data for 3D reconstruction. Specifically, the acquired image data and motion-sensing data are time-synchronized and aligned, for example, by precisely matching video frames from different viewpoints with their corresponding UAV poses and IMU data based on timestamps, ensuring that each frame has accurate spatial pose information. Furthermore, the image data is filtered based on image sharpness, viewpoint coverage, or data integrity, such as removing blurry frames, overexposed frames, or frames with duplicate viewpoints, thereby obtaining effective data for 3D reconstruction and improving the quality of subsequent reconstruction.

[0104] Based on the effective data, pixel-level depth estimation is performed on the image data using a multi-view stereo matching method to generate dense point cloud data of the target area. For example, by matching the same building structure or vehicle outline from different viewpoints, its depth information is calculated, thereby restoring the three-dimensional geometric structure of the target area. The dense point cloud data is then subjected to denoising and dynamic target removal processing, such as filtering out dynamic point cloud interference caused by vehicle movement, pedestrian walking, etc., and retaining only stable static structural point clouds, thereby improving the stability and accuracy of three-dimensional reconstruction.

[0105] A 3D scene model of the target region is constructed based on the processed dense point cloud data, and the 3D scene model is subjected to texture mapping processing to obtain a 3D scene model with texture information.

[0106] This application enables automatic multi-view refined data acquisition of the target area after an event is triggered, and combines motion perception data to achieve high-precision temporal and spatial alignment, thereby effectively improving the geometric consistency and detail representation capability of 3D reconstruction. Simultaneously, through multi-view stereo matching and dense point cloud generation, it achieves precise restoration of complex scene structures, and improves model stability through denoising and dynamic target culling. The final generated 3D scene model with texture information can intuitively reflect the real environmental state of the event occurrence area, providing high-precision spatial foundational data support for subsequent event review, accountability tracing, and safety assessment.

[0107] For example, in an emergency response scenario involving a traffic accident on a highway, the road monitoring system or vehicle reporting system first generates structured event data, identifies it as a "traffic accident event," and determines the actual spatial coordinates of the accident (e.g., the middle section of the eastbound lane at K125+300 on the highway). This structured event data also includes the event type, initial confidence level, and basic target information. Once the event analysis confirms it belongs to a preset event (e.g., a high-risk traffic accident event), an unmanned aerial vehicle (UAV) emergency response mission is automatically triggered based on the spatial coordinates in the structured event data. First, the target area is determined based on these spatial coordinates. For example, using the accident location as the center, and considering the road direction and lane width, a coverage area extending 200 meters upstream and downstream is formed. This central location serves as the spatial reference point for the UAV's multi-view data collection. Based on this, the UAV's multi-view data collection path is planned based on this central location. For example, a "circling + segmented overhead" flight path is planned along the longitudinal direction of the highway, allowing the UAV to comprehensively cover the accident scene from above, upstream oncoming traffic, downstream evacuation directions, and lateral heights. For example, the drone first takes a vertical overhead shot from above the accident site, then descends to fly back and forth along the road, and finally takes a diagonal shot from the side of the road to acquire image data from different perspectives and corresponding motion sensing data (including GPS position, IMU attitude angle, and flight altitude). During the multi-view data acquisition process, the acquired image data and motion sensing data undergo time synchronization and alignment processing. For example, each frame of the accident scene image is precisely matched with the corresponding drone pose information to ensure that each image has an accurate spatial projection relationship. Based on this, the image data is filtered based on image clarity, view coverage, and data integrity. For example, blurred frames caused by high-speed vehicle passage, overexposed frames due to backlighting, and frames with repeated angles are removed, thus retaining key view data that can effectively reflect the structure of the accident scene. Subsequently, based on the effective data, pixel-level depth estimation is performed on the image data using a multi-view stereo matching method. For example, by performing feature matching on the vehicle collision position, guardrail deformation area, and debris distribution area at different angles, the depth information of each point is calculated to generate dense point cloud data covering the accident area. After generating a dense point cloud, it is subjected to noise reduction and dynamic target removal processing. For example, the dynamic point clouds of moving rescue vehicles, traffic police vehicles and passing vehicles are filtered out, and only the road structure, static shape of the colliding vehicles and road facilities at the moment of the accident are retained, so as to avoid dynamic interference affecting the reconstruction accuracy.Finally, a 3D scene model of the accident scene is constructed based on the processed dense point cloud data. This model restores the vehicle collision position relationships, guardrail damage patterns, lane occupancy, and spatial distribution of debris. Furthermore, texture mapping processing is applied to the 3D scene model to map the real texture information of the scene images onto the surface of the 3D model, presenting the accident scene in a highly realistic manner. This includes details of vehicle damage, road surface brake marks, and guardrail deformation, thereby generating a 3D scene model with spatial coordinate accuracy and visual realism.

[0108] Optionally, in some embodiments of this application, during the multi-view data acquisition process, the image data acquisition adopts an adaptive sampling strategy based on changes in image feature points, specifically including:

[0109] During the multi-view data acquisition process of the UAV, image feature points are extracted from the continuously acquired video frames; for example, road edge points, vehicle outline points, building structure corner points and texture salient points are extracted, and the corresponding feature descriptors are calculated simultaneously to characterize the local image structure information under different viewpoints.

[0110] The similarity between adjacent video frames is calculated based on the changes in the number and spatial distribution of image feature points in adjacent video frames. For example, when the drone is flying smoothly and the scene changes little, the number of feature points extracted between adjacent video frames is basically stable and the spatial distribution difference is small, so the calculated similarity is high. Conversely, when the drone turns quickly, flies over accident areas or enters structurally complex areas, the number and distribution of feature points will change significantly, so the similarity is low.

[0111] Based on the comparison result between the similarity and the preset threshold, the sampling interval of the video frames is adjusted according to the preset sampling control rules;

[0112] Specifically, when the similarity is higher than the preset threshold, it indicates that the current scene changes little, such as when the drone is flying on a straight section of a highway or in an open area. In this case, the sampling interval of the video frames is extended according to the preset sampling control rules, thereby reducing the collection of redundant information and reducing data redundancy and transmission and computing burden. When the similarity is lower than the preset threshold, it indicates that the current scene changes significantly, such as when the drone is approaching a traffic accident scene, a densely trafficked area, or a complex road structure area. In this case, the sampling interval of the video frames is shortened according to the preset sampling control rules, thereby increasing the sampling density of key areas and retaining more detailed information.

[0113] This application enables dynamic sampling control based on the degree of scene change, allowing UAVs to adaptively adjust the sampling strategy according to changes in image feature points during multi-view data acquisition without using a fixed sampling frequency. This ensures the integrity of key scene information such as accident sites and complex structural areas while effectively reducing redundant data acquisition, improving data utilization efficiency and the quality and efficiency of subsequent 3D reconstruction, and enhancing adaptability and practicality in complex dynamic scenes.

[0114] In this embodiment, the method further includes: during the flight mission of the UAV, acquiring currently collected video frame data and corresponding motion perception data, extracting image feature points based on the video frame data, and matching the image feature points with image feature points in the feature point database to obtain feature point matching results; for example, during the inspection of industrial parks or urban road areas, when the UAV flies to a fixed intersection or factory area, image feature points such as road marking intersections, building edge points or tower structure points can be extracted and matched with stable landmark feature points pre-stored in the feature point database to determine whether the current viewpoint has consistent structural features with the historical map.

[0115] The feature point matching rate is calculated based on the feature point matching results. When the feature point matching rate is lower than the preset matching threshold, the corresponding scene area is determined as a candidate change area. For example, when a drone inspects a factory area again, if the number of matching feature points at the intersection of road markings or the edge of the factory decreases significantly due to the obstruction of new equipment or changes in the building structure, it indicates that there is a structural difference between the area and the original 3D semantic map, and thus the factory area is determined as a candidate change area.

[0116] Dense point cloud data of the current region is generated based on the video frame data corresponding to the candidate change region. The dense point cloud data is then compared with the 3D spatial points of the corresponding region in the 3D semantic map for spatial consistency. Simultaneously, semantic consistency detection is performed by combining the semantic classification information in the 3D semantic map to confirm the change of the candidate change region and determine the change type based on semantic differences. For example, in road construction or factory expansion scenarios, the dense point cloud may show newly added structures or road widening structures, which are inconsistent with the semantic classification results of the original road surface, illegal parking areas, etc. The structural or semantic change of the region can be confirmed by the spatial point offset and semantic label change, and the change type can be further determined based on semantic differences, such as the addition of buildings, road reconstruction, or obstacle removal.

[0117] For the confirmed change area, the UAV is controlled to collect incremental data in the change area, and feature point extraction, matching and three-dimensional reconstruction processing are re-executed based on the newly added video frame data and the corresponding motion perception data to generate updated sparse point cloud, dense point cloud and corresponding three-dimensional mesh model.

[0118] The updated 3D mesh model and the original 3D semantic map are geometrically and semantically fused within the corresponding scene area. Geometric fusion is based on 3D spatial point alignment results to ensure spatial structural consistency; semantic fusion is based on semantic classification results to ensure semantic label uniformity. For example, in a newly added equipment area within a factory, the newly generated 3D mesh model can be spatially aligned and fused with the existing factory structure, while the newly added equipment area is labeled with a new semantic category, achieving map semantic expansion. The fusion result is verified based on the consistency of feature points, spatial points, and semantics before and after fusion, generating an updated 3D semantic map.

[0119] In this embodiment, the specific application scenario of "inspection and updating after a traffic accident near a highway service area" is used for illustration. For example, a multi-vehicle rear-end collision occurs on a highway. The traffic police department has completed the initial accident handling, but the road markings at the scene are covered by skid marks, some guardrails are deformed, and rescue vehicles are temporarily parked, causing changes to the original scene structure. First, based on the spatial coordinates in the structured event data generated by the accident report, the drone is controlled to fly over the accident area for inspection. During the flight, the currently collected video frame data and corresponding motion perception data are acquired, and image feature points are extracted based on the video frame data. The extracted image feature points are then matched with a pre-built feature point database in the 3D semantic map. For example, in the original 3D semantic map of this highway section, the accident section contains clear stable feature points such as the intersection of the road centerline, guardrail endpoints, and shoulder boundary points. However, after the accident, in the current video frame collected by the drone, due to guardrail bending, vehicle obstruction, and ground debris coverage, some road marking intersections and guardrail endpoints cannot be stably extracted, resulting in a significant decrease in the number of feature point matches. Based on this, the feature point matching rate is calculated. When the matching rate is lower than a preset matching threshold, the highway accident section is identified as a candidate change area. Subsequently, dense point cloud data is generated for the video frame data corresponding to the candidate change area. For example, when a drone flies around the accident area at low altitude from multiple angles, it samples the vehicle collision point, the guardrail deformation area, and the distribution of road debris from multiple perspectives, generating a local high-density point cloud. When the spatial consistency of this dense point cloud is compared with the corresponding 3D spatial points of the road area in the original 3D semantic map, it is found that the guardrail position has been significantly displaced, and the height distribution of some road areas is abnormal. At the same time, semantic consistency detection is performed by combining semantic classification information (such as road surface, guardrail area, emergency lane area), confirming that the area has undergone structural changes, and further determining the change type as "road facility damage and temporary obstacle occupation". After confirming the changes, the drone was controlled to collect incremental data on the core accident area, such as focusing on the collision point, damaged guardrail sections, and vehicle wreckage areas for supplementary filming. Based on the newly added video frame data and motion perception data, feature point extraction, matching, and 3D reconstruction were re-executed to generate updated sparse point clouds, dense point clouds, and corresponding 3D mesh models. This updated model can more accurately reflect the bending shape of the guardrail, the extent of vehicle occupancy, and the degree of road damage. Furthermore, the updated 3D mesh model was geometrically and semantically fused with the original 3D semantic map at the accident site. Geometric fusion, based on 3D spatial point alignment results, spatially matched and replaced the deformed guardrail with the original guardrail structure; semantic fusion, based on semantic classification results, locally updated the original "passable road area" to "accident-occupied area" or "damaged road area," achieving dynamic updates at the semantic level. Simultaneously, the consistency of feature points, spatial points, and semantics before and after fusion was verified to avoid erroneous updates caused by temporary vehicles or obstructions.The resulting updated 3D semantic map not only accurately reflects the actual spatial structure of the accident scene, but can also be continuously used for subsequent traffic management, accident debriefing analysis, and road restoration assessment. For example, traffic management departments can directly calculate the width of passable lanes, the distribution range of obstacles, and the priority of obstacle removal based on this updated map, thereby significantly improving accident handling efficiency and spatial decision-making capabilities.

[0120] In other embodiments of this application, after the event analysis result is determined to be a preset event (e.g., traffic accident, abnormal gathering of people, initial fire, etc.), the multi-view data acquisition and 3D modeling steps are initiated. First, based on the structured event data generated in the preceding steps, the true spatial coordinates, spatial range (e.g., center point, major and minor axis range, or bounding box size), and event type information of the target area are parsed. Combining the UAV's current pose, battery level, and environmental information (e.g., wind speed, obstacle distribution), a preset layered acquisition strategy is automatically invoked to generate a set of multi-view acquisition paths. These paths typically include three types: a surround panoramic view, a key point hovering view, and a close-up perspective, to achieve progressively detailed observation of the target area from the overall to the local. For example, in a rear-end collision scenario, a circular flight path centered on the accident center is first planned, and low-speed circular shooting is conducted at a relatively high altitude (e.g., 40 meters) to obtain the overall layout of the accident scene and road environment information. Subsequently, multiple hovering points are set up around the collision point of the two vehicles to collect high-resolution images from different directions (front, rear, and side) to capture details of vehicle damage. Finally, the drone is controlled to lower its altitude and approach the core area of ​​the collision, and close-up data of key parts (such as deformed structures and scattered debris) is obtained through zoom or short-distance flight.

[0121] During flight data acquisition, the UAV simultaneously acquires video frames, LiDAR point clouds, and IMU pose data at a dynamically adjusted sampling rate. It also adaptively extracts frames based on motion saliency and geometric complexity, retaining only keyframes to reduce redundant data. Simultaneously, real-time image quality assessment (e.g., sharpness, exposure) and obstacle detection mechanisms are used to correct the acquisition path online. For example, it automatically re-captures images when occlusions or blurry images occur, or briefly pauses when dynamic targets (e.g., moving pedestrians or vehicles) are detected to avoid interference.

[0122] After completing multi-view data acquisition, keyframe data is input into the 3D reconstruction module, and a Gaussian representation-based 3D modeling method is preferred for scene reconstruction. Specifically, firstly, Gaussian ellipsoids in 3D space are initialized using LiDAR point clouds, with each point corresponding to a learnable Gaussian distribution unit containing parameters such as position, scale, orientation, and color. Then, combined with the precise pose information of the UAV, these Gaussian units are projected onto the image planes of various viewpoints using differentiable rendering technology, and the difference loss between the rendered result and the real image is calculated. Through multiple rounds of iterative optimization, the Gaussian unit parameters are continuously adjusted, gradually making the reconstruction result approximate the real scene. This method can generate high-fidelity 3D models in a short time and supports browsing from any viewpoint.

[0123] For example, in fire accident scenarios, images and point cloud data acquired from multiple perspectives can be used to reconstruct 3D models of building facades, fire locations, and smoke diffusion paths. In cases of large gatherings, crowd density and spatial relationships can be reconstructed, providing a basis for subsequent safety assessments. The final generated 3D scene model not only includes geometric structural information but can also be overlaid with event semantic tags (such as accident type and key target locations), forming an interactive digital representation of the scene and providing intuitive support for command and dispatch, accident analysis, and evidence collection.

[0124] In practical applications of this application, the method further includes: mapping the real spatial coordinates to the corresponding spatial location in the three-dimensional scene model based on the target identifier, real spatial coordinates, and event type information in the structured event data, and marking and displaying the corresponding target and event at the spatial location to achieve spatial visualization of the event; for example, in highway or industrial park inspection scenarios, when a drone detects events such as "vehicle driving in the wrong direction," "personnel entering the emergency lane," or "equipment malfunctioning," the real spatial coordinates of the target can be directly mapped to the specific road area or equipment location in the three-dimensional scene model, and different event types can be marked with different colors (e.g., red indicates high-risk events, yellow indicates general abnormal events), while simultaneously displaying the target identifier and behavior status, enabling managers to intuitively view the location of the event and its surrounding environment in three-dimensional space, thereby achieving a visualization transformation from a "two-dimensional event list" to a "three-dimensional spatial scene."

[0125] When the event type in the event analysis results belongs to a preset alarm event, alarm information is generated and output based on the event type, event confidence level, and real spatial coordinates in the event analysis results. For example, in a highway scenario, when a "multi-vehicle collision accident" or "severe traffic congestion and wrong-way driving" is detected, the event is determined to be a preset alarm event. At this time, combined with the event confidence level (e.g., 0.92) and the real spatial coordinates of the accident, structured alarm information is automatically generated and pushed in real time through the command and dispatch platform, mobile terminal, or monitoring center, thereby realizing rapid response and coordinated handling of high-risk events.

[0126] Event records are generated based on the structured event data and the event analysis results, and these records are stored and managed. Each event record includes at least a target identifier, event type, real-world spatial coordinates, time information, and event confidence level. The time information is obtained based on the timestamps of the video frame data. For example, in a drone inspection mission, each detected event (such as "vehicle occupying the emergency lane") is recorded as a structured event record, containing the specific occurrence time (e.g., 2026-05-21 10:32:15), spatial coordinates (latitude, longitude, and 3D altitude), event type, and confidence level information. These records can be uniformly stored in an event database for subsequent traceability analysis, statistical modeling, or law enforcement evidence collection.

[0127] This application enables deep integration of structured event data and 3D scene models, allowing events to exist not only in text or list form but also to be intuitively visualized in real spatial locations, significantly improving the intuitiveness of event perception and spatial understanding. Simultaneously, by automatically identifying preset alarm events and generating alarm information, rapid response and real-time push notifications for high-risk events can be achieved, improving emergency handling efficiency. Furthermore, by managing the structured storage of event records, events possess a complete information chain encompassing time, space, and semantics, facilitating subsequent traceability and analysis, and providing long-term data support for traffic management, industrial inspection, and safety supervision.

[0128] In one optional embodiment of this application, the unmanned aerial vehicle (UAV) device refers to an aircraft body employing a multi-rotor, fixed-wing, or compound-wing structure, possessing stable hovering, high-speed cruise, and vertical takeoff and landing (VTOL) capabilities to adapt to flight mission requirements under complex terrain and variable weather conditions. Specifically, multi-rotor UAVs are suitable for low-altitude precision inspections and fixed-point hovering observations in complex environments; fixed-wing UAVs are suitable for long-endurance, high-efficiency cruise operations over large areas; and compound-wing UAVs combine VTOL and high-speed cruise capabilities, enabling rapid deployment without runways and flexible switching between hovering and cruise states during missions, thereby meeting the spatial data acquisition and intelligent analysis needs of different application scenarios. Through these different structural forms of UAV devices, this application can adapt to various application environments such as complex terrain, urban environments, and industrial scenarios, improving the overall environmental adaptability and mission execution capabilities of the system.

[0129] In one specific embodiment of this application, the multi-source data also includes device status data, which is used for real-time monitoring and operation and maintenance management of the UAV's operational health status, thereby improving stability and mission reliability. Specifically, key operational status data of the UAV's energy system, flight control system, and communication system are collected in real time through various sensors and monitoring units installed on the UAV itself. Energy system data includes battery level, flight time, and charging / discharging status; flight control system data includes flight attitude parameters (such as pitch angle, roll angle, and yaw angle), positioning accuracy, and attitude stability indicators; communication system data includes signal strength, link quality, and data transmission rate. Furthermore, device operation data, such as motor speed, CPU utilization, memory utilization, GPU temperature, and communication signal strength, can be further acquired, thus forming a multi-dimensional monitoring system for the overall operational status of the UAV. During data acquisition, a comprehensive evaluation is performed based on the multi-source device status data to determine the current health status of the UAV and to predict and analyze potential failure risks. For example, when a battery charge is detected to be decreasing rapidly and accompanied by abnormal fluctuations in motor speed, it can be preliminarily determined that there is an abnormal risk in the energy system; when the CPU utilization rate is detected to be in a high-load state for a long time and the GPU temperature is continuously rising, it can be predicted that there is a risk of computing resource overload or abnormal heat dissipation; when the communication signal strength decreases and the data transmission rate is unstable, it can be determined that there is a risk of communication link quality degradation.

[0130] Furthermore, the device status is managed hierarchically based on preset multi-level warning thresholds, such as setting normal status thresholds, warning thresholds, and severe fault thresholds. When device status data exceeds or falls below the corresponding threshold, different levels of warning information are automatically triggered, and maintenance plans are formulated in advance according to the warning level. For example, a low battery warning is triggered when the battery level is below the first preset threshold, prompting the user to return to base or replace the battery; a medium-level warning is triggered when the GPU temperature exceeds the second preset threshold, suggesting reducing the computing load or suspending high-intensity tasks; and a high-level warning is triggered when the positioning accuracy of the flight control system continues to decline, automatically generating a maintenance task. Simultaneously, a device maintenance log is established to structurally record each maintenance process of the UAV. The maintenance log includes at least maintenance personnel information, maintenance time, maintenance content, and maintenance results. For example, after the UAV completes a battery replacement or flight control system calibration, the maintenance personnel's identity, maintenance operation type, specific replaced parts, and post-maintenance test results are automatically recorded, thus forming a complete maintenance record chain. This application enables real-time status monitoring and health assessment of key components such as the energy system, flight control system, and communication system of unmanned aerial vehicles (UAVs). Through the fusion and analysis of multi-source equipment status data, it improves the predictive ability for potential faults and achieves graded risk control through a multi-level early warning threshold mechanism, thereby effectively reducing the probability of sudden equipment failures. Simultaneously, by establishing an equipment maintenance log, it achieves full lifecycle management of equipment operation data and maintenance records, making the equipment status traceable and analyzable, which is beneficial to improving the operational efficiency and long-term operational reliability of UAVs.

[0131] See Figure 4 This application also provides an intelligent view analysis system based on unmanned aerial vehicles (UAVs), comprising: at least one UAV; and an intelligent view analysis platform, see [link to relevant documentation]. Figure 5 The intelligent view analysis platform includes:

[0132] The data acquisition module is used to acquire multi-source data collected by the UAV during flight, including at least video data and motion sensing data.

[0133] The view sampling module is used to process the video data to obtain video frame data;

[0134] The spatial perception algorithm module is used to determine the spatial pose information of the UAV based on the video frame data and the motion perception data, combined with a pre-constructed three-dimensional semantic map.

[0135] The video analysis algorithm module is used to perform target detection based on the video frame data, obtain the pixel coordinate information of the target, and convert the pixel coordinate information into the real spatial coordinates of the target by combining the spatial pose information; and to perform target tracking and behavior analysis based on the video frame data and the real spatial coordinates of the target, and generate structured event data containing spatial coordinates by combining scene semantic information.

[0136] The large view model module is used to perform semantic verification based on the video frame data and the structured event data to obtain event analysis results;

[0137] The 3D reconstruction module is used to control the UAV to perform multi-view data acquisition based on the spatial coordinates in the structured event data when the event analysis result is a preset event, and to generate a 3D scene model of the target area.

[0138] The intelligent view analysis platform in this application embodiment is presented in the form of functional modules. Here, a module refers to an ASIC (Application Specific Integrated Circuit), a processor and memory executing one or more software or fixed programs, and / or other devices that can provide the aforementioned functions. Furthermore, the specific implementation process of the intelligent view analysis platform described in the above embodiments is similar to the implementation process of the UAV-based intelligent view analysis method described in the above embodiments, and will not be described in detail here.

[0139] Please see Figure 6 This is a schematic diagram of an intelligent view analysis platform structure provided in one embodiment of this application, as shown below. Figure 6 As shown, the intelligent view analytics platform includes:

[0140] One or more processors, memory, and communication interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other using different buses and can be mounted on a common motherboard or otherwise as required. The processor can process instructions executed within the server, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interface). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories, if desired. Figure 6 Let's take a processor as an example.

[0141] The processor can be a central processing unit, a network processor, or a combination thereof. The processor may further include hardware chips. These hardware chips can be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The programmable logic devices can be complex programmable logic devices (CLPs), field-programmable gate arrays (FPGAs), general-purpose array logic (GDAs), or any combination thereof.

[0142] The memory stores instructions executable by at least one processor to enable the at least one processor to perform the UAV-based intelligent view analysis method shown in the above embodiments.

[0143] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on server usage. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory, or other non-transitory solid-state storage device. In some alternative embodiments, the memory may include memory remotely located relative to the processor, which can be connected to the server via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0144] The memory may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory may also include a combination of the above types of memory.

[0145] The communication interface is used for the server to communicate with other devices or communication networks.

[0146] This application also provides a computer-readable storage medium. The UAV-based intelligent view analysis method described above according to this application embodiment can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the UAV-based intelligent view analysis method described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that a computer, processor, microprocessor controller, or programmable hardware includes storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the UAV-based intelligent view analysis method shown in the above embodiments is implemented.

[0147] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0148] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for intelligent view analysis based on unmanned aerial vehicle, characterized in that, include: Acquire multi-source data collected by the drone during flight, wherein the multi-source data includes at least video data and motion sensing data; The video data is processed to obtain video frame data; Based on the video frame data and the motion perception data, combined with the pre-constructed three-dimensional semantic map, the spatial pose information of the UAV is determined. Target detection is performed based on the video frame data to obtain the pixel coordinate information of the target. The pixel coordinate information is then converted into the real spatial coordinates of the target by combining the spatial pose information and the pre-constructed three-dimensional semantic map. Target tracking and behavior analysis are performed based on the video frame data and the target's real spatial coordinates, and structured event data containing spatial coordinates is generated by combining the scene semantic information provided by the three-dimensional semantic map. Semantic verification is performed based on the video frame data and the structured event data to obtain event analysis results; If the event analysis result is a preset event, the UAV is controlled to perform multi-view data acquisition based on the spatial coordinates in the structured event data, and a three-dimensional scene model of the target area is generated.

2. The intelligent view analysis method based on unmanned aerial vehicles according to claim 1, characterized in that, The process of constructing a 3D semantic map includes: Perform time synchronization and alignment processing on multi-source data, and perform image distortion correction; Based on this, sparse point clouds are generated by feature point extraction and matching, and dense point clouds and 3D mesh models are generated by multi-view stereo matching. At the same time, a 3D semantic map and feature point database are generated by combining semantic segmentation algorithms.

3. The intelligent view analysis method based on unmanned aerial vehicles according to claim 2, characterized in that, The process of generating a 3D semantic map and feature point database includes: Image feature points are extracted from the corrected video frame data, and corresponding feature descriptors are calculated at the same time as the image feature points are extracted. Image feature points between adjacent video frames are also matched. Based on the matched image feature points and the motion sensing data, a camera motion trajectory is constructed, and the camera motion trajectory and the corresponding three-dimensional point coordinates are jointly optimized by an optimization method to obtain the optimized camera pose and three-dimensional spatial points, thereby generating a sparse point cloud. Based on the sparse point cloud, a multi-view stereo matching algorithm is used to perform pixel-level depth estimation on the video frame data to generate a dense point cloud, and a three-dimensional mesh model is constructed based on the dense point cloud. The three-dimensional mesh model is semantically classified based on a semantic segmentation algorithm to generate a three-dimensional semantic map; A feature point database is constructed by establishing associations between the image feature points, the feature descriptors and their corresponding three-dimensional spatial points. The image feature points in the feature point database are key scene feature points that can be stably detected under different time and viewing angle conditions. These key scene feature points include fixed landmark feature points such as building structure edge points, road sign points, and tower structure points.

4. The intelligent view analysis method based on unmanned aerial vehicles according to claim 3, characterized in that, Based on the video frame data and the motion perception data, combined with a pre-constructed 3D semantic map, the spatial pose information of the UAV is determined, specifically including: Image feature points are extracted from the video frame data and matched with image feature points in the feature point database to obtain the correspondence between image feature points in the current video frame data and image feature points in the feature point database; Based on the correspondence, obtain the three-dimensional spatial points corresponding to the image feature points in the current video frame data, and calculate the pose information of the current video frame data in three-dimensional space according to the correspondence between the image feature points and the three-dimensional spatial points; Based on the motion perception data, the pose information is constrained and corrected to obtain the spatial pose information of the UAV.

5. The intelligent view analysis method based on unmanned aerial vehicles according to claim 4, characterized in that, Target detection is performed based on the video frame data to obtain the target's pixel coordinate information. This pixel coordinate information is then converted into the target's true spatial coordinates using the spatial pose information. Specifically, this includes: The target is detected based on the video frame data to obtain the pixel coordinate information of the target in the video frame data; Based on the spatial pose information, the camera extrinsic parameters corresponding to the video frame data are determined, and the mapping relationship between the pixel coordinate system and the camera coordinate system is established in combination with the camera intrinsic parameters. Based on the pixel coordinate information and the mapping relationship, the target pixel coordinates are converted into spatial direction vectors in the camera coordinate system; By combining the three-dimensional spatial points in the region corresponding to the spatial direction vector in the three-dimensional semantic map, spatial constraint calculations are performed on the spatial direction vector to determine the three-dimensional spatial coordinates of the target in real space.

6. The intelligent view analysis method based on unmanned aerial vehicles according to claim 5, characterized in that, Target tracking and behavior analysis are performed based on the video frame data and the target's real spatial coordinates. Combined with scene semantic information, structured event data containing spatial coordinates is generated, specifically including: Based on the video frame data, the target is associated across frames, and the target's real spatial coordinates are combined to perform consistent matching of the cross-frame targets, establishing the correspondence between the targets in consecutive video frames, so as to realize multi-target tracking of the target and obtain the target's motion trajectory; The motion parameters of the target are calculated based on the target's motion trajectory and corresponding time series information, and the target's behavior state is determined based on the motion parameters. The real spatial coordinates of the target are mapped to the corresponding scene region in the 3D semantic map to obtain the scene semantic information of the target's location; wherein, the scene region is a region with preset semantic attributes in the 3D semantic map based on the semantic classification result; the scene semantic information includes the semantic category of the scene region and the corresponding attribute information; Based on the target's behavioral state, real spatial coordinates, and corresponding scene semantic information, an association judgment is made to determine the event type corresponding to the target, and structured event data containing target identifier, spatial coordinates, behavioral state, and event type is generated. The association determination is based on preset rules, which include the correspondence between target behavior state, spatial location and scene semantic information.

7. The intelligent view analysis method based on unmanned aerial vehicles according to claim 6, characterized in that, Based on the video frame data and the structured event data, a semantic review is performed to obtain event analysis results, specifically including: Based on the event types and their corresponding judgment confidence levels in the structured event data, the initial judgment result of the event and the corresponding confidence level information are determined. When the confidence information is lower than a preset threshold or the event type belongs to a preset high-risk event, the structured event data and the corresponding video frame data are input into a preset view analysis model for semantic verification processing. Based on the view analysis model, semantic understanding is performed on the video frame data. Combined with the target behavior state, real space coordinates and scene semantic information in the structured event data, consistency verification and semantic reasoning are performed on the initial judgment result to obtain the review judgment result. Based on the confidence levels of the initial judgment result and the review judgment result, the two are weighted and fused or the result is selected based on preset rules to determine the event analysis result; The event analysis results include at least the event type, target identifier, real spatial coordinates, and event confidence level.

8. The intelligent view analysis method based on unmanned aerial vehicles according to claim 7, characterized in that, When the event analysis result is a preset event, the UAV is controlled to perform multi-view data acquisition based on the spatial coordinates in the structured event data, and a three-dimensional scene model of the target area is generated, specifically including: The target area is determined based on the spatial coordinates in the structured event data, and the center position of the target area is calculated based on the spatial coordinates. The multi-view acquisition path of the UAV is planned based on the center position. The drone is controlled to fly along the multi-view acquisition path to perform circumnavigation or multi-angle data acquisition of the target area, thereby obtaining image data and corresponding motion perception data from different perspectives. The acquired image data and motion sensing data are time-synchronized and aligned, and the image data is filtered based on image clarity, viewpoint coverage, or data integrity to obtain effective data for 3D reconstruction. Based on the effective data, pixel-level depth estimation is performed on the image data using a multi-view stereo matching method to generate dense point cloud data of the target region, and the dense point cloud data is then subjected to denoising and dynamic target removal processing. A 3D scene model of the target region is constructed based on the processed dense point cloud data, and the 3D scene model is subjected to texture mapping processing to obtain a 3D scene model with texture information.

9. The intelligent view analysis method based on unmanned aerial vehicles according to claim 8, characterized in that, During the multi-view data acquisition process, the image data acquisition adopts an adaptive sampling strategy based on changes in image feature points, specifically including: During the multi-view data acquisition process of the UAV, image feature points are extracted from the continuously acquired video frames; The similarity between adjacent video frames is calculated based on the changes in the number and spatial distribution of image feature points in adjacent video frames. Based on the comparison result between the similarity and the preset threshold, the sampling interval of the video frames is adjusted according to the preset sampling control rules; Specifically, when the similarity is higher than the preset threshold, the sampling interval of the video frames is extended according to the preset sampling control rules; when the similarity is lower than the preset threshold, the sampling interval of the video frames is shortened according to the preset sampling control rules.

10. The intelligent view analysis method based on unmanned aerial vehicles according to claim 3, characterized in that, The method further includes: During the flight mission of the UAV, the currently collected video frame data and corresponding motion perception data are acquired, and image feature points are extracted based on the video frame data. The image feature points are then matched with image feature points in the feature point database to obtain feature point matching results. The feature point matching rate is calculated based on the feature point matching results. When the feature point matching rate is lower than the preset matching threshold, the corresponding scene area is determined as a candidate change area. Dense point cloud data of the current region is generated based on the video frame data corresponding to the candidate change region. The dense point cloud data is then compared with the three-dimensional spatial points of the corresponding region in the three-dimensional semantic map to ensure spatial consistency. At the same time, semantic consistency detection is performed by combining the semantic classification information in the three-dimensional semantic map to confirm the change of the candidate change region and determine the change type based on semantic differences. For the confirmed change area, the UAV is controlled to collect incremental data in the change area, and feature point extraction, matching and three-dimensional reconstruction processing are re-executed based on the newly added video frame data and the corresponding motion perception data to generate updated sparse point cloud, dense point cloud and corresponding three-dimensional mesh model. The updated 3D mesh model and the original 3D semantic map are geometrically and semantically fused within the corresponding scene area. The geometric fusion is based on the 3D spatial point alignment results, and the semantic fusion is based on the semantic classification results. The fusion results are verified based on the consistency of feature points, spatial points, and semantics before and after fusion, and an updated 3D semantic map is generated.

11. The intelligent view analysis method based on unmanned aerial vehicles according to claim 7, characterized in that, The method further includes: Based on the target identifier, real spatial coordinates and event type information in the structured event data, the real spatial coordinates are mapped to the corresponding spatial positions in the three-dimensional scene model, and the corresponding targets and events are labeled and displayed at the spatial positions to achieve spatial visualization of the events. When the event type in the event analysis result belongs to a preset alarm event, alarm information is generated based on the event type, event confidence level and real spatial coordinates in the event analysis result, and the alarm information is output. Event records are generated based on the structured event data and the event analysis results, and the event records are stored and managed. The event records include at least a target identifier, event type, real spatial coordinates, time information, and event confidence level. The time information is obtained based on the timestamps of the video frame data.

12. The intelligent view analysis method based on unmanned aerial vehicles according to any one of claims 1-11, characterized in that, The video frame data includes: Video stream data obtained by decoding and format conversion of the video data for real-time transmission; The first type of video frame data extracted from the video data based on a pre-set fixed time interval or fixed frame interval; The second type of video frame data is extracted from the video data based on an adaptive frame extraction strategy that adapts to changes in video content. The second type of video frame data is video frame data obtained through the following methods: Based on image feature points and corresponding feature descriptors in video frames, calculate the content similarity between adjacent video frames; When the content similarity is greater than or equal to the first preset threshold, the video data is extracted according to the first preset frame extraction interval. When the content similarity is less than or equal to the second preset threshold, the video data is frame-sampling according to the second preset frame-sampling interval. Wherein, the first preset frame extraction interval is greater than the second preset frame extraction interval.

13. An intelligent view analysis system based on unmanned aerial vehicles (UAVs), characterized in that, include: At least one drone; The intelligent view analysis platform includes: The data acquisition module is used to acquire multi-source data collected by the UAV during flight, including at least video data and motion sensing data. The view sampling module is used to process the video data to obtain video frame data; The spatial perception algorithm module is used to determine the spatial pose information of the UAV based on the video frame data and the motion perception data, combined with a pre-constructed three-dimensional semantic map. The video analysis algorithm module is used to perform target detection based on the video frame data, obtain the pixel coordinate information of the target, and convert the pixel coordinate information into the real spatial coordinates of the target by combining the spatial pose information; and to perform target tracking and behavior analysis based on the video frame data and the real spatial coordinates of the target, and generate structured event data containing spatial coordinates by combining scene semantic information. The large view model module is used to perform semantic verification based on the video frame data and the structured event data to obtain event analysis results; The 3D reconstruction module is used to control the UAV to perform multi-view data acquisition based on the spatial coordinates in the structured event data when the event analysis result is a preset event, and to generate a 3D scene model of the target area.

14. An intelligent view analysis platform, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the UAV-based intelligent view analysis method according to any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the intelligent view analysis method based on unmanned aerial vehicles as described in any one of claims 1-12.

Images