Real-time three-dimensional modeling system fused with unmanned aerial vehicle target recognition

Abstract

The invention relates to a real-time three-dimensional modeling system fused with unmanned aerial vehicle target recognition, and relates to the technical field of unmanned aerial vehicle application, and the system comprises a data collection module which is used for obtaining multi-source sensor data; the real-time modeling module is used for constructing a real-time three-dimensional model of the environment The target identification module is used for identifying a specific target in the scene and associating the specific target with the three-dimensional model. The information quality evaluation module is used for quantifying the observation information quality of the recognized target in real time according to factors such as the shielding degree, the resolution ratio and the observation angle; and when the information quality evaluation result is lower than a preset threshold value, the flight path planning module is triggered to autonomously plan a dynamic flight path going to the optimal viewpoint, and the flight control module executes the flight path and drives the unmanned aerial vehicle to perform supplementary data acquisition. According to the method, the autonomous closed loop of the perception evaluation decision action is established, so that active and intelligent fine modeling of the key target is realized, and the integrity of model information and the operation efficiency are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned aerial vehicle application technology, and in particular to a real-time three-dimensional modeling system fusing unmanned aerial vehicle target recognition. BACKGROUND

[0002] In recent years, with the rapid development of unmanned aerial vehicle platforms and sensor technologies, real-time three-dimensional modeling of the environment using unmanned aerial vehicles equipped with laser radars, visible light cameras and other sensors has become a key technology and has been widely used in fields such as surveying and mapping, building supervision, emergency rescue and security patrol.

[0003] The current mainstream unmanned aerial vehicle three-dimensional modeling operation mode is usually that an operator pre-plans a fixed flight path, such as a zigzag or grid-shaped flight path, covering a target area. The unmanned aerial vehicle passively collects data along the flight path, and the data is spliced into a three-dimensional model by an on-board processor or ground station software. However, this passive modeling method has inherent limitations.

[0004] This method lacks real-time feedback and intelligent adjustment capabilities for the modeling process and results. In actual operations, key targets of interest may be blocked by obstacles, or the pre-set flight path may be at an unfavorable observation angle, resulting in incomplete or low-quality data being collected.

[0005] To make up for this deficiency, current solutions rely heavily on the continuous monitoring and manual intervention of ground operators. Operators need to rely on their personal experience to manually take over the control of the unmanned aerial vehicle when they observe that the real-time returned images or the preliminary model have obvious defects, and to manipulate the unmanned aerial vehicle to fly to a viewpoint that can compensate for the missing data for supplementary collection. This method not only greatly increases the workload and skill requirements of the operator, but also has a lagging reaction and low efficiency, which destroys the autonomy of the unmanned aerial vehicle operation. In addition, due to the lack of quantitative evaluation means for model quality, the effect of manual intervention is difficult to guarantee, and often requires multiple attempts to obtain satisfactory results. SUMMARY

[0006] The purpose of the present application is to provide a real-time three-dimensional modeling system fusing unmanned aerial vehicle target recognition, which aims to solve the problem that when the unmanned aerial vehicle passively collects data along a fixed flight path, it cannot autonomously discover and compensate for missing model data and low quality of key targets caused by factors such as obstruction and poor observation angle.

[0007] In a first aspect, the real-time three-dimensional modeling system fusing unmanned aerial vehicle target recognition provided by the present application adopts the following technical solution: It comprises: a data acquisition module configured to acquire raw data of the flight environment of the unmanned aerial vehicle; a real-time modeling module, connected with the data acquisition module, configured to generate a real-time three-dimensional model of a three-dimensional scene in real time according to the raw data; a target recognition module, configured to recognize a preset target based on the raw data or the real-time three-dimensional model, and provide target semantic information for the target; an information quality evaluation module, connected with the real-time modeling module and the target recognition module, configured to evaluate observation information quality of the target based on the real-time three-dimensional model and a current observation viewpoint of the unmanned aerial vehicle; a flight path planning module, connected with the information quality evaluation module, configured to autonomously plan a dynamic flight path towards an optimal viewpoint when the observation information quality is lower than a preset information quality threshold; a flight control module, connected with the flight path planning module, configured to control the unmanned aerial vehicle to execute the dynamic flight path to perform supplementary data acquisition; a ground command module, configured to receive and visualize the real-time three-dimensional model in real time, and provide size measurement and coordinate extraction functions based on the real-time three-dimensional model; a change detection module, configured to detect target state changes in the scene by comparing the real-time three-dimensional models at different times and combining the target semantic information.

[0008] The data acquisition module includes a laser radar, a visible light camera, and a thermal infrared camera, and correspondingly, the real-time modeling module is specifically configured to: real-time solve a pose of the unmanned aerial vehicle, and fuse laser radar data collected by the laser radar, visible light data collected by the visible light camera, and thermal infrared data collected by the thermal infrared camera to generate the real-time three-dimensional model with texture and temperature information.

[0009] The information quality evaluation module calculates the observation information quality of the target by using a comprehensive information quality measurement function, which can be expressed as: ; wherein: is a quantitative value of the observation information quality of the target at the observation viewpoint ; is the target being evaluated; is the current observation viewpoint of the unmanned aerial vehicle; is a weight coefficient of an occlusion degree factor; is a weight coefficient of a resolution factor; is an observation angle factor, whose value is calculated according to an angle between a line of sight from an observation viewpoint is an occlusion degree factor, whose value is calculated by dividing a number of rays not occluded by the real-time three-dimensional model by a total number of rays; is calculated by dividing a number of rays not occluded by the real-time three-dimensional model by a total number of rays; is a resolution factor, whose value is calculated by a function , where is a distance between an observation viewpoint and a target , is a preset optimal observation distance, is a preset distance sensitivity parameter; is an observation angle factor, whose value is calculated according to an angle between a line of sight from an observation viewpoint to a target and a preset key normal vector of a target surface.

[0010] The path planning module, when planning the dynamic path, specifically operates as follows: generates a plurality of candidate viewpoints in a three-dimensional space with a target as a center; then, evaluates each candidate viewpoint by a utility function to determine the optimal viewpoint, which can be expressed as: ; wherein: is the optimal viewpoint selected; is a candidate viewpoint; is a set of candidate viewpoints; is a utility value of a candidate viewpoint ; is an expected information gain, whose value is determined by a difference between an expected observation information quality of the candidate viewpoint and a highest observation information quality currently obtained; is an action cost, whose value is determined according to a path length of the unmanned aerial vehicle from a current position to the candidate viewpoint , after the optimal viewpoint is determined, the path planning module plans a collision-free flight path from a current position of the unmanned aerial vehicle to the optimal viewpoint with the real-time three-dimensional model as an environment map, to form the dynamic path.

[0011] ​​The ground command module is further configured to receive an operation instruction of virtual path planning or virtual marker deployment of a user on the real-time three-dimensional model, and fuse and display the instruction information with the real-time three-dimensional model.

[0012] The flight path planning module is further configured to trigger planning of the dynamic flight path for supplementary data collection of the changed target area when the change detection module detects a preset target state change.

[0013] The ground command module is further configured to receive a specified target selected by a user on the real-time three-dimensional model, and transmit identification information of the specified target to the information quality assessment module to trigger assessment of the specified target.

[0014] In a second aspect, the application provides a real-time three-dimensional modeling method fusing target recognition of a UAV, which adopts the following technical scheme: comprising the following steps: Collecting original data of a UAV flight environment; Generating a real-time three-dimensional model of a three-dimensional scene in real time according to the original data; Identifying a preset target based on the original data or the real-time three-dimensional model, and providing target semantic information for the target; Assessing observation information quality of the target based on the real-time three-dimensional model and a current observation viewpoint of the UAV; When the observation information quality is lower than a preset information quality threshold, autonomously planning a dynamic flight path towards an optimal viewpoint; Controlling the UAV to execute the dynamic flight path for supplementary data collection; Real-time receiving and visualizing the real-time three-dimensional model, and providing size measurement and coordinate extraction functions based on the real-time three-dimensional model; Detecting target state changes in a scene by comparing the real-time three-dimensional models at different times and combining the target semantic information.

[0015] In summary, the application has the following at least one beneficial technical effect: 1. The application generates a three-dimensional model in real time according to the collected original data during the flight of the UAV, and transmits the model data to the ground command module in real time, which shortens the time delay between data collection and obtaining available three-dimensional scene information compared with the traditional offline post-processing mode after data collection, and improves the timeliness of on-site environment perception. 2.The application constructs a closed-loop feedback between observation information quality evaluation and dynamic flight path planning by setting information quality evaluation module and flight path planning module, when the information quality evaluation module determines that the observation information quality of the target is insufficient, the flight path planning module can autonomously generate and drive the unmanned aerial vehicle to perform a supplementary detection task to fly to the optimal viewpoint, this mechanism enables the system to actively obtain high-quality information of the target that is blocked or has poor observation angle, solves the problem of missing key information caused by passive collection along the preset flight line, and reduces the dependence on manual intervention flight; 3.The application can detect not only geometric changes in the scene, but also the category attributes of the target by comparing real-time three-dimensional models at different times by setting a change detection module and combining it with the target semantic information provided by the target recognition module, therefore, the system can output event information with context such as displacement of a specific type of vehicle or appearance of a new target in a specific area, instead of indiscriminate point cloud data changes, thereby enhancing the analysis capability of the scene dynamic event. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is the system architecture diagram of the application; Figure 2 is the method flowchart of the application; Figure 3 is a hardware structure schematic diagram of the data acquisition module of the application; Figure 4 is a target recognition and three-dimensional association schematic diagram of the application.

[0017] Mark explanation: 10, data acquisition module; 20, real-time modeling module; 30, target recognition module; 40, information quality evaluation module; 50, flight path planning module; 60, flight control module; 70, ground command module; 80, change detection module. DETAILED DESCRIPTION

[0018] The following will be combined with the Figure 1 -Appendix Figure 4 , the application is further explained in detail.

[0019] Embodiment 1: A real-time three-dimensional modeling system integrating unmanned aerial vehicle target recognition, referring to Figure 1 , Figure 3 and Figure 4 , including: The data acquisition module 10 is the perception front end of the device, which is installed on the unmanned aerial vehicle platform, in the embodiment, the data acquisition module 10 integrates multiple types of sensors, including laser radar, visible light camera, thermal infrared camera and inertial measurement unit, these sensors are rigidly fixed on the body of the unmanned aerial vehicle.

[0020] The laser radar is used to acquire three-dimensional spatial coordinate data of the environment around the unmanned aerial vehicle. In each scanning cycle, the laser radar emits a laser pulse and receives a return signal. The distance and orientation of a large number of discrete points relative to the coordinate system of the sensor itself are calculated by measuring the time of flight of the signal, so as to generate a set of three-dimensional coordinate points, i.e. a point cloud, each point of which can also be attached with reflection intensity information.

[0021] The visible light camera is used to collect high-resolution two-dimensional color image data. In an embodiment, in order to reduce image distortion caused by high-speed motion of the unmanned aerial vehicle, the visible light camera adopts a global shutter type, which ensures that all pixels of the image sensor are exposed at the same time in a single exposure moment, so as to obtain clear and non-smear scene texture information.

[0022] The thermal infrared camera is used to collect thermal radiation information of the environment and target surface. It converts the detected infrared radiation of different intensities into a two-dimensional gray scale image. The gray scale value of each pixel in the image corresponds to a temperature value or radiation intensity, thereby forming a temperature distribution map of the scene.

[0023] The inertial measurement unit contains a gyroscope and an accelerometer, which are used to measure three-dimensional angular velocity and three-dimensional linear acceleration in the body coordinate system of the unmanned aerial vehicle at a high frequency. These inertial measurement data are the basis for subsequent high-precision pose calculation.

[0024] In order to realize effective fusion of multi-source heterogeneous data, the data acquisition module 10 must ensure strict alignment of the sensor data in time. In this embodiment, a hardware time synchronization scheme is adopted. The global positioning system module provides a unified second pulse signal for the laser radar, the visible light camera, the thermal infrared camera and the IMU. Each sensor takes the high-precision pulse signal as a reference, and adds a high-precision time stamp under the unified time source to each frame of data collected by it.

[0025] After the collection and time stamp addition are completed, the data acquisition module 10 packs the data from different sensors into a synchronous data set. For any time stamp , the data acquisition module 10 outputs a data set , wherein is the laser radar point cloud, is the visible light image, is the thermal infrared image, is a set of inertial measurement unit measurement values near the time stamp. These synchronous data packets with accurate time stamps are transmitted to the real-time modeling module 20 for subsequent processing. In addition, the external parameters of each sensor relative to the body coordinate system of the unmanned aerial vehicle, i.e. the conversion relationship of spatial position and attitude, have been accurately determined through a pre-calibration process, and are provided to the real-time modeling module 20 as fixed parameters.

[0026] ; in: To obtain the optimal state trajectory; The set of state variables to be optimized; , , These are the index sets for inertial measurement unit measurements, lidar measurements, and visible light image measurements within a sliding window, respectively. It is based on the measurement value of the inertial measurement unit. The residual term between the state transition constraints obtained from pre-integration and the optimization variables; It is the point cloud of the current frame. According to position Projected onto global map In this context, the calculated distance residuals from points to the map plane or edge are used. It is to determine the 3D spatial feature points FF observed in historical frames based on their pose. Reprojected onto the current image The above is the pixel residual between the calculated reprojection position and the actual observation position; , , These are the covariance matrices corresponding to the residuals of each term, used to characterize the measurement uncertainty. This optimization problem is iteratively solved on the onboard computing unit, resulting in a smooth and high-frequency output of the UAV pose, i.e., from the body coordinate system. To the world coordinate system Transformation matrix .

[0027] In terms of model generation, the real-time modeling module 20 utilizes the calculated high-precision pose. Unify multi-source data into a global world coordinate system Below, for the lidar in its own coordinate system Any three-dimensional point collected below Its coordinates in the world coordinate system Calculated using the following formula: ; in, It is the external parameter transformation matrix of the lidar relative to the UAV body, which is obtained through pre-calibration.

[0028] To assign texture color and temperature information to 3D points, the real-time modeling module 20 converts the points in the world coordinate system... The image is projected back onto the image planes of the visible light camera and the thermal infrared camera, which are synchronized with it in time. Taking the visible light camera as an example, the projection process includes: The first step is to set the world coordinates. Transform to visible light camera coordinate system Next, get the point : ; wherein, is the exterior parameter transformation matrix of the visible light camera relative to the UAV body.

[0029] Second, use the pinhole model of the camera to project to get the pixel coordinates of the point on the image : ; wherein, is the internal parameter matrix of the visible light camera.

[0030] If the calculated pixel coordinates are located within the valid range of the image, the RGB color value of the pixel point is extracted, and the same process is used to project the same point onto the thermal infrared image using the internal and external parameters of the thermal infrared camera, and the corresponding temperature or grayscale value is extracted.

[0031] Finally, the real-time modeling module 20 associates the spatial coordinates, color values, and temperature values of each three-dimensional point, and organizes these point sets with rich attributes into a global three-dimensional model. The model can be stored and managed using data structures such as voxel grid or octree, to facilitate efficient updating, querying, and transmission. This model is the real-time three-dimensional model provided to other modules.

[0032] The target recognition module 30 is responsible for detecting and recognizing pre-set targets from the raw data obtained by the data acquisition module 10 or the model generated by the real-time modeling module 20, and assigning unique identity and category attributes to these targets, i.e., outputting target semantic information.

[0033] In an embodiment, the target recognition module 30 mainly performs target detection based on the two-dimensional color images collected by the visible light camera. The target recognition module 30 internally deploys a pre-trained deep learning target detection model, which is trained to recognize specific categories of targets such as vehicles, people, buildings, etc.

[0034] For each input image , the target recognition module 30 performs the following operations: first, input the image into the target detection model, and the model calculates the forward propagation to directly output a series of detection results, each detection result containing the following information: ; wherein: is the two-dimensional bounding box of the target, , is the pixel coordinate of the center point of the bounding box, , is the width and height of the bounding box; is the recognized target class label; is the confidence score of the detection result, the target recognition module 30 will first filter out the detection results with confidence scores lower than the preset threshold, to reduce false positives.

[0035] Next, the target recognition module 30 needs to map the detection results on the two-dimensional image to the three-dimensional space. To this end, it obtains from the real-time modeling module 20 the real-time three-dimensional model strictly time-synchronized with the current image and the corresponding UAV pose .

[0036] For each valid two-dimensional detection bounding box , the target recognition module 30 performs a two-dimensional to three-dimensional association process.

[0037] Specifically, it finds the corresponding point cloud subset in the three-dimensional model for all pixel points within the bounding box region using the reverse lookup of the projection relationship provided by the real-time modeling module 20 This process can be formalized as: traversing each pixel within the bounding box and combining the depth information to project it back to the world coordinate system , thereby filtering out the point cloud cluster belonging to the target.

[0038] After obtaining the point cloud subset of the target , the target recognition module 30 assigns it a unique instance ID. If it is a newly appearing target, the system generates a new ID; if the target has been detected in previous frames, the system tracks it through data association algorithms and assigns it the same ID to ensure the continuity of the target in time sequence.

[0039] Finally, the target recognition module 30 outputs a data structure containing complete semantic information for each recognized and located target : ; Wherein: is the unique instance ID of the target; is the class label of the target; is the three-dimensional point cloud subset constituting the target; is the geometric center point coordinate of the target in the world coordinate system , which can be obtained by calculating the centroid of the point cloud subset .

[0040] These target data containing rich semantic information Subsequently, the information quality assessment module 40 and the change detection module 80 are transmitted as the basis for their respective functions to perform.

[0041] The information quality assessment module 40 is connected with the target recognition module 30 and the real-time modeling module 20, and its function is to receive the specific target information provided by the target recognition module 30, and combine the real-time three-dimensional model provided by the real-time modeling module 20 and the UAV pose, to quantitatively assess the observation information quality of the target under the current observation viewpoint.

[0042] When the information quality assessment module 40 receives the assessment instruction about the target , it first obtains the complete semantic information of the target from the target recognition module 30 , and obtains the current global real-time three-dimensional model and the UAV pose from the real-time modeling module 20, the position of the UAV in the world coordinate system is determined as the current observation viewpoint .

[0043] Subsequently, the information quality assessment module 40 calculates the quantitative value of the observation information quality through a comprehensive information quality measurement function . The function is composed of multiple weighted factors: ; Wherein: is the final quantitative value of the observation information quality of the target under the observation viewpoint ; , , are preset weight coefficients for adjusting the importance of each factor, and satisfy ; is the occlusion degree factor; is the resolution factor; is the observation angle factor.

[0044] The calculation of the occlusion degree factor uses ray tracing method, first, uniformly sample three-dimensional points from the three-dimensional point cloud subset of the target, then, for each sampling point, emit a virtual light from the observation viewpoint to the point, and the information quality assessment module 40 uses the global real-time three-dimensional model ​As an obstacle map, it detects whether each ray of light collides with other objects before reaching the target sampling point, and counts the number of light rays that do not collide. The occlusion factor is calculated using the following formula: ; resolution factor It is used to measure the suitability of the observation distance, and its calculation is based on a preset optimal observation distance. The Gaussian function model centered on the observation viewpoint is first calculated. With the target center point Euclidean distance between The resolution factor is calculated using the following formula: ; in, This is a preset distance sensitivity parameter that controls the rate at which the factor value decays as the distance deviation increases.

[0045] Observation angle factor Used to measure the effectiveness of the current observation direction in revealing key features of the target, for a specific target category. A set of key surface normal vectors is predefined. These normal vectors represent the orientation of the target surface most desired to be observed, such as the side of a vehicle. First, the orientation from the target center is calculated. Pointing to the observation viewpoint Normalized line-of-sight vector Then, calculate the angle between the line-of-sight vector and the normal vector of each key surface. The observation angle factor takes the maximum value among all angle evaluations, and the evaluation function maximizes the value near the 90-degree angle (directly facing the surface). ; After calculating the final quantitative value of the observation information quality Then, the information quality assessment module 40 compares it with a preset information quality threshold. If a comparison is made, If the current observation information quality is insufficient, the information quality assessment module 40 will generate a trigger signal containing the target ID and send it to the trajectory planning module 50.

[0046] The trajectory planning module 50 is connected to the information quality assessment module 40. When it receives information containing a specific target from the information quality assessment module 40... When the information triggers, the trajectory planning module 50 is activated to autonomously plan a dynamic trajectory for supplementing data collection. The planning process is divided into two main stages: optimal viewpoint determination and collision-free path generation.

[0047] In the optimal viewpoint determination phase, the goal of the trajectory planning module 50 is to find the observation position that is expected to yield the highest information gain. This process begins by generating a set of candidate viewpoints. In this embodiment, the target to be observed is... center point Centered on the sphere, at the optimal observation distance defined in Information Quality Assessment Module 40. Using a radius of 1, samples are taken on a 3D sphere to generate multiple candidate viewpoints. .

[0048] Subsequently, the trajectory planning module 50 uses the utility function For sets Each candidate viewpoint The utility function is defined as the ratio of expected information gain to action cost. ; in: For the expected information gain, this value is determined by the candidate viewpoint. The quality of the observation information of the target is pre-calculated and then subtracted from the highest quality of the observation information of the target that has been obtained so far. The pre-calculation process calls the same function as the information quality assessment module 40. Among them, the occlusion factor The calculation is based on the existing global real-time 3D model. Ray tracing simulations were performed.

[0049] For the cost of the operation, this value is based on the drone's distance from its current location. Fly to candidate viewpoint The Euclidean distance is used to determine this, i.e. ,in Candidate viewpoints The location coordinates.

[0050] The trajectory planning module iterates through all candidate viewpoints, calculates their respective utility values, and selects the candidate viewpoint that maximizes the utility function as the optimal viewpoint. : ; During the collision-free path generation phase, the task of the trajectory planning module 50 is to generate a path from the UAV's current position to the determined optimal viewpoint. The flight trajectory is determined by the trajectory planning module 50, which obtains the latest global real-time 3D model from the real-time modeling module 20. It is used as an environment map for collision detection.

[0051] This embodiment uses the RRT* algorithm to plan the path. This algorithm starts from the current state of the UAV and uses the optimal viewpoint. For the target point, the algorithm randomly samples in 3D space and expands a path tree in the direction of the sampled point. During each expansion, a real-time 3D model is utilized. Collision detection is performed to ensure that each edge of the tree represents a collision-free flight path. The RRT* algorithm also includes a path optimization process, which involves continuously reconnecting the nodes of the tree during the expansion process to find a shorter path.

[0052] Once the algorithm finds a collision-free path connecting the starting point and the target point, the trajectory planning module 50 parameterizes this path, which consists of a series of three-dimensional waypoints, in time to form a smooth dynamic trajectory that includes position, velocity, and acceleration constraints. This dynamic trajectory is then sent to the flight control module 60 for execution.

[0053] The flight control module 60 is connected to the trajectory planning module 50. Its function is to convert the high-level dynamic trajectory output by the trajectory planning module 50 into low-level control commands that can be executed by the UAV's underlying flight controller.

[0054] A dynamic trajectory is a time-parameterized path that includes the desired state at any given moment. The trajectory defines the desired state vector. ,in , , These are the expected position, expected velocity, and expected acceleration of the drone at that moment, all in the global world coordinate system. The following indicates.

[0055] The flight control module 60 employs a cascaded controller architecture to achieve precise tracking of the desired trajectory. This controller obtains the current actual state of the UAV, including its real-time position, from the real-time modeling module 20. and real-time speed .

[0056] The outer loop of the controller is a position controller. It calculates the error vector between the desired position and the actual position: ; This positional error The input to the PID controller is added to the feedforward term of the desired speed to generate a commanded speed vector. : ; in, , , These are the three-dimensional proportional, integral, and differential gain matrices of the position control loop, respectively.

[0057] The inner loop of the controller is the speed controller. It calculates the error vector between the commanded speed generated by the outer loop and the actual speed of the UAV. ; This speed error The input is fed into another PID controller, whose output is added to the feedforward term of the desired acceleration and the gravity compensation term to generate the total thrust command vector required by the UAV. : ; in, It's about the quality of the drone; , , These are the three-dimensional proportional, integral, and differential gain matrices of the speed control loop, respectively. It is the vector of gravitational acceleration.

[0058] Finally, the flight control module 60 will calculate the thrust command vector. This is broken down into commands that the drone's underlying flight controller can directly execute; specifically, the total throttle command. Set as The magnitude of the component along the Z-axis of the UAV body, and the desired attitude (roll angle). and pitch angle )according to The direction was calculated to tilt the drone's body, thus aligning the main thrust direction with... Orientation alignment, desired yaw angle It can be set according to the needs of the task, for example, to align it with the tangent direction of the trajectory.

[0059] The final instruction set The data is sent at high frequency to the drone's underlying flight controller, which is responsible for the specific motor output distribution and attitude stabilization, thereby driving the drone to fly precisely along the planned dynamic trajectory.

[0060] The ground command module 70 is a user interaction terminal deployed at the ground station. It communicates bidirectionally with the airborne equipment via a wireless data link. The core function of the ground command module 70 is to receive, process, and visualize the real-time 3D model generated by the real-time modeling module 20, and to provide ground operators with a series of interactive functions based on the model.

[0061] In order to realize the visualization of real-time three-dimensional model, the ground command module 70 first receives the data stream from the real-time modeling module 20 through the wireless data link, in order to reduce the transmission bandwidth, the data stream adopts the incremental update mode, that is, after the first transmission of the complete global three-dimensional model, only the changed or newly added model part is transmitted subsequently, in this embodiment, the incremental data is encapsulated and transmitted in the form of change node of octree structure.

[0062] The ground command module 70 is built-in three-dimensional rendering engine, which analyzes the model data after receiving the data, and integrates the incremental data into the locally stored global three-dimensional model, then, using the graphics processing unit, the updated complete three-dimensional model is rendered into a two-dimensional image observable by the user in real time, and displayed on the screen, the user can perform translation, scaling and rotation operations on the virtual viewpoint of the three-dimensional scene through the mouse, keyboard or touch screen and other input devices, and can observe the site environment from any angle.

[0063] The ground command module 70 provides size measurement function, when the user activates this function, the module allows the user to select two points in the rendered three-dimensional scene in turn, for each selection of the user, the module calculates the intersection of the pixel corresponding to the line of sight and the surface of the three-dimensional model through the inverse projection transformation of the two-dimensional pixel coordinates on the screen, so as to obtain the three-dimensional coordinates of the point in the world coordinate system And , then the module calculates the Euclidean distance between the two three-dimensional coordinate points, and displays the calculation result to the user.

[0064] The ground command module 70 provides coordinate extraction function, when the user activates this function and selects any point on the model, the module uses the same inverse projection transformation method as size measurement to obtain the three-dimensional coordinates of the point in the world coordinate system , and directly presents the numerical value to the user.

[0065] The ground command module 70 also provides the function of virtual information fusion display, the user can draw virtual path or place virtual marker on the visualized three-dimensional model, when the user performs drawing operation on the screen, the ground command module 70 converts the series of two-dimensional points input by the user into a polyline or a group of points in three-dimensional space, these virtual geometric information is stored and rendered together with the real-time three-dimensional model, and is displayed in a semi-transparent or other way, so as to realize the scene plotting of virtual and real fusion.

[0066] ​​In addition, the ground command module 70 supports the evaluation of user-specified targets. Users can directly click on the target identified by the target recognition module 30 on the visual interface. The ground command module 70 captures the click event, determines the selected target instance ID, and sends the target ID back to the UAV via the uplink data link. After receiving this ID, the airborne information quality evaluation module 40 triggers the evaluation of the observation information quality of the specified target.

[0067] The change detection module 80 is connected to the real-time modeling module 20 and the target recognition module 30. Its function is to detect and report the state changes of targets in the scene by comparing real-time 3D models at different times and combining the target semantic information provided by the target recognition module 30.

[0068] In a specific embodiment, the change detection module 80 operates within a fixed global three-dimensional voxel grid covering the entire work area. This change detection module 80 continuously receives the latest real-time three-dimensional model from the real-time modeling module 20 and uses it to update the state of the voxel grid, with each voxel in the grid... Having a state This state can be occupied, idle, or unknown.

[0069] The change detection module 80 will display the current time. voxel grid state Reference time voxel grid state The module performs an image-by-image comparison. Through this comparison, it identifies two basic types of variation: The set of voxels whose state changes from idle to occupied is denoted as . .

[0070] The set of voxels whose state changes from occupied to free is denoted as . .

[0071] Performing only geometric comparisons will generate a large amount of raw change data, but lack semantic content. Therefore, the change detection module 80 further performs a semantic association step. The change detection module 80 obtains information on all identified targets at the current time from the target recognition module 30, including the instance ID, category, and corresponding 3D point cloud subset of each target.

[0072] For a detected voxel cluster, the change detection module 80 calculates its spatial bounding box and checks whether the bounding box spatially overlaps with any newly identified target point cloud subset at the current moment. If there is an overlap, the geometric change is associated with the corresponding target, a target occurrence event is generated, and the instance ID and category of the target are recorded.

[0073] For a detected disappearing voxel cluster, the change detection module 80 checks whether its location overlaps with the spatial location of a certain known target at the reference time , and if so, preliminarily determines that the target has undergone a state change.

[0074] Further, the change detection module 80 identifies a moving event of a target by correlating disappearing and appearing events, if a target with a certain instance ID has its voxel state changed from occupied to free at the reference time , while at the current time , a target with the same instance ID is identified at a new location, and the voxel state corresponding to the new location is changed from free to occupied, the change detection module 80 merges the two events and generates a moving event of the target.

[0075] Finally, the change detection module 80 generates a structured event record for each detected and semantically correlated change, which contains the event type, the instance ID and category of the related target, the geographic coordinates where the event occurred, and the timestamp, and these event records are transmitted to the ground command module 70 for highlighting display on the three-dimensional model and issuing a prompt to the operator. In some configurations, the change detection module 80 can also directly send a signal to the flight path planning module 50 to trigger autonomous close-in observation of the changed area when a change of a certain type or in a certain area is detected.

[0076] Embodiment 2: A real-time three-dimensional modeling method integrating unmanned aerial vehicle target recognition, referring to Figure 2 , comprising the following steps: Collecting raw data of the unmanned aerial vehicle flight environment; Generating a real-time three-dimensional model of the three-dimensional scene in real time according to the raw data; Identifying a preset target based on the raw data or the real-time three-dimensional model, and providing target semantic information for the target; Evaluating the quality of observation information of the target based on the real-time three-dimensional model and the current observation viewpoint of the unmanned aerial vehicle; When the quality of observation information is lower than a preset information quality threshold, autonomously planning a dynamic flight path towards an optimal viewpoint; Controlling the unmanned aerial vehicle to execute the dynamic flight path for supplementary data collection; Receiving and visualizing the real-time three-dimensional model in real time, and providing size measurement and coordinate extraction functions based on the real-time three-dimensional model; Detecting target state changes in the scene by comparing real-time three-dimensional models at different times and combining target semantic information.

Claims

1. A real-time three-dimensional modeling system fused with unmanned aerial vehicle target recognition, characterized in that, The system comprises: a data acquisition module configured to acquire raw data of a flight environment of a UAV; a real-time modeling module connected with the data acquisition module and configured to generate a real-time three-dimensional model of a three-dimensional scene in real time according to the raw data; a target recognition module configured to recognize a preset target based on the raw data or the real-time three-dimensional model and provide target semantic information for the target; an information quality assessment module connected with the real-time modeling module and the target recognition module and configured to assess observation information quality of the target based on the real-time three-dimensional model and a current observation viewpoint of the UAV; a flight path planning module connected with the information quality assessment module and configured to autonomously plan a dynamic flight path toward an optimal viewpoint when the observation information quality is lower than a preset information quality threshold; a flight control module connected with the flight path planning module and configured to control the UAV to execute the dynamic flight path to perform supplementary data acquisition; a ground command module configured to receive and visualize the real-time three-dimensional model in real time and provide size measurement and coordinate extraction functions based on the real-time three-dimensional model; a change detection module configured to detect a target state change in a scene by comparing the real-time three-dimensional models at different times and combining the target semantic information.

2. The real-time three-dimensional modeling system fused with target recognition of unmanned aerial vehicle according to claim 1, characterized in that, The data acquisition module comprises a laser radar and a visible light camera. The real-time modeling module is specifically configured to: real-time solve a pose of the UAV, and fuse laser radar data acquired by the laser radar and visible light data acquired by the visible light camera to generate the real-time three-dimensional model with texture.

3. The real-time three-dimensional modeling system fused with UAV target recognition according to claim 1, characterized in that, The information quality assessment module is specifically configured to: calculate at least one of an occlusion degree, an observation resolution, and an observation angle of the target relative to the observation viewpoint based on the real-time three-dimensional model to determine the observation information quality of the target.

4. The real-time three-dimensional modeling system fusing target recognition of a UAV according to claim 1, wherein The flight path planning module is specifically configured to: generate a plurality of candidate viewpoints in a three-dimensional space; determine the optimal viewpoint according to an expected information gain of the candidate viewpoints and an action cost of flying to the candidate viewpoints.

5. The real-time three-dimensional modeling system fused with UAV target recognition according to claim 1, wherein, The flight path planning module is further configured to: plan a collision-free flight path from a current position of the UAV to the optimal viewpoint using the real-time three-dimensional model as an environment map to form the dynamic flight path.

6. The real-time three-dimensional modeling system fused with UAV target recognition according to claim 1, wherein, The data acquisition module further comprises a thermal infrared camera. The real-time modeling module is further configured to fuse thermal infrared data acquired by the thermal infrared camera onto the real-time three-dimensional model to generate the real-time three-dimensional model with temperature information.

7. The real-time three-dimensional modeling system for unmanned aerial vehicle target recognition fusion according to claim 1, wherein, The ground command module is further configured to: allow a user to perform virtual path planning or deploy virtual markers on the real-time three-dimensional model to realize virtual-real fusion interaction.

8. The real-time three-dimensional modeling system that fuses UAV target identification according to claim 1, wherein, The flight path planning module is further configured to: trigger planning of the dynamic flight path for supplementary data acquisition of a target area where a change occurs when the change detection module detects a preset target state change.

9. The real-time three-dimensional modeling system fused with UAV target recognition of claim 1, wherein, The ground command module is further configured to: Receiving a specified target selected by a user on the real-time three-dimensional model and triggering the information quality assessment module to assess the specified target.

10. A real-time three-dimensional modeling method of fusing unmanned aerial vehicle target recognition, according to the real-time three-dimensional modeling system of fusing unmanned aerial vehicle target recognition of any one of claims 1-9, characterized in that, The method comprises the following steps: Collecting original data of the flight environment of the UAV; Generating a real-time three-dimensional model of a three-dimensional scene in real time according to the original data; Identifying a preset target based on the original data or the real-time three-dimensional model and providing target semantic information for the target; Assessing the observation information quality of the target based on the real-time three-dimensional model and the current observation viewpoint of the UAV; When the observation information quality is lower than a preset information quality threshold, autonomously planning a dynamic flight path towards an optimal viewpoint; Controlling the UAV to execute the dynamic flight path to collect supplementary data; Receiving and visualizing the real-time three-dimensional model in real time and providing size measurement and coordinate extraction functions based on the real-time three-dimensional model; Detecting the state change of a target in a scene by comparing the real-time three-dimensional models at different time points and combining the target semantic information.

Images