A safety target tracking and early warning method applied to a river bank area

Abstract

The application discloses the technical field of intelligent inspection and safety monitoring, and relates to a safety target tracking and early warning method applied to a river shoreline area. The method synchronously collects laser point cloud data and aerial survey image data of the river shoreline area through a UAV cluster, and constructs a three-dimensional semantic environment model integrating texture, elevation and semantic information. An RL-QPSO algorithm is adopted for single-UAV flight path planning. An improved dragonfly algorithm is adopted to realize multi-UAV cooperative task allocation and dynamic flight path optimization. An improved YOLOv8 algorithm is used for real-time image data target recognition, and the detection result containing abnormal target information is outputted, task re-allocation, path re-planning and risk assessment are performed. The method integrates environment modeling, target recognition and intelligent path planning algorithms, and constructs a complete closed loop from perception, path planning, multi-UAV cooperation to target recognition, thereby significantly improving the intelligent level, early warning response speed and overall execution efficiency of shoreline safety inspection.

Description

Technical Field

[0001] This invention relates to the field of intelligent inspection and safety monitoring technology, and in particular to a method for tracking and early warning of safety targets applied to river shoreline areas. Background Technology

[0002] Riverbanks, serving as crucial ecological barriers and economic activity zones at the junction of land and water, face diverse security risks, including illegal sand mining, unauthorized vessel mooring, debris blocking waterways, and damage to shoreline facilities. Traditional riverbank monitoring methods primarily rely on manual patrols, fixed camera surveillance, and single-drone inspections, all of which have significant shortcomings. Manual patrols are limited by weather, terrain, and manpower, resulting in low efficiency, limited coverage, and difficulty in achieving continuous, all-weather, and all-area monitoring. They also suffer from delayed response to dangerous events. Fixed camera surveillance has numerous blind spots, is particularly costly to deploy in narrow riverbank areas, is susceptible to interference from lighting and fog, has low target recognition accuracy, and lacks dynamic tracking capabilities. Single-drone patrols have low levels of intelligence, mostly relying on preset flight paths, lacking real-time environmental awareness and dynamic adjustment capabilities. They are prone to collisions in complex riverbank environments, and the limited coverage of a single drone makes it difficult to meet the inspection needs of large river areas.

[0003] In recent years, although multi-UAV collaborative inspection technology has made some progress, existing solutions still have many shortcomings: In terms of environmental perception, they lack high-precision 3D semantic environment modeling capabilities, failing to provide complete environmental information for path planning; in terms of path planning, traditional algorithms are prone to getting trapped in local optima and struggle to cope with dynamically changing river environments; in terms of multi-UAV collaboration, most adopt a centralized control architecture, resulting in poor robustness, and a failure of the central node will paralyze the entire system; in terms of target recognition, existing algorithms are not adaptable to complex riverbank scenarios, exhibiting low detection accuracy and poor robustness for multi-scale, arbitrarily oriented vessels and floating objects. Therefore, there is an urgent need for a fully automated riverbank safety early warning system capable of intelligent perception, accurate recognition, autonomous planning, and multi-UAV collaboration. Summary of the Invention

[0004] This application provides a safety target tracking and early warning method for river shoreline areas to solve the problems of low efficiency, poor real-time performance, weak environmental adaptability, insufficient target recognition accuracy, and lack of multi-machine collaboration capabilities in traditional shoreline monitoring methods, thereby significantly improving the intelligence level, early warning response speed, and overall execution efficiency of shoreline safety inspection.

[0005] This application provides a method for tracking and early warning of safety targets in riverbank areas, including the following steps: S1: Simultaneously collect laser point cloud data and aerial survey image data of the river shoreline area through a cluster of drones to construct a three-dimensional semantic environment model that integrates texture, elevation and semantic information; S2: Based on a three-dimensional semantic environment model, the RL-QPSO Net algorithm is used for single UAV trajectory planning; S3: Based on UAV trajectory planning, an improved Dragonfly algorithm is used to achieve multi-UAV collaborative task allocation and dynamic trajectory optimization; S4: Use the improved YOLOv8 algorithm to perform target recognition on real-time image data collected by the UAV, output detection results containing abnormal target information, and perform task reassignment, path replanning and risk assessment accordingly. S5: Generate early warning information based on the risk assessment results and send it to the monitoring center and terminals.

[0006] The technical advantages of the above embodiments are as follows: The present invention uses a cluster of drones equipped with multimodal sensors to integrate environmental modeling, target recognition and intelligent path planning algorithms to construct a complete closed-loop system from three-dimensional environmental perception, global path planning, multi-drone collaborative optimization to target recognition and early warning. This enables fully automated operation of the entire process of river shoreline area from three-dimensional environmental perception, accurate multi-target recognition to multi-drone autonomous collaborative tracking, thereby significantly improving the intelligence level of shoreline safety inspection, early warning response speed and overall execution efficiency.

[0007] Based on the above embodiments, this application can be further improved as follows: In one embodiment of this application, the specific process of constructing the three-dimensional semantic environment model in step S1 is as follows: S1.1. Noise is filtered out from the collected laser point cloud data. A cloth simulation filtering algorithm is used to separate ground points from non-ground points. The ground points are extracted by simulating the gravity settling process of the cloth on the flipped point cloud. S1.2. For the missing point cloud areas caused by the removal of non-ground points and water reflection, spatial inverse distance weighted interpolation is used to fill and smooth the elevation grid to generate a digital elevation model; S1.3. Perform aerial triangulation of aerial survey images without field control based on point cloud feature points, and generate orthophotos by combining with digital elevation models; S1.4. The laser point cloud and orthophoto are registered and fused through affine transformation to construct a three-dimensional semantic environment model containing fused texture, elevation and semantic information. Technical results: High-precision 3D environment reconstruction without field control points was achieved, effectively solving the problem of missing point clouds in water areas. The generated 3D semantic model provides an accurate environmental basis for subsequent path planning and target localization.

[0008] In one embodiment of this application, the process of using the RL-QPSO Net algorithm for trajectory planning in step S2 is as follows: S2.1. Model the global path planning problem as a dynamic optimization problem in a high-dimensional space, establish a UAV dynamic model that includes obstacle avoidance constraints, dynamic constraints and boundary constraints, and set the objective function to minimize flight distance, control cost and collision risk; S2.2. Integrate the gated loop unit into the particle update process of quantum behavior particle swarm optimization, and use the hidden state encoding of GRU to encode the particle history search experience and environmental interaction context to generate adaptive weights to adjust the particle search step size and direction. S2.3. A deep reinforcement learning module employing a dual-objective reward function mechanism adjusts the quantum potential well length and adaptive weight parameters of quantum behavior particle swarm optimization in real time based on environmental feedback, thereby achieving global trajectory dynamic planning; S2.4. When the reward function tends to stabilize or reaches the maximum number of iterations, output the optimal flight path of the UAV. Technical effects: It integrates the global search capability of quantum behavior particle swarm optimization with the real-time decision-making capability of deep reinforcement learning, effectively avoiding path planning from getting stuck in local optima, while improving the algorithm's adaptability to dynamic environments. The generated path has both global optimality and real-time responsiveness.

[0009] In one embodiment of this application, the dual-objective reward function in step S2 simultaneously includes a target approach reward term and an obstacle avoidance penalty term, and its specific expression is as follows: ; In the formula, the first term Drive the drone to the target location Closer, the second item Approaching obstacles by drones The behavior is subject to exponential penalties.

[0010] Technical effect: By designing a dual-objective reward function, a balance between path efficiency and safety is achieved, ensuring that the drone maintains a sufficient safe distance from obstacles while rapidly approaching the target.

[0011] In one embodiment of this application, the specific process of using the improved Dragonfly algorithm to achieve multi-UAV cooperative task allocation and dynamic trajectory optimization in step S3 is as follows: Treating each drone as an individual dragonfly, the speed adjustment of the drone is synthesized based on three improvement rules: predictive separation behavior, target tendency and dynamic path feedback behavior, and context-aware aggregation behavior. When multiple targets to be inspected are identified, each target is regarded as a virtual gravitational source, the comprehensive attraction of each drone to each target is calculated, and the task allocation of multiple drones is completed through a distributed negotiation mechanism. Based on the assigned mission objectives and the synthesized speed adjustment, real-time flight commands are generated for each UAV and input into the underlying flight control system for execution. Technical effects: It realizes distributed collaborative decision-making of multiple UAVs, and can complete task allocation and obstacle avoidance without central control, which improves the robustness and response speed of the system. The improved behavior rules enable UAVs to avoid dynamic obstacles in advance while maintaining formation and advancing towards the mission area.

[0012] In one embodiment of this application, the calculation of the comprehensive attractiveness includes three parts: basic distance attractiveness, DRL-based value prior, and power loss penalty, specifically expressed as follows: . Technical effect: It comprehensively considers factors such as distance, target priority, path value and remaining drone battery power to ensure the fairness and efficiency of task allocation, avoid drones with insufficient battery power from undertaking remote tasks, and prioritize the allocation of high-priority targets to the most suitable drones.

[0013] In one embodiment of this application, the specific structure of the improved YOLOv8 algorithm in step S4 is as follows: Efficient multi-scale attention modules are added to the three C2 modules of the Neck part of YOLOv8. These modules capture contextual information under different receptive fields through a parallel multi-branch structure and adaptively calculate the weights of feature maps in the spatial and channel dimensions. A rotation angle prediction branch is added to the detection head of YOLOv8, and the loss function is optimized by adding an angle regression loss term to improve the detection accuracy of targets facing any direction. The mean precision and F1-Score are used as the core evaluation metrics for model training to ensure the model's detection accuracy and generalization performance. Technical effects: It enhances the model's ability to extract features from multi-scale targets and improves detection accuracy in scenarios with complex lighting, target scale and rotation changes. In particular, for the characteristics of ships with unpredictable orientation in river shoreline areas, the rotation angle prediction branch significantly improves the localization accuracy of targets with arbitrary orientations.

[0014] In one embodiment of this application, the specific process of abnormal target risk assessment in step S4 is as follows: Based on the pinhole camera projection model and combined with the elevation constraints in the digital elevation model, the detection results in the two-dimensional image are mapped to the three-dimensional world coordinate system to obtain the three-dimensional position of the target. A unique ID is assigned to the same target by continuous inter-frame feature matching, and the Kalman filter algorithm is used to calculate the target's motion speed and trajectory. Based on target type, location, speed, and trajectory information, a model combining rules and deep learning is used for risk assessment, outputting a comprehensive risk score and three levels of risk labels: high, medium, and low. Technical effects: It achieves accurate mapping from two-dimensional detection results to three-dimensional space, can accurately estimate the motion state of the target, and the risk assessment model combines the determinism of rules with the generalization ability of deep learning, thereby improving the accuracy and reliability of risk classification.

[0015] In one embodiment of this application, the specific process of task reassignment and path replanning in step S4 is as follows: The identified dynamic targets and their predicted trajectories are used as new obstacles to update the three-dimensional semantic environment model in real time, triggering the RL-QPSO Net algorithm to recalculate the globally optimal flight path. Set high-risk targets as the highest priority tasks, update the target priority weights in the comprehensive attraction calculation, and trigger the distributed task redistribution mechanism of the improved Dragonfly algorithm; The optimal drone can be quickly assigned to a multi-drone for verification or tracking through rapid consultation among multiple drones. Technical effects: The system achieves closed-loop autonomous operation, can dynamically adjust path planning and task allocation according to real-time environmental changes, realizes "detection and response" of abnormal targets, and ensures the continuous and stable operation of the system in dynamic and complex environments. Attached Figure Description

[0016] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0017] Figure 1 This is a flowchart illustrating the steps of a safety target tracking and early warning method applied to river shoreline areas in an embodiment of this application; Figure 2 This is a flowchart illustrating the steps involved in implementing multi-UAV collaborative task allocation and dynamic trajectory optimization using the improved Dragonfly algorithm in this application embodiment. Detailed Implementation

[0018] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0019] Example 1: like Figure 1 As shown, a method for tracking and early warning of safety targets applied to river shoreline areas includes the following steps: S1: By simultaneously collecting laser point cloud data and aerial survey image data of the river shoreline area through a cluster of drones equipped with 3D LiDAR and orthophoto cameras, a 3D semantic environment model integrating texture, elevation and semantic information is constructed.

[0020] Specifically: S1.1: Data Acquisition and Preprocessing: First, the aerial survey area was determined based on the demarcated river channel. Then, the aerial survey route was planned. Next, a UAV equipped with a 3D LiDAR and an orthophoto camera was used to collect data on the demarcated river channel, obtaining 3D LiDAR point cloud and aerial survey image data acquired simultaneously. The UAV was used to simultaneously carry a LiDAR and an orthophoto camera to acquire 3D LiDAR point cloud data. and aerial survey image sequence .

[0021] For the initial point cloud acquired by the UAV's lidar, automatic noise filtering is first performed based on local elevation statistics and a preset elevation range. Then, a cloth simulation filtering algorithm is used to construct a surface model by simulating the gravity settlement and internal force equilibrium process of virtual cloth on the flipped point cloud. Accurate separation of ground points and non-ground points is achieved based on the vertical distance from the point to the model and height thresholds. The point cloud is flipped to simulate the cloth settlement process. ; Among them Fabric node positions, For the corresponding laser point, To tighten the weights, the implementation steps of the cloth simulation algorithm are as follows: Step 1: Flip the laser point cloud data horizontally, that is, take the opposite value of the original elevation value; Step 2: Rasterize the point cloud data according to the preset grid size. The elevation value of each grid is the maximum elevation value of the laser foot point within that grid. The entire grid surface can be regarded as a piece of cloth, and each grid unit is regarded as a cloth unit. Step 3: Calculate and record the initial elevation difference IHV (i.e., the maximum movement distance) from each laser foot point to the corresponding cloth unit, and set the current state of all cloth units to the movable state; Step 4: Then, calculate the movement distance of each cloth unit according to the gravity motion model, update the elevation value of the cloth unit and compare it with the IHV value. If it is less than or equal to the IHV, it means that the cloth unit touches the ground and its motion state becomes immobile. Step 5: Calculate the return distance of each fabric unit under the tension of the surrounding fabric units and update its elevation value; Step 6: Repeat steps (4) to (5) until all the cloth units are basically no longer moving or the maximum number of iterations preset by the user is reached; Step 7: Calculate the vertical height of each laser point from the fabric, and classify them into ground points or non-ground points according to the preset height threshold.

[0022] S1.2: For ground point cloud data with holes after extraction, the process first involves gridding and discretizing, followed by interpolation to fill the hole areas using a spatial inverse distance weighted model. Then, a mean filtering algorithm is applied to smooth areas such as water surfaces, resulting in a continuous and complete ground elevation dataset. Finally, the ground point cloud is reconstructed based on the grid center points, and an irregular triangular mesh model is constructed. This is then used to generate a high-precision digital elevation model using ArcGIS rasterization. The spatial inverse distance weighted interpolation model is as follows: , ; in, , , Let N be the elevation of the point to be interpolated, and N be the number of neighboring points. The weights of neighboring points i, Let be the elevation of the nearest point i.

[0023] S1.3: Aerial triangulation without field control is performed based on the feature points of the lidar point cloud. Digital differential correction is performed in combination with the digital elevation model. After image stitching and color equalization processing, a true color orthophoto map with high geometric accuracy is generated.

[0024] S1.4: After projecting the radar point cloud and the UAV orthophoto onto the same plane coordinate system, feature points, lines, or areas are extracted from the point cloud data, and corresponding point sets are obtained on the orthophoto. High-precision registration of the radar point cloud and the orthophoto is achieved by establishing an affine transformation relationship based on the corresponding point sets. The affine transformation model involved in the registration process of corresponding point pairs is as follows: ; In the formula: and These represent the x and y coordinates of the i-th pair of corresponding points in the point cloud and the image, respectively. These are the affine transformation coefficients, which can be obtained by selecting at least six pairs of corresponding point sets and solving using least squares. After the orthophoto and radar point cloud are registered, assume that the coordinates of a point P in the laser point cloud on the projection plane are... Then the pixel coordinates of point P on the orthophoto are: ; In the formula: and These represent the row and column numbers of point P in the image pixel coordinate system, respectively. and —These represent the projected coordinates of the lower left corner of the image; and These represent the horizontal and vertical resolutions of the image, respectively.

[0025] 3D Model Representation: 3D Semantic Voxel Mesh Construction: Establishing Voxel Mesh Data Structure Each voxel stores: occupancy state Elevation RGB texture Semantic tags Calculate the symbolic distance field (SDF): ; Output a three-dimensional semantic environment model By establishing a two-dimensional and three-dimensional mapping relationship through an affine transformation model between three-dimensional laser point clouds and orthophotos, the laser point clouds collected by the UAV radar system and aerial survey images can be seamlessly integrated to generate a true-color three-dimensional point cloud map. Then, three-dimensional visualization engine technologies such as ArcScene and Global Mapper are used to construct a three-dimensional terrain model of rivers and lakes.

[0026] S2: Based on a three-dimensional semantic environment model, the RL-QPSO Net algorithm is used for single UAV trajectory planning.

[0027] RL-QPSO is a path planning model that combines deep reinforcement learning with quantum behavior particle swarm optimization. This model employs a two-layer control mechanism: deep reinforcement learning handles real-time obstacle avoidance and local navigation decisions, while quantum behavior particle swarm optimization enhances global search capabilities through its quantized stochastic behavior, optimizing the overall path and avoiding getting trapped in local optima.

[0028] Specifically: S2.1: Problem Modeling: In view of the complex dynamic environment faced by UAVs in river shoreline inspection, this invention models the global path planning problem as a dynamic optimization problem in a high-dimensional space, and uses the quantum behavior particle swarm optimization (RL-QPSO Net) algorithm enhanced by deep reinforcement learning to solve it.

[0029] Environment and State Representation: The three-dimensional space of the inspection is defined as a bounded region. Includes a set of static obstacles (Such as bridges, fixed facilities) and dynamic obstacles (Such as ships, floating objects). The drone's status is... in For location, This is the yaw angle.

[0030] Establishing a UAV dynamics model: To formalize the UAV path planning problem, the environment is represented as a bounded space E, containing static obstacles and dynamic elements, respectively... and This indicates that the drone's mission is to start from a given starting position. Navigate to the specified desired location The goal is to avoid all obstacles and minimize the total travel cost. This cost can be a combination of factors such as distance, energy consumption, and safety margin. The state of the drone at any time step t is determined by... It means that, among them Indicates the location of the drone in E. It is the azimuth angle relative to the global coordinate system. The drone's motion is controlled by the input. Control, among which and Let represent the translational and rotational velocities, respectively. Therefore, the dynamics of the UAV can be described by the following formula: ; in, It refers to the discrete time step. Path planning can be described as an optimization problem where the objective is to find the optimal sequence of states. and corresponding control inputs Its minimum cost function It is constrained by the dynamics, control inputs, and the interaction of the environment. The general form of the objective function number can be expressed as: ; in Measure current position and expected state The Euclidean distance between them Representation and Input Related control costs, It is for approaching obstacles The penalty function, and and It is a weighted factor that balances the trade-offs between achieving the goal, controlling effort, and safety.

[0031] For a drone to navigate effectively, it must meet several constraints. Obstacle avoidance requires the drone to maintain a safe distance from all obstacles at any location. For those located in Static obstacles, represented as: ; For dynamic obstacles, the safe distance must take into account their position over time. The formula is as follows: ; Kinematic and dynamic constraints require control inputs Meet the physical limitations of the drone, such as maximum speed. and maximum rotational speed : :

[0032] The boundary constraints that ensure the drone's path remains within the boundaries of environment E are typically expressed as:

[0033] The control law guiding the drone can be expressed as:

[0034] Fitness function (coupled environment model): Path cost calculation: , Among them, the environmental cost item is: , in, It is the shortest distance between the drone and the obstacle. Distance to the target: , It is the location of the obstacle. This is the location of the drone.

[0035] S2.2: QPSO Synthetic Module Design: To improve the environmental awareness and memory capabilities of particle swarm optimization in sequential decision-making, a gated recurrent unit (GRU) is integrated into the particle update process of QPSO. GRU-driven particle state update: Each particle i carries a GRU unit, whose hidden state... The particle's historical search experience and environmental interaction context are encoded. The particle position update formula is: ; in: The length of the quantum potential well. The best position in particle history. : Global optimal position.

[0036] GRU network outputs adaptive weights : ; Adaptive weights The hidden state of the GRU and the output of the DRL module are jointly determined, which allows the particle to dynamically adjust its exploration step size and direction not only based on historical and global optima, but also in combination with real-time environmental assessment.

[0037] S2.3: Dual-objective reward and decision-making mechanism of DRL module: As the intelligent decision-making center of the system, DRL module learns the optimal path point selection strategy by maximizing cumulative rewards.

[0038] Bi-objective reward function mechanism: reward function Designed to simultaneously propel the drone efficiently toward the target and away from all obstacles, the drone's state is set based on environmental characteristics and the current performance metrics of the QPSO: ,in For dynamic target information, in order to adjust the adaptive weights of QPSO and quantum potential well length Introduce the action state update formula: Finally, a reward function (dual-objective) mechanism is set: ; In the formula, the first term Drive the drone to the target location Approaching. Second item. Approaching obstacles by drones The behavior is subject to exponential penalties.

[0039] The DRL module adopts an Actor-Critic architecture, in which the policy network outputs action instructions based on the current environmental state, while the value network evaluates the state value to guide policy optimization. Through an online learning mechanism, the system can adapt to unplanned complex situations in the river environment, such as changes in lighting and sudden dynamic vessel events.

[0040] S2.4: DRL performs global dynamic trajectory planning for QPSO parameter adjustments: Define context function Its value is a weighted sum of various environmental factors (such as risk area density, water flow impact, and airspace control). This function value is input as an additional signal to the DRL module and used to fine-tune the cost assessment of QPSO, enabling the planned path to have an intelligent understanding of the geographical and mission context. Specifically: 1. First, initialize the QPSO data and generate the particle swarm. Step vector .

[0041] 2: Next, an interactive loop is performed: QPSO executes the path search and returns a reward function based on environmental parameters. .

[0042] 3: The DRL module uses the returned reward function Calculate new and Update the QPSO parameters and continue the search.

[0043] 4. Finally, convergence is performed when the reward stabilizes or the maximum number of iterations is reached, and the optimal path is output. .

[0044] S3: Based on UAV trajectory planning, an improved Dragonfly algorithm is used to achieve multi-UAV collaborative task allocation and dynamic trajectory optimization.

[0045] The RL-QPSO Net algorithm in S2 generates a reference track optimized globally and temporally for a single UAV. This invention introduces a dragonfly algorithm based on neighborhood behavior in S3, aiming to perform real-time, distributed local optimization of the output of S2, specifically implemented through the following interface: Input interface: S2 for each drone Output global reference trackpoint sequence

[0046] Optimize the target interface: without significantly deviating from global track guidance, based on the context function. and optimal path It can solve the problem of coordinated collision avoidance and formation maintenance among multiple aircraft in real time, and make agile avoidance of sudden close-range dynamic obstacles (such as birds or ships that suddenly enter) that S2 does not detect in time.

[0047] like Figure 2 As shown, specifically: S3.1: Dragonfly Algorithm-Driven Neighborhood Behavior Modeling: Each drone is treated as a dragonfly individual, and its next motion is synthesized from five behavior vectors. These behaviors depend only on information within its local perception range (other drones, obstacles, and target points). Let X be the current individual's position vector. Let N be the position vector of the k-th neighboring individual, and N be the number of neighboring individuals. X is the step size vector.

[0048] Alignment behavior: This ensures that the direction of movement of the local aircraft aligns with the average direction of its neighbors, forming the basis for formation flying and cooperative search. Among other things... This represents the permutation of the i-th drone. This represents the velocity vector of the adjacent UAV.

[0049] ; Separation behavior: Avoiding collisions with any neighboring objects within the perception range is the core of safe obstacle avoidance. Among these, This represents the separation of the i-th drone.

[0050] ; Clustering behavior: This drives drones to move towards the center of their neighbors' locations to prevent the swarm from losing coordination or creating gaps in coverage due to dispersion. This represents the cohesion of the i-th drone.

[0051] ; Tracking target: ,in, This represents the flight target of the i-th UAV, ensuring that the UAV flies to the next global waypoint provided by S2. To ensure that we do not deviate from our core mission.

[0052] Obstacle avoidance: ,in, This represents the dynamic obstacle that the i-th drone needs to avoid during flight. This represents the location of obstacles near the drone.

[0053] Coupling with the environmental model: Flight target Set as the next waypoint on the global path. ( The globally optimal path from the output of S2. .

[0054] Obstacle avoidance :Include: Static obstacle enemies: from the environment model Search for drones Nearby occupying voxels: ; Choose the nearest static obstacle as the primary enemy: ; Dynamic obstacle enemies: a dynamic target list from step four. Target location: ; Other drone enemies: In multi-drone collaboration, other drones are also "enemies" that need to be avoided: ; Select another nearby drone: ; Determining enemy positions based on priority: .

[0055] S3.2: Dynamic task allocation mechanism based on contextual attraction: Task attraction modeling: When the S2 module identifies M targets to be inspected (such as suspicious vessels or floating objects), each target is considered... As a virtual gravitational source. Drone i towards the target. Overall attractiveness The calculation is as follows: ; Basic distance attraction: follows the law of universal gravitation and is inversely proportional to the square of the distance. For the goal The inherent priorities (e.g., "fire" has a higher priority than "routine patrol") Value priors based on DRL: This is a key innovation in combination with RL-QPSO. This is the adjustment coefficient. It is the DRL module in S2 Towards the target The value network assessment value of the states traversed.

[0056] Battery drain penalty: Consider the remaining battery power of the drone to prevent drones with low battery from being used for remote missions.

[0057] S3.3: Multi-UAV Distributed Negotiation and Task Matching: Each UAV calculates its attractiveness to all M targets in parallel and selects the target with the highest attractiveness as its intended target. Subsequently, a round of negotiation is conducted via lightweight communication. 1: If a target is selected by only one drone, the match is successful.

[0058] 2: If a target is attacked by multiple drones Competition involves comparing their overall attractiveness. .

[0059] 3. The drone with the highest appeal wins the target. The remaining drones that failed to win the bid are removed from their list of interests.

[0060] 4: Recalculate the attractiveness of the remaining targets and select new target preferences until all drones are assigned to a single target or all targets have been assigned.

[0061] S3.4: Real-time Collaborative Flight Optimization: During the mission execution phase, each UAV needs to fly towards its assigned target. Simultaneously, it enables real-time collision avoidance with friendly aircraft and dynamic obstacles. This module improves upon the five behavioral rules of the classic Dragonfly algorithm, enabling it to utilize S2 prediction information.

[0062] Define neighborhood: , Step 1: Improve Behavioral Rule 1: Predictive Separation Behavior Separation behavior not only requires avoiding current neighbors, but also requires proactive avoidance based on S2 dynamic obstacle prediction.

[0063] ; in, Directly use the dynamic obstacle prediction function in S2 The predicted location. and The risk coefficient can be derived from the risk assessment in the S2 context function. Step 2: Improve Behavioral Rule 2: Goal Orientation and Dynamic Path Feedback: The target directional force is directly directed towards the current target point assigned by S2. But weight Receive real-time rewards in S2 Adjustment: ; in This is the sigmoid function. It's used when the DRL module evaluates the real-time reward for the current path. When the threshold is lower (indicating unexpected difficulties or increased risks), the drone automatically reduces its "attachment" to fixed targets, allowing it to maneuver more flexibly around them.

[0064] Step 3: Improved Behavioral Rule 3: Context-Aware Aggregation Behavior The aggregation behavior guides drones to move closer to their teammates, but the calculation of their "center point" incorporates mission-related factors: ; This ensures that the formation is not only based on the geometric center, but also slightly offset from the average position of the mission target. (As a bias weight), so that the formation as a whole can still advance towards the mission area while coordinating obstacle avoidance.

[0065] Step 4: Update the step vector: ; Step 5: Update location: ; Step 6: Track Synthesis and Optimization: Finally, the velocity adjustment of UAV i at time t. Synthesized by the following formula ; Among them, weight It can be dynamically configured according to the drone mode (patrol, track, return to home). The synthesized speed command is input into the underlying flight control system to achieve smooth real-time trajectory optimization.

[0066] S4: Use the improved YOLOv8 algorithm to perform target recognition on real-time image data collected by the UAV, output detection results containing abnormal target information, and perform task reassignment, path replanning and risk assessment accordingly.

[0067] This step is the core hub of the "perception-decision-response" closed loop of UAV inspection. It is responsible not only for real-time processing of the fused visual and lidar data from the UAV, accurately identifying and locating key targets, but also for generating a global panoramic perception map, assessing risks, and triggering multi-level responses. Specifically, based on real-time trajectory and attitude data from S2 / S3, combined with its own sensor input, it completes target identification and classification using an improved YOLOv8 algorithm, then fuses spatial information to generate a multi-level panoramic semantic map, locates dangerous targets, and dynamically feeds back structured alarms and mission instructions to S2 (replanning), S3 (cooperative response), and the ground detection center, forming a complete autonomous operation closed loop.

[0068] Specifically: S4.1: Construction of an ensemble model for efficient multi-scale attention modules (EMA-YOLOv8): Step 1: Construct an image dataset containing various scenes of the river shoreline, covering boats of different scales and orientations, various types of floating objects, and shore facilities.

[0069] Step 2: Enhance the dataset by performing random rotations, adjusting brightness and contrast, and simulating fog to improve the model's generalization ability.

[0070] Step 3: Utilize the Exponential Moving Average (EMA) technique to smooth model weight updates, improving training stability and the final model's generalization performance. The model structure follows YOLOv8, with EMA modules added to the three C2 modules in the Neck part. The EMA module parameter update rules are as follows: ; in This is the smoothing coefficient (values ​​close to 1 are typically 0.999).

[0071] S4.2: Introducing an attention mechanism: By using a parallel multi-branch structure, contextual information under different receptive fields is captured, and a cross-dimensional interactive attention mechanism is used to adaptively calculate the importance weights of feature maps in the spatial and channel dimensions.

[0072] ; in , , They are obtained from the input feature maps through different linear transformations. denoted as the dimension of the key vector. The EMA module is embedded into the feature fusion path, performing attention-weighted processing on the multi-level features from the backbone before feature upsampling, downsampling, and fusion to enhance the expressive fusion capability of the output features.

[0073] S4.3: Enhanced Rotation Robustness Design: To address the variable orientation of vessels in rivers, a rotation angle is added to the improved YOLOv8 head. Prediction Branch. In addition to the existing bounding box regression (center point, width and height) and classification branches, a regression branch is added to predict the target's rotation angle. The loss function is based on the original localization loss. Confidence loss and classification loss Based on this, add angle regression loss. Typically, smoothed L1 loss is used.

[0074] Calculate the loss function: ; in: = Location loss (using mean squared error, MSE) = : Confidence loss (using binary cross-entropy). = The classification loss (using binary cross-entropy) is represented by λ, which is the balance coefficient. This design enables the model to output oriented target bounding boxes (rotated rectangles), significantly improving the detection accuracy and localization accuracy of ships facing any direction.

[0075] S4.4: Use Mean Precision (mAP) and F1-Score as core metrics for evaluation: Performance evaluation metric: mAP (mean accuracy): ; The model's detection accuracy across all categories is comprehensively evaluated, with particular attention paid to its performance at different cross-union thresholds to ensure accurate localization. For the number of categories, This is the precision-recall curve for class c.

[0076] F1-Score: , in: , ; Balancing precision and recall is a key indicator for evaluating the overall performance of a model in practical applications. A high F1-Score means that both false negatives and false negatives are kept at a low level.

[0077] S4.5: Register the detection results with the three-dimensional semantic environment model to achieve three-dimensional localization and risk assessment of abnormal targets; Calculate 2D to 3D coordinate mapping: Camera projection model (standard pinhole model): ; For targets on the water surface, elevation constraints are used: in From the DEM obtained in the first step, the world coordinates are calculated: ; in: Camera main point, Camera focal length : Camera translation vector.

[0078] Step 1: For the input real-time image frames The system first performs forward propagation through the EMA-YOLOv8 network, outputting the target's bounding box, category, and confidence score.

[0079] Step 2: After non-maximum suppression (NMS) filtering of overlap detection, the 2D detection results are mapped to the 3D world coordinate system, and a unique ID is assigned to the same target in consecutive frames.

[0080] Step 3: The final output is a dynamic list of targets, including their identity, category, 3D location, velocity, and confidence level. .

[0081] S4.6: Multimodal perception fusion and dynamic environment closed-loop update: Step 1: Send the identified target category, 3D position (including rotation angle), confidence level, and velocity estimation, etc., to the S2 RL-QPSO Net model.

[0082] Step 2: Newly identified moving vessels and other targets will be treated as dynamic obstacles. Real-time addition to the S2 environment model In the middle, the context-aware function of S2 is triggered. Perform future trajectory prediction. Conduct online trajectory replanning to generate new paths that avoid high-risk areas or actively observed targets.

[0083] Step 3: Using the UAV pose and camera parameters fed back by S3, the LiDAR point cloud is projected onto the image coordinate system to achieve pixel-level alignment. The depth and reflection intensity of the LiDAR are used as additional channels and fused with the RGB image in the early stage of the EMA-YOLOv8 Backbone.

[0084] Step 4: Based on real-time video frames, UAV pose, and fused LiDAR data, the system generates a high-priority task event when an abnormal target is detected, triggering a multi-UAV collaborative task reallocation mechanism to guide the nearest or most suitable UAV to the verification or tracking, achieving "detection and response".

[0085] Step 5: Use pose data to stitch and correct consecutive frames of images, project the target category, confidence level and 3D coordinates to the global coordinate system, and dynamically update the multi-layer panoramic semantic map that integrates visual, semantic, risk and spatial states through a sliding window mechanism.

[0086] Step 6: Combine image pixel positions with LiDAR point clouds, and calculate the target's precise position and velocity vector in the world coordinate system through triangulation or deep learning regression; at the same time, provide pixel-level distance and orientation estimation for sudden obstacles at close range to enhance the accuracy of S3 macro obstacle avoidance.

[0087] Step 7: Construct a risk assessment model that combines rules and learning, where the rules part sets risk scores based on attributes such as target type, speed, location, and trajectory.

[0088] Step 8: The learning part introduces the long-term value function of the DRL module in S2 to quantify the risk of complex multi-objective interaction scenarios, and finally outputs the comprehensive risk score and grade label of each objective.

[0089] S5: Generate early warning information based on the risk assessment results and send it to the monitoring center and relevant personnel terminals.

[0090] Specifically, it sends structured alarm information packages and panoramic snapshots to the ground monitoring center, and displays the target location, risk level, and drone status in real time at the control center, providing decision support for manual supervision.

[0091] This method uses an improved YOLOv8 algorithm to identify and classify targets, then integrates spatial information to generate a multi-layer panoramic semantic map, locates dangerous targets, and dynamically feeds back structured alarms and task instructions to S2 (replanning), S3 (cooperative response) and the ground detection center, forming a complete autonomous operation closed loop.

[0092] Example 2: A safety target tracking and early warning system applied to river shoreline areas, employing the safety target tracking and early warning method of Embodiment 1 described above, includes: The environment modeling module is used to simultaneously collect laser point cloud data and aerial survey image data of the river shoreline area through a cluster of drones equipped with 3D LiDAR and orthophoto cameras, and to construct a 3D semantic environment model that integrates texture, elevation and semantic information. The global trajectory planning module is used to plan the trajectory of a single UAV based on a 3D semantic environment model and using the RL-QPSO Net algorithm. The multi-drone collaboration module is used to achieve multi-drone collaborative task allocation and dynamic trajectory optimization based on UAV trajectory planning and using an improved Dragonfly algorithm. The target recognition module is used to identify targets in real-time image data collected by UAVs using the improved YOLOv8 algorithm, and output detection results containing abnormal target information. Based on this, task reassignment, path replanning and risk assessment are performed. The early warning module is used to generate early warning information based on the risk assessment results and send it to the monitoring center and relevant personnel terminals.

[0093] The functions of each module in Example 2 correspond to the contents of its corresponding method example, and will not be repeated here.

[0094] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages: This method constructs a fully autonomous inspection closed loop that deeply integrates "perception-planning-coordination" capabilities. The S4 module, acting as the intelligent perception hub, fuses LiDAR point cloud data and visual data using an improved EMA-YOLOv8 algorithm to achieve accurate multi-scale target identification and 3D localization, outputting real-time semantically rich environmental cognition. This information dynamically updates the global environment model of the S2 module (RL-QPSO Net), driving it to perform forward-looking trajectory planning or online replanning. Furthermore, key identification events (especially high-risk targets) directly trigger the flexible task allocation and collaborative response mechanism of the S3 module (improved Dragonfly algorithm). Based on the macro-planning of S2 and the real-time target information provided by S4, the S3 module calculates the optimal cooperative motion commands for each UAV through improved neighborhood behavior rules and distributed decision-making, achieving collision-free, adaptive formation dynamic inspection of multiple UAVs. As the UAVs execute optimized trajectories, their onboard sensors continue to collect data and feed it back to S4, thus initiating the next "perception-decision-execution" cycle. This closed-loop architecture enables the system to possess real-time understanding, intelligent decision-making, and collaborative response capabilities for complex and dynamic river environments.

[0095] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for tracking and early warning of safety targets applied in riverbank areas, characterized in that, Includes the following steps: S1: Simultaneously collect laser point cloud data and aerial survey image data of the river shoreline area through a cluster of drones to construct a three-dimensional semantic environment model that integrates texture, elevation and semantic information; S2: Based on a three-dimensional semantic environment model, the RL-QPSO Net algorithm is used for single UAV trajectory planning; S3: Based on UAV trajectory planning, an improved Dragonfly algorithm is used to achieve multi-UAV collaborative task allocation and dynamic trajectory optimization; S4: Use the improved YOLOv8 algorithm to perform target recognition on real-time image data collected by the UAV, output detection results containing abnormal target information, and perform task reassignment, path replanning and risk assessment accordingly. S5: Generate early warning information based on the risk assessment results and send it to the monitoring center and terminals.

2. The security target tracking and early warning method according to claim 1, characterized in that: The specific process of constructing the three-dimensional semantic environment model in step S1 is as follows: S1.

1. Noise is filtered out from the collected laser point cloud data. A cloth simulation filtering algorithm is used to separate ground points from non-ground points. The ground points are extracted by simulating the gravity settling process of the cloth on the flipped point cloud. S1.

2. For the missing point cloud areas caused by the removal of non-ground points and water reflection, spatial inverse distance weighted interpolation is used to fill and smooth the elevation grid to generate a digital elevation model; S1.

3. Perform aerial triangulation of aerial survey images without field control based on point cloud feature points, and generate orthophotos by combining with digital elevation models; S1.

4. The laser point cloud and orthophoto are registered and fused through affine transformation to construct a three-dimensional semantic environment model containing fused texture, elevation and semantic information.

3. The security target tracking and early warning method according to claim 1, characterized in that: The process of trajectory planning using the RL-QPSO Net algorithm in step S2 is as follows: S2.

1. Model the global path planning problem as a dynamic optimization problem in a high-dimensional space, establish a UAV dynamic model that includes obstacle avoidance constraints, dynamic constraints and boundary constraints, and set the objective function to minimize flight distance, control cost and collision risk; S2.

2. Integrate the gated loop unit into the particle update process of quantum behavior particle swarm optimization, and use the hidden state encoding of GRU to encode the particle history search experience and environmental interaction context to generate adaptive weights to adjust the particle search step size and direction. S2.

3. A deep reinforcement learning module employing a dual-objective reward function mechanism adjusts the quantum potential well length and adaptive weight parameters of quantum behavior particle swarm optimization in real time based on environmental feedback, thereby achieving global trajectory dynamic planning; S2.

4. When the reward function tends to stabilize or reaches the maximum number of iterations, output the optimal flight path of the UAV.

4. The security target tracking and early warning method according to claim 3, characterized in that: The dual-objective reward function in step S2 includes both a target approach reward term and an obstacle avoidance penalty term, and its specific expression is as follows: ； In the formula, the first term Drive the drone to the target location Closer, the second item Approaching obstacles by drones The behavior is subject to exponential penalties.

5. The security target tracking and early warning method according to claim 1, characterized in that: The specific process of using the improved Dragonfly algorithm to achieve multi-UAV cooperative task allocation and dynamic trajectory optimization in step S3 is as follows: Treating each drone as an individual dragonfly, the speed adjustment of the drone is synthesized based on three improvement rules: predictive separation behavior, target tendency and dynamic path feedback behavior, and context-aware aggregation behavior. When multiple targets to be inspected are identified, each target is regarded as a virtual gravitational source, the comprehensive attraction of each drone to each target is calculated, and the task allocation of multiple drones is completed through a distributed negotiation mechanism. Based on the assigned mission objectives and the synthesized speed adjustment, real-time flight commands are generated for each UAV and input into the underlying flight control system for execution.

6. The security target tracking and early warning method according to claim 5, characterized in that: The calculation of the overall attractiveness comprises three parts: basic distance attractiveness, DRL-based value prior, and power loss penalty. The specific expression is as follows: .

7. The security target tracking and early warning method according to claim 1, characterized in that: The specific structure of the improved YOLOv8 algorithm in step S4 is as follows: Efficient multi-scale attention modules are added to the three C2 modules of the Neck part of YOLOv8. These modules capture contextual information under different receptive fields through a parallel multi-branch structure and adaptively calculate the weights of feature maps in the spatial and channel dimensions. A rotation angle prediction branch is added to the detection head of YOLOv8, and the loss function is optimized by adding an angle regression loss term to improve the detection accuracy of targets facing any direction. The mean precision and F1-Score are used as the core evaluation metrics for model training to ensure the model's detection accuracy and generalization performance.

8. The security target tracking and early warning method according to claim 1, characterized in that: The specific process of abnormal target risk assessment in step S4 is as follows: Based on the pinhole camera projection model and combined with the elevation constraints in the digital elevation model, the detection results in the two-dimensional image are mapped to the three-dimensional world coordinate system to obtain the three-dimensional position of the target. A unique ID is assigned to the same target by continuous inter-frame feature matching, and the Kalman filter algorithm is used to calculate the target's motion speed and trajectory. Based on target type, location, speed, and trajectory information, a model combining rules and deep learning is used for risk assessment, outputting a comprehensive risk score and three levels of risk labels: high, medium, and low.

9. The method for tracking and early warning of security targets according to claim 1, characterized in that: The specific process of task reassignment and path replanning in step S4 is as follows: The identified dynamic targets and their predicted trajectories are used as new obstacles to update the three-dimensional semantic environment model in real time, triggering the RL-QPSO Net algorithm to recalculate the globally optimal flight path. Set high-risk targets as the highest priority tasks, update the target priority weights in the comprehensive attraction calculation, and trigger the distributed task redistribution mechanism of the improved Dragonfly algorithm; The optimal drone can be quickly assigned to a multi-drone for verification or tracking through rapid consultation among multiple drones.

10. A safety target tracking and early warning system applied to river shoreline areas, characterized in that, The security target tracking and early warning method as described in any one of claims 1-9 includes: The environmental modeling module is used to simultaneously collect laser point cloud data and aerial survey image data of river shoreline areas through a cluster of drones, and construct a three-dimensional semantic environment model that integrates texture, elevation and semantic information. The global trajectory planning module is used to plan the trajectory of a single UAV based on a 3D semantic environment model and using the RL-QPSO Net algorithm. The multi-drone collaboration module is used to achieve multi-drone collaborative task allocation and dynamic trajectory optimization based on UAV trajectory planning and using an improved Dragonfly algorithm. The target recognition module is used to identify targets in real-time image data collected by UAVs using the improved YOLOv8 algorithm, and output detection results containing abnormal target information. Based on this, task reassignment, path replanning and risk assessment are performed. The early warning module is used to generate early warning information based on the risk assessment results and send it to the monitoring center and terminals.

Images