A method and system for unmanned aerial vehicle video linkage disposal

Abstract

This application relates to the field of UAV collaborative control and security monitoring technology, and discloses a UAV video linkage processing method and system. The method includes: a fixed camera sensing abnormal events and constructing an initial line-of-sight vector; generating a spatial search envelope region by combining a digital elevation model and the fixed camera calibration error covariance matrix, and calculating the spatial volume parameters of the spatial search envelope region; substituting the spatial volume parameters of the spatial search envelope region as an uncertainty penalty term into a comprehensive scoring function to execute dynamic scheduling decisions, guiding the UAV to the target area; the UAV generating a secondary observation line-of-sight vector, calculating the common perpendicular of the two line-of-sight vectors, and obtaining the coordinates of the midpoint of the common perpendicular as the converged precise three-dimensional geographic coordinates; executing dual-end field-of-view co-orientation control, extracting multi-source data to generate a data summary to complete the evidence chain closure. This application overcomes the shortcomings of monocular vision in terms of depth loss, avoids the power consumption of blind UAV searches, and ensures the tamper-proof nature of the evidence data.

Description

Technical Field

[0001] This invention relates to the field of drone collaborative control and security monitoring technology, specifically a drone video linkage processing method and system. Background Technology

[0002] With the integration of drone technology and high-altitude video surveillance equipment, the coordinated operation of fixed monitoring and drones is gradually becoming an important technical means for large-scale outdoor security and comprehensive rural governance. Traditional purely manual patrol methods are often limited by complex terrain and labor costs, resulting in slow response and large blind spots. By deploying fixed pan-tilt-zoom (PTZ) cameras for 24 / 7 wide-area patrols, drones can be called in for close-up reconnaissance upon detecting anomalies. This combined air-ground operational model has demonstrated its mobility advantages and application value in scenarios such as forest fire monitoring, illegal construction inspections, and security of key waterways.

[0003] In existing conventional video-linked processing systems, a unidirectional static coordinate mapping and open-loop scheduling strategy is commonly used in engineering. Specifically, when a fixed camera identifies an abnormal target in the image using a visual algorithm, the system extracts the target's two-dimensional pixel coordinates in the image. Subsequently, the platform uses the camera's intrinsic and extrinsic parameter matrices calibrated during installation, combined with a preset static plane elevation assumption, to directly calculate these two-dimensional pixel coordinates into a single three-dimensional absolute geographic coordinate. In the task scheduling phase, the platform typically generates a dispatch instruction based solely on the initial risk level of the event and the physical distance to the equipment, issuing the calculated geographic coordinates as the sole navigation endpoint. The drone then relies on its own satellite navigation system to fly unidirectionally to that location to perform the filming and evidence collection task.

[0004] However, in complex terrain, existing technologies, particularly monocular cameras, are prone to failure due to the inherent lack of depth information. Their reliance on static elevation assumptions is easily invalidated in undulating terrain, and even slight gimbal mechanical disturbances, amplified by the mapping matrix, can generate significant target spatial positioning deviations. This results in the target not being within the downward field of view after the UAV reaches the calculated coordinates. Existing fixed dispatch mechanisms do not incorporate the geometric uncertainty of target positioning into the evaluation dimension, forcing tasks with spatial errors to be assigned. This compels multi-rotor UAVs with limited endurance to perform large-scale blind searches over the target, leading to significant power waste and scheduling resource consumption. Furthermore, the fixed monitoring end and the UAV end are disconnected in the end-of-line evidence collection stage, lacking heterogeneous field-of-view collaborative control based on unified and precise physical coordinates. Consequently, the final archived business data is merely a simple integration of discrete images and wide-angle videos, failing to form a consistent evidence loop that meets the requirements of grassroots law enforcement compliance and tamper-proofing. Therefore, this invention provides a UAV video linkage processing method and system to address the shortcomings of existing technologies. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method and system for drone video linkage processing. It solves the problems of existing drone video linkage systems in complex terrain scenarios, where fixed monocular cameras lack depth information, leading to elevation errors in spatial mapping, which in turn causes ineffective drone scheduling and blind searching. Furthermore, the data is fragmented between the front-end perception, resource scheduling, and end-point control links, making it difficult to form a closed loop for heterogeneous device collaborative orientation and tamper-proof evidence.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] The first aspect of this invention provides a method for coordinated processing of drone video, comprising the following steps:

[0008] A fixed camera detects abnormal events and constructs an initial line-of-sight vector from the fixed camera coordinate system to the world coordinate system;

[0009] An elevation tolerance range is set along the initial line-of-sight vector and combined with the digital elevation model. The spatial search envelope is generated by boundary expansion using the fixed camera calibration error covariance matrix. The spatial volume parameters of the spatial search envelope are then calculated.

[0010] The spatial volume parameter of the spatial search envelope is used as an uncertainty penalty term and substituted into the comprehensive scoring function to execute dynamic scheduling decisions. If the comprehensive scheduling score is greater than the dispatch threshold, the UAV is guided to the target area.

[0011] The UAV generates a secondary observation line-of-sight vector, calculates the common perpendicular line between the initial line-of-sight vector and the secondary observation line-of-sight vector, and obtains the coordinates of the midpoint of the common perpendicular line as the accurate three-dimensional geographic coordinates after convergence.

[0012] The target attitude angle is solved by using precise three-dimensional geographic coordinates to perform dual-field-of-view collaborative orientation control, and multi-source data is extracted simultaneously to generate data summaries to complete the evidence chain loop.

[0013] Furthermore, the steps for a fixed camera to detect abnormal events and construct an initial line-of-sight vector from the fixed camera coordinate system to the world coordinate system include: extracting the geometric center position of the bounding box of the abnormal target within the video frame image to generate the center pixel coordinates; obtaining the absolute geographic coordinates of the fixed camera installation and converting them into three-dimensional world coordinates, and simultaneously reading the current real-time yaw angle and real-time pitch angle of the fixed camera; calling the intrinsic parameter matrix to transform the center pixel coordinates to the fixed camera coordinate system, constructing a rotation matrix using the real-time yaw angle and real-time pitch angle, transforming the vector in the fixed camera coordinate system to the world coordinate system and performing a normalization operation to obtain the unit direction vector, and combining the three-dimensional world coordinates and the unit direction vector to establish the line-of-sight vector equation to generate the initial line-of-sight vector.

[0014] Furthermore, the steps of setting an elevation tolerance range along the initial line-of-sight vector and combining it with the digital elevation model, using the fixed camera calibration error covariance matrix to expand the boundary and generate a spatial search envelope, and calculating the spatial volume parameters of the spatial search envelope include: using a spatial ray tracing algorithm to calculate the three-dimensional intersection point of the initial line-of-sight vector and the surface of the digital elevation model to obtain the reference elevation; superimposing a preset ground deviation threshold along the direction of the initial line-of-sight vector to set the elevation tolerance range; solving for the lower and upper bounds of the ray distance parameters based on the elevation tolerance range; intercepting the corresponding three-dimensional vector line segments; extracting the comprehensive angle error standard deviation from the fixed camera calibration error covariance matrix; calculating the lateral expansion radius with the three-dimensional vector line segments as the central axis to generate the spatial search envelope; integrating the cross-sectional area formed by the lateral expansion radius along the central axis direction of the three-dimensional vector line segments; and using the calculated physical volume as the spatial volume parameter of the spatial search envelope.

[0015] Furthermore, the spatial volume parameters of the spatial search envelope are used as uncertainty penalty terms and substituted into the comprehensive scoring function to execute dynamic scheduling decisions. If the comprehensive scheduling score is greater than the dispatch threshold, the steps to guide the UAV to the target area include: establishing the ratio relationship between the spatial volume parameters of the spatial search envelope and the benchmark reference volume constant, and calculating the uncertainty penalty term based on the logarithmic mapping function; collecting the abnormal event risk level score, target recognition confidence parameter, and UAV resource availability score, and combining the normalized weight coefficient and the uncertainty penalty term to calculate the comprehensive scheduling score; comparing the comprehensive scheduling score with the dispatch threshold, and if the comprehensive scheduling score is less than or equal to the dispatch threshold, storing the scheduling task in the task deferred queue; if the comprehensive scheduling score is greater than the dispatch threshold, extracting the boundary vertex coordinates of the spatial search envelope and converting them into waypoint instructions, and issuing flight routes to guide the UAV to the target area.

[0016] Furthermore, the data collection steps for the drone resource availability score include: extracting the current remaining battery power of the drone and the estimated flight time to reach the target area; converting the flight time into the estimated power consumption, comparing the difference between the current remaining battery power and the estimated power consumption; converting the difference into an availability score within a unified range using a preset normalization function, and outputting the drone resource availability score.

[0017] Furthermore, the steps of generating a secondary observation line-of-sight vector by the UAV, calculating the common perpendicular of the initial line-of-sight vector and the secondary observation line-of-sight vector, and obtaining the midpoint coordinates of the common perpendicular as the converged precise 3D geographic coordinates include: obtaining the UAV's 3D world coordinates; locking abnormal targets through a visual target matching algorithm; calculating the secondary unit direction vector by combining real-time fuselage and gimbal attitude data; constructing the secondary observation line-of-sight vector; calculating the spatial common perpendicular of the initial line-of-sight vector and the secondary observation line-of-sight vector based on a spatial analytical geometry solution algorithm; solving for the intersection parameters of the common perpendicular on the initial line-of-sight vector and the intersection parameters on the secondary observation line-of-sight vector; calculating the coordinate positions of the two ends of the common perpendicular based on the intersection parameters; extracting the midpoint coordinates of the line connecting the two ends of the common perpendicular; and using the midpoint coordinates as the final converged precise 3D geographic coordinates.

[0018] Furthermore, the steps of using precise 3D geographic coordinates to inversely solve the target attitude angle to perform dual-field-of-view collaborative orientation control include: acquiring precise 3D geographic coordinates and the UAV's real-time 3D world coordinates; calculating the target yaw angle and target pitch angle of the UAV gimbal based on the arctangent function using a spatial relative coordinate system; issuing gimbal attitude adjustment commands; dynamically calculating the focal length based on the straight-line spatial distance of the UAV relative to the precise 3D geographic coordinates; issuing focal length adjustment commands to drive the UAV zoom camera to acquire the target image; extracting the static 3D world coordinates of the fixed camera; calculating the true yaw angle and true pitch angle of the fixed camera pointing to the precise 3D geographic coordinates based on spatial coordinate inverse solving logic; encapsulating these into gimbal drive commands to drive the fixed camera's field of view center to align with the precise 3D geographic coordinates.

[0019] In a preferred embodiment of the present invention, the step of calculating the target pitch angle of the UAV gimbal based on the spatial relative coordinate system using the arctangent function specifically includes: determining whether the denominator of the coordinate difference in the arctangent function operation logic is less than a preset denominator threshold; when it is determined that the denominator of the coordinate difference is less than the preset denominator threshold, the system stops performing the division operation and sets the target pitch angle to -90°.

[0020] Furthermore, the step of simultaneously extracting multi-source data to generate a data digest to complete the evidence chain loop includes: simultaneously extracting video stream segments from fixed cameras, target images acquired by drones, and coordinate convergence calculation logs; concatenating binary sequences from the video stream segments from fixed cameras, target images acquired by drones, and coordinate convergence calculation logs; using a one-way hash function to perform a concatenation operation on the concatenated binary sequence to generate a fixed-length hash string as a data digest; and packaging the data digest and source data files together to construct a business work order and uploading it to the system database.

[0021] A second aspect of the present invention provides a drone video-linked processing system, comprising:

[0022] The perception calibration module is used to fix the camera's perception of abnormal events and construct the initial line-of-sight vector from the fixed camera coordinate system to the world coordinate system;

[0023] The envelope calculation module is used to set the elevation tolerance range along the initial line-of-sight vector and the digital elevation model, use the fixed camera calibration error covariance matrix to expand the boundary to generate the spatial search envelope area, and calculate the spatial volume parameters of the spatial search envelope area.

[0024] The scheduling decision module is used to substitute the spatial volume parameters of the spatial search envelope as uncertainty penalty terms into the comprehensive scoring function to execute dynamic scheduling decisions, and guide the UAV to the target area when the comprehensive scheduling score is greater than the dispatch threshold.

[0025] The coordinate convergence module is used to calculate the common perpendicular line between the initial line-of-sight vector and the secondary line-of-sight vector after the UAV generates the secondary line-of-sight vector, and obtain the coordinates of the midpoint of the common perpendicular line as the accurate three-dimensional geographic coordinates after convergence.

[0026] The directional evidence storage module is used to reverse engineer the target attitude angle using precise 3D geographic coordinates to perform dual-field-of-view collaborative orientation control, and simultaneously extract multi-source data to generate data summaries to complete the evidence chain loop.

[0027] This invention provides a method and system for coordinated video processing of unmanned aerial vehicles (UAVs). It has the following beneficial effects:

[0028] 1. This invention generates a spatial search envelope region by combining a digital elevation model with error covariance, and calculates the midpoint of the common perpendicular using a secondary observation line-of-sight vector generated by an UAV. This feature utilizes a dual-view spatial intersection mechanism to overcome the lack of depth information in monocular cameras and reduce the elevation interference of complex terrain on spatially mapped coordinates.

[0029] 2. This invention incorporates the volume parameters of the spatial search envelope as an uncertainty penalty term into the comprehensive scoring function to perform dynamic scheduling decisions. This feature transforms geometric positioning errors into quantitative constraints for task assignment. When the target location is extremely uncertain, the task is postponed, avoiding large-scale blind searches by UAVs and reducing power waste caused by ineffective flights.

[0030] 3. This invention utilizes converged, precise 3D geographic coordinates to calculate the attitude angles of the fixed camera and the UAV gimbal, driving the heterogeneous devices to synchronously align their fields of view with the target's physical location. Based on collaborative orientation, the system extracts video and image data from multiple perspectives, performs hash concatenation operations to generate data digests, ensuring the consistency and immutability of the linked evidence collection data. Attached Figure Description

[0031] Figure 1 This is a system architecture diagram of the present invention;

[0032] Figure 2 This is a flowchart of the method steps of the present invention;

[0033] Figure 3 This is a schematic diagram illustrating the initial line-of-sight vector construction principle of the present invention.

[0034] Figure 4 This is a schematic diagram illustrating the principle of generating the spatial search envelope region in this invention.

[0035] Figure 5 This is a flowchart of the dynamic scheduling decision-making process of the present invention;

[0036] Figure 6 This is a schematic diagram illustrating the precise spatial coordinate convergence principle of the present invention.

[0037] Figure 7 This is a flowchart illustrating the collaborative targeting and evidence chain solidification process of this invention.

[0038] Figure 8 This is a three-dimensional data path diagram of line intersection and coordinate convergence according to an embodiment of the present invention. Detailed Implementation

[0039] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] See attached document Figure 1 The present invention provides a drone video linkage processing system, which may include: a perception calibration module, an envelope calculation module, a scheduling decision module, a coordinate convergence module, and a directional evidence storage module.

[0041] The perception and calibration module receives video stream data from a fixed camera. It extracts the pixel coordinates of abnormal targets in the image coordinate system using an image recognition algorithm. The module also acquires the yaw angle, pitch angle, and geographic coordinate parameters of the fixed camera to construct the line-of-sight vector from the fixed camera coordinate system to the world coordinate system.

[0042] The envelope calculation module is connected to the perception calibration module. The envelope calculation module extracts the system's preset digital elevation model (DEM) data and calculates the elevation of the intersection point between the line-of-sight vector and the surface of the DEM. The envelope calculation module sets an elevation tolerance range along the line-of-sight vector, combines it with the covariance matrix of the fixed camera calibration error to generate a spatial search envelope region, and calculates the spatial volume parameters of this spatial search envelope region.

[0043] The scheduling decision module receives the spatial volume parameters of the spatial search envelope region output by the envelope solving module. The scheduling decision module then uses these volume parameters as an uncertainty penalty term in the comprehensive scoring function for calculation. Based on the calculated score, the scheduling decision module generates a UAV navigation route or stores the task in a task deferred queue.

[0044] The coordinate convergence module receives the secondary observation line-of-sight vector generated by the UAV in the edge region of the envelope area. It then calculates the common perpendicular between the fixed camera's line-of-sight vector and the secondary observation line-of-sight vector, solving for the parameters of this common perpendicular on the two line-of-sight vectors. Finally, the module obtains the coordinates of the midpoint of this common perpendicular and uses these coordinates as the converged, precise 3D geographic coordinates.

[0045] The targeted evidence storage module generates collaborative control commands using the converged, precise 3D geographic coordinates. It then sends gimbal drive commands and focus adjustment commands to both the fixed camera and the drone. Simultaneously, the module extracts the global video stream from the fixed camera, close-up images from the drone, and coordinate convergence calculation logs, performs hash operations to generate data digests, and solidifies the evidence chain data.

[0046] See attached document Figure 2 This invention provides a method for coordinated processing of drone video, comprising the following steps:

[0047] S10, the fixed camera senses abnormal events and constructs the initial line-of-sight vector from the fixed camera coordinate system to the world coordinate system;

[0048] S20, combine the digital elevation model and the initial line-of-sight vector to generate the spatial search envelope region, and calculate the spatial volume parameters of the spatial search envelope region.

[0049] S30 substitutes the spatial volume parameters of the spatial search envelope into the comprehensive scoring function to execute dynamic scheduling decisions and guide the UAV to the target area;

[0050] S40: The UAV generates a secondary observation line-of-sight vector, calculates the midpoint of the common perpendicular based on the heterogeneous line-of-sight vector spatial intersection algorithm, and completes accurate spatial coordinate convergence.

[0051] The S50 utilizes precise spatial coordinates to perform dual-field-of-view collaborative orientation control, extracts multi-source data to generate data summaries, and completes the evidence chain closure.

[0052] The technical solution of the present invention will be described in detail below with reference to specific embodiments.

[0053] See attached document Figure 3 The specific implementation method for constructing the initial line-of-sight vector in step S10 includes:

[0054] S101, the perception calibration module receives real-time video stream data output by a fixed camera and performs target detection on the video frame images to lock abnormal target events.

[0055] Specifically, the perception and calibration module calculates the bounding box of the abnormal target in the current frame, extracts the geometric center position of the bounding box, and generates the center pixel coordinates of the target in the image coordinate system. .

[0056] In this embodiment, the perception calibration module incorporates a target detection neural network model based on the YOLO architecture. The input data dimension of this model is a fixed-resolution RGB three-channel image tensor. The preprocessing logic includes frame extraction and normalization of the real-time video stream. The internal hierarchical structure of the model sequentially includes a backbone network for feature extraction, a neck network for feature fusion, and a detection head for outputting prediction results.

[0057] The model's output corresponds to the classification confidence and bounding box coordinates of abnormal events (such as smoke, fire, or illegal buildings) in specific business scenarios. Its training steps include: collecting historical fixed surveillance video data and labeling abnormal target bounding boxes as tags; calculating the classification loss using the cross-entropy loss function; calculating the bounding box regression loss using the intersection-over-union loss function; and updating the model weight parameters through the backpropagation algorithm until the loss function converges.

[0058] S102, the perception and calibration module acquires the absolute geographic coordinates of the fixed camera installation, and converts the longitude, latitude, and elevation data into a local three-dimensional rectangular coordinate system, denoted as three-dimensional world coordinates. Simultaneously, the device interface reads the current gimbal mechanical status data of the fixed camera to obtain the real-time yaw angle. With real-time pitch angle .

[0059] Subsequently, the perception calibration module calls the intrinsic parameter matrix, which has been pre-determined through checkerboard calibration and stored in the configuration file, to set the center pixel coordinates. Transform to a fixed camera coordinate system. Utilize real-time yaw angle. With real-time pitch angle Construct a rotation matrix from the fixed camera coordinate system to the world coordinate system, transform the vectors in the fixed camera coordinate system to the world coordinate system, and then perform vector normalization to obtain the unit direction vector of the fixed camera pointing at the abnormal target. .

[0060] In practice, the camera intrinsic parameter matrix is ​​determined through a pre-defined checkerboard calibration process, and its specific values ​​are stored fixedly in the system configuration file. The principle behind the aforementioned vector transformation lies in establishing a ray mapping relationship from the two-dimensional pixel plane to the three-dimensional real physical space.

[0061] Combining three-dimensional world coordinates With unit direction vector The perception calibration module establishes the line-of-sight vector equation in three-dimensional space. Initial line-of-sight vector. The parameterized equation is:

[0062] ;

[0063] In the formula, Let be the three-dimensional coordinates of any point on the line of sight; To fix the three-dimensional world coordinates of the camera; The unit direction vector for fixing the camera's direction towards the abnormal target; The spatial distance variable parameter along the line of sight is configured with the following value range: , This is the maximum detection distance preset for the system. The perception calibration module uses this distance to convert two-dimensional pixel features into three-dimensional spatial orientation.

[0064] See attached document Figure 4 The specific implementation method for generating the spatial search envelope region and calculating the volume parameters in step S20 includes:

[0065] S201, the envelope calculation module obtains the initial line-of-sight vector and the system's preset digital elevation model data.

[0066] The spatial ray tracing algorithm is used to calculate the three-dimensional intersection point between the initial line-of-sight vector and the surface of the digital elevation model, and the reference elevation of this intersection point is obtained. A preset terrain difference threshold is superimposed along the direction of the initial line-of-sight vector. Set elevation tolerance range .

[0067] S202, the envelope calculation module, based on the elevation tolerance range, inversely solves for the lower bound of the ray distance parameter. With the upper realm The corresponding 3D vector line segment is then extracted. Based on the preset fixed camera calibration error covariance matrix, boundary expansion calculations are performed on this line segment.

[0068] Specifically, extract the comprehensive angular error standard deviation from the covariance matrix. Using the three-dimensional vector line segment as the central axis, calculate each distance parameter. Lateral expansion radius at the location A three-dimensional spatial search envelope is generated based on this lateral expansion radius.

[0069] S203, the envelope calculation module performs volume integration on the generated 3D spatial search envelope region, and calculates the spatial volume parameters of the spatial search envelope region. The calculation formula is:

[0070] ;

[0071] In the formula, Spatial volume parameters of the spatial search envelope region; Pi is a constant. For integration variables; and These are the lower and upper bounds of the ray distance parameter obtained from the inverse kinematics, respectively; Distance parameter The lateral expansion radius at that location. The envelope calculation module calculates the spatial volume parameters of the spatial search envelope region. The output serves as a quantization input parameter characterizing the geometric uncertainty.

[0072] See attached document Figure 5 The specific implementation method for the process of performing dynamic scheduling decisions and guiding the UAV to the target area in step S30 includes:

[0073] S301, The scheduling decision module obtains the spatial volume parameters of the spatial search envelope region. And calculate the uncertainty penalty term based on the logarithmic mapping function. Uncertainty penalty term The calculation formula is:

[0074] ;

[0075] In the formula, This is an uncertainty penalty term; The system's preset penalty adjustment coefficient has a configuration range of 0.5 to 2.0; It is the natural logarithm function; It serves as a baseline reference volume constant, the value of which is determined based on the effective coverage volume of a single frame of the UAV model at a specific altitude.

[0076] S302, the scheduling decision module collects abnormal event attribute data, sensor-based identification data, and UAV status data to calculate a comprehensive scheduling score. The calculation formula is:

[0077] ;

[0078] In the formula, For comprehensive scheduling scoring; , , These are the corresponding normalized weight coefficients, satisfying ; This refers to the risk level score of abnormal events obtained based on the pre-set business dictionary mapping; The target recognition confidence parameter output by the image recognition algorithm; This is an uncertainty penalty term.

[0079] The scheduling decision module scores the availability of drone resources and extracts the remaining battery power of the current drone. 1. Estimated flight time to reach the target area and the safe reserve power for drones .

[0080] In one embodiment, the average power consumption per unit time is determined based on the historical flight logs of the corresponding drone model. Calculate the estimated power consumption according to the following formula. :

[0081] ;

[0082] in, and , and Use consistent units of electrical charge; when electrical charge is expressed as a percentage of battery capacity. The unit is the percentage of electricity consumed per unit of time.

[0083] The scheduling decision module calculates the difference in available power after task execution using the following formula. :

[0084] ;

[0085] Reserved power for safety This should at least cover the power required for the drone to perform its return, landing, or entry into a safe area. Further, the difference in available power should be considered. Normalize to the closed interval [0,1] to obtain the UAV resource availability score. :

[0086] ;

[0087] In the formula, The minimum available power difference required to allow the task to be performed; This is the reference sufficient battery difference for the corresponding drone under the current mission type; This indicates the net remaining amount of electricity available that exceeds the minimum requirement. This represents the span of the energy difference range from the minimum permissible state to the reference sufficient state; Indicates when Output 0 when Time output ,when Output 1 when the time is right. and Pre-set parameters based on drone model, mission distance, environmental conditions, and historical valid flight logs.

[0088] when When the drone's resource availability score is set to 0, the drone is marked as unsuitable for performing the current task; when At that time, the drone resource availability score is set to 1.

[0089] Subsequently, the scheduling decision module will calculate the comprehensive scheduling score. With the preset distribution threshold The distribution threshold is compared. The score is set based on the average score of historical valid scheduling logs. If the judgment... When the task is in a deferred state, the scheduling decision module stores the task in the task deferred queue; if it determines... If the task is ready, it will be marked as executable and the drone delivery process will begin.

[0090] S303, For tasks marked as executable, the scheduling decision module extracts the boundary vertex coordinates of their corresponding spatial search envelope.

[0091] The boundary vertex coordinates are converted into latitude, longitude, and altitude waypoint commands that the UAV flight control system can recognize. Flight paths and takeoff commands are then issued via network communication links to guide the UAV to the target area.

[0092] See attached document Figure 6 The specific implementation method for the process of achieving accurate spatial coordinate convergence in step S40 includes:

[0093] S401, the coordinate convergence module acquires the UAV's three-dimensional world coordinates. The system acquires real-time images of the target area using an airborne camera and generates at least one candidate target within the image projection range corresponding to the spatial search envelope. The coordinate convergence module obtains the abnormal target category, target recognition time, target appearance features, and spatial search envelope output from the fixed camera. Using this information as a target matching benchmark, it sequentially performs category consistency screening, spatial location constraints, appearance feature similarity calculation, and temporal correlation judgment on each candidate target.

[0094] In one embodiment, the comprehensive matching score of the candidate target Calculate according to the following formula:

[0095] ;

[0096] In the formula, For the target category consistency score, The spatial location matching score of the relative spatial search envelope region for candidate targets is calculated. This refers to the similarity in appearance features between candidate targets and anomalous targets identified by a fixed camera. The score is based on time correlation; among which , The normalized weighting coefficients corresponding to each score.

[0097] When the overall matching score When the value is greater than or equal to the preset matching threshold, the corresponding candidate target is identified as the same abnormal target identified by the fixed camera; when there are multiple candidate targets that meet the preset matching threshold, the candidate target with the highest comprehensive matching score is selected; when there are no candidate targets that meet the preset matching threshold, the drone is controlled to adjust the observation position or gimbal attitude and then reacquire the image, and the subsequent spatial coordinate convergence calculation is not performed for the time being.

[0098] The coordinate convergence module extracts the geometric center pixel coordinates of the bounding box of the identified anomalous target and, combined with real-time fuselage and gimbal attitude data, calculates the secondary unit direction vector of the UAV pointing towards the anomalous target. Based on three-dimensional world coordinates With secondary unit direction vector Construct a secondary observation line-of-sight vector for the UAV.

[0099] S402, due to the spatial measurement differences of multiple source sensors, the initial line-of-sight vector and the secondary observation line-of-sight vector are in a state of non-planar straight lines in three-dimensional space.

[0100] The coordinate convergence module calculates the spatial common perpendicular of the two line-of-sight vectors and solves for the intersection parameters of this common perpendicular on the initial line-of-sight vector. And the intersection parameters on the secondary observation line-of-sight vector. .

[0101] S403, based on the acquired intersection parameters, the coordinate convergence module calculates the coordinates of the midpoint of the common perpendicular. Midpoint coordinates The calculation formula is:

[0102] ;

[0103] In the formula, The calculated three-dimensional coordinates of the midpoint; The fixed camera's 3D world coordinates are calculated in the pre-processing step; The initial unit direction vector of the fixed camera; For the three-dimensional world coordinates of the drone; This is the secondary unit direction vector of the UAV; and These are the parametric solutions for the common perpendicular on the initial line-of-sight vector and the secondary observation line-of-sight vector, respectively.

[0104] The coordinate convergence module calculates the three-dimensional coordinates of the midpoint. The final, accurate 3D geographic coordinates are then output to the directional evidence storage module.

[0105] See attached document Figure 7 The specific implementation method for the process of performing dual-field-of-view cooperative orientation control and completing the evidence chain closure in step S50 includes:

[0106] S501, the targeted evidence storage module obtains precise three-dimensional geographic coordinates. and the real-time three-dimensional world coordinates of the drone In a local three-dimensional rectangular coordinate system, the coordinate difference between the UAV pointing to the precise three-dimensional geographic coordinates is defined as:

[0107] ;

[0108] ;

[0109] ;

[0110] Target yaw angle of the drone gimbal Calculate according to the following formula:

[0111] ;

[0112] Target pitch angle of the drone gimbal Calculate according to the following formula:

[0113] ;

[0114] In the formula, The arctangent function is used to determine the angular quadrant based on the difference between two coordinates; These are its coordinate components in the local three-dimensional Cartesian coordinate system; These are its coordinate components in the local three-dimensional Cartesian coordinate system; These represent the coordinate differences of the UAV pointing to the precise 3D geographic coordinates in the X, Y, and Z axes, respectively. and These are the yaw installation compensation angle and pitch installation compensation angle of the UAV gimbal relative to the local three-dimensional Cartesian coordinate system, respectively, and their values ​​are obtained through gimbal installation calibration.

[0115] Define the horizontal distance component as:

[0116] ;

[0117] The system presets a denominator threshold. The denominator threshold uses the same length unit as the world coordinates and is determined based on coordinate measurement accuracy, equipment calibration error, and floating-point arithmetic accuracy. ≥ When, the calculation is performed according to the above target pitch angle formula; when And the precise three-dimensional geographic coordinates are located below the drone, that is... At that time, the system stops performing the division operation and sets the target pitch angle to -90 degrees.

[0118] The orientation storage module normalizes the target yaw angle and target pitch angle to the range of mechanical angles allowed by the UAV gimbal and sends them to the flight control system.

[0119] The targeted evidence storage module uses the real-time 3D world coordinates of the UAV. With precise three-dimensional geographic coordinates Calculate the straight-line spatial distance between the drone and the anomalous target. :

[0120] ;

[0121] In one embodiment, the spatial distance and optical focal length of the drone's zoom camera are pre-calibrated. At multiple calibration distances, the focal lengths corresponding to achieving a preset image proportion for a target of a preset size are recorded, and a spatial distance and optical focal length lookup table is established based on the calibration results. The directional evidence storage module uses linear spatial distance... Query the corresponding initial target focal length .

[0122] When the linear spatial distance When the target is located between two adjacent calibration distances, linear interpolation is used to determine the initial target focal length. Let the two adjacent calibration distances be... and The corresponding focal lengths are respectively and Then the initial target focal length Determined according to the following formula:

[0123] ;

[0124] In the formula, To correspond to the first calibration distance The calibrated focal length; To correspond to the second calibration distance The calibrated focal length; It represents the proportion of the actual distance within two adjacent calibrated distance intervals.

[0125] The directional evidence storage module will set the initial target focal length. Limiting the minimum permissible focal length of a drone zoom camera With maximum permissible focal length In between, it sends an initial focus adjustment command to the drone's zoom camera.

[0126] After the drone's zoom camera acquires a preliminary image at the initial target focal length, the orientation and evidence storage module calculates the size proportion of the abnormal target's bounding box in the current image. When the target size proportion is less than the preset lower limit, the focal length is increased; when the target size proportion is greater than the preset upper limit, the focal length is decreased; when the target size proportion is within the preset target proportion range, the focal length adjustment is stopped and the target image is acquired.

[0127] In this control process, the initial target focal length is the initial set value of the target focal length of the UAV zoom camera. After the closed-loop adjustment based on the image size ratio, the final locked target focal length is formed. Thus, the directional evidence storage module can determine the initial target focal length according to the mapping relationship between spatial distance and optical focal length, and perform closed-loop focal length adjustment according to the actual size of the abnormal target in the image.

[0128] S502, the directional evidence storage module extracts the static 3D world coordinates of the fixed camera:

[0129] ;

[0130] And define the coordinate difference value for a fixed camera pointing to precise 3D geographic coordinates as:

[0131] ;

[0132] ;

[0133] ;

[0134] The true yaw angle of a fixed camera pointing to precise 3D geographic coordinates and the true pitch angle Calculate according to the following formulas respectively:

[0135] ;

[0136] ;

[0137] In the formula, The static 3D world coordinates of the fixed camera; These are its coordinate components in the three-dimensional coordinate system; These represent the coordinate differences of a fixed camera pointing to precise 3D geographic coordinates in the X, Y, and Z axes. and These are the yaw installation compensation angle and pitch installation compensation angle determined during the installation and calibration of the fixed camera; This is the straight-line distance between the fixed camera and the precise 3D geographic coordinates projected onto the horizontal plane. The orientation and evidence storage module converts the calculated true yaw angle and true pitch angle to the device coordinate system of the fixed camera's pan-tilt unit, and performs angle truncation or equivalent angle transformation according to the pan-tilt unit's mechanical limits. It then encapsulates this into a pan-tilt unit drive command and sends it out to drive the fixed camera's field of view center to align with the precise 3D geographic coordinates.

[0138] S503, the targeted evidence storage module synchronously extracts video stream segments from fixed cameras. Target images acquired by drones and coordinate convergence calculation log To ensure the temporal correspondence between data from different sources, both video streams from fixed cameras and UAV target images carry acquisition timestamps generated by the system's unified time synchronization; the coordinate convergence calculation log records at least the initial line-of-sight vector, secondary observation line-of-sight vector, common perpendicular parameters, midpoint coordinates, algorithm version, and calculation time.

[0139] The targeted evidence storage module generates business metadata. Business metadata should include at least the unique identifier of the business work order, the identifier of the fixed camera equipment, the identifier of the drone equipment, the start and end time of data collection, the precise three-dimensional geographic coordinates, and the algorithm version number.

[0140] The targeted evidence storage module performs binary concatenation and splicing on business metadata and three types of source data according to a preset and fixed data order, and generates a data digest according to the following formula. :

[0141] ;

[0142] In the formula, The output is a fixed-length hash string; It is a one-way hash function; This is wide-angle global video stream data; For business metadata; Target image data acquired by the drone; Log data is calculated for coordinate convergence; This represents the cascading and concatenation operation of data sequences.

[0143] In a preferred embodiment, the system utilizes a signature private key pre-installed in the secure cryptographic module. Data summary Perform a digital signature to obtain the digest signature value:

[0144] ;

[0145] In the formula, For use with a pre-set signature private key The digital signature algorithm function.

[0146] The system will summarize the data. Summary signature value The unique identifier of the business work order and the time of its summary generation are written to a trusted storage area separate from the source data file or to a log database that only allows append writing, and the source data file, business metadata, data summary and summary signature value are associated with the same business work order.

[0147] When verifying evidence data, the system reassembles the business metadata and source data in the same data order and recalculates the hash value. The recalculated hash value is compared with the data digest in the trusted storage area, and the digest signature value is verified using the public key corresponding to the signing private key. When the hash values ​​match and the signature verification passes, it confirms that the source data in the business work order has not changed since the digest was generated. When the hash values ​​do not match or the signature verification fails, a data integrity exception message is output.

[0148] Therefore, the constructed business work order has the ability to associate data sources, verify integrity, and detect tampering, and is uploaded to the system database to complete the closed-loop process of the entire video linkage processing.

[0149] To further clarify the collaborative working process of the technical solution of this invention, the following will illustrate it through a specific application scenario example of forest fire monitoring in a township.

[0150] See attached document Figure 8 In a routine patrol scenario in a rural forest area, a fixed camera deployed at the foot of a mountain detected unusual smoke on a distant hillside. The system extracts the fixed camera's 3D world coordinates, such as the starting coordinates (0,0,50) shown in the image, and combines this with the current pan-tilt-zoom (PTZ) orientation to generate an initial line-of-sight vector pointing towards the smoke. Since the monitoring direction is undulating mountainous terrain, monocular vision cannot accurately determine the true depth of the smoke. The system extracts digital elevation model data for the area, calculates ray distance parameters along the direction of the initial line-of-sight vector within a set elevation tolerance, and then generates a 3D spatial search envelope in an area far from the fixed camera.

[0151] The system calculates the spatial volume parameters of the 3D search envelope and substitutes them into the comprehensive scoring function to calculate the uncertainty penalty term. Due to the high initial risk level of the fire event, although the envelope volume introduces a certain penalty score, the comprehensive scheduling score is still higher than the system's dispatch threshold. The scheduling decision module then converts the boundary coordinates of the envelope into waypoint instructions and dispatches the UAV to take off and execute the mission.

[0152] See attached document Figure 8 Following waypoint instructions, the UAV flies to the edge of the envelope area, reaching its current 3D world coordinates (1100, 700, 400). The UAV uses its onboard camera's target matching algorithm to lock onto smoke pixels and generate a secondary observation line-of-sight vector. In the 3D coordinate system, the initial line-of-sight vector of the fixed camera and the UAV's secondary line-of-sight vector appear as two non-intersecting skew lines.

[0153] The system extracts the spatial equations of the two line-of-sight vectors and directly solves for their common perpendicular in three-dimensional space. By calculating the parametric solutions at both ends of the common perpendicular, the system obtains the midpoint of the common perpendicular, yielding the precise three-dimensional geographic coordinates (1010, 815, 205) shown in the figure. This solution process, through dual-view spatial intersection, directly overcomes the problem of distance misjudgment caused by terrain undulations and achieves accurate convergence of spatial coordinates.

[0154] After acquiring precise 3D geographic coordinates, the system synchronously sends coordinated orientation commands to both the fixed camera and the drone. The fixed camera uses the inverse kinematics-derived true yaw and pitch angles to fine-tune its gimbal, eliminating mechanical errors and locking the field of view center to coordinates (1010, 815, 205). The drone's gimbal synchronously adjusts its attitude and extends its focal length to capture close-up images of this coordinate point. The system captures video images and coordinate logs from both ends, stitches them together, and generates a unique data digest through hash calculation, forming an immutable fire evidence chain work order, thus completing the entire video-linked response workflow.

[0155] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these specific embodiments are merely illustrative. Those skilled in the art can omit, substitute, and modify the details of the above methods and systems in various ways without departing from the principles and essence of the present invention. For example, combining the above method steps to perform substantially the same function and achieve substantially the same result using substantially the same method falls within the scope of the present invention.

Claims

1. A method for coordinated processing of drone video, characterized in that, Includes the following steps: A fixed camera detects abnormal events and constructs an initial line-of-sight vector from the fixed camera coordinate system to the world coordinate system; An elevation tolerance range is set along the initial line-of-sight vector and combined with the digital elevation model. The spatial search envelope is generated by boundary expansion using the fixed camera calibration error covariance matrix. The spatial volume parameters of the spatial search envelope are then calculated. The spatial volume parameter of the spatial search envelope is used as an uncertainty penalty term and substituted into the comprehensive scoring function to perform dynamic scheduling decision. If the comprehensive scheduling score is greater than the dispatch threshold, the target area is determined according to the spatial search envelope and the UAV is guided to the target area. After the UAV reaches the target area, a secondary observation line-of-sight vector is generated. Calculate the common perpendicular line between the initial line-of-sight vector and the secondary observation line-of-sight vector, and obtain the coordinates of the midpoint of the common perpendicular line as the accurate three-dimensional geographic coordinates after convergence; The target attitude angle is solved by using the precise three-dimensional geographic coordinates to perform dual-field-of-view collaborative orientation control, and multi-source data is extracted simultaneously to generate a data digest to complete the evidence chain loop.

2. The UAV video linkage processing method according to claim 1, characterized in that, The steps for the fixed camera to sense abnormal events and construct the initial line-of-sight vector from the fixed camera coordinate system to the world coordinate system include: Extract the geometric center of the bounding box of the abnormal target within the video frame image and generate the center pixel coordinates; Obtain the absolute geographic coordinates of the fixed camera installation and convert them into three-dimensional world coordinates, and synchronously read the current real-time yaw angle and real-time pitch angle of the fixed camera; The intrinsic parameter matrix is ​​called to transform the center pixel coordinates to the fixed camera coordinate system. The rotation matrix is ​​constructed using the real-time yaw angle and the real-time pitch angle. The vector in the fixed camera coordinate system is transformed to the world coordinate system and normalized to obtain the unit direction vector. The line-of-sight vector equation is established by combining the three-dimensional world coordinates and the unit direction vector to generate the initial line-of-sight vector.

3. The UAV video linkage processing method according to claim 1, characterized in that, The steps of setting an elevation tolerance range along the initial line-of-sight vector using a digital elevation model, generating a spatial search envelope region by boundary expansion using a fixed camera calibration error covariance matrix, and calculating the spatial volume parameters of the spatial search envelope region include: The reference elevation is obtained by calculating the three-dimensional intersection of the initial line-of-sight vector and the surface of the digital elevation model using a spatial ray tracing algorithm. A preset ground deviation threshold is superimposed along the direction of the initial line-of-sight vector to set the elevation tolerance range. Based on the elevation tolerance range, the lower and upper bounds of the ray distance parameter are solved, the corresponding three-dimensional vector line segments are extracted, the comprehensive angle error standard deviation in the fixed camera calibration error covariance matrix is ​​extracted, and the horizontal expansion radius is calculated with the three-dimensional vector line segments as the central axis to generate the spatial search envelope. Integrate the cross-sectional area formed by the lateral expansion radius along the central axis of the three-dimensional vector line segment, and use the calculated physical volume as the spatial volume parameter of the spatial search envelope region.

4. The UAV video linkage processing method according to claim 1, characterized in that, The step of substituting the spatial volume parameter of the spatial search envelope as an uncertainty penalty term into the comprehensive scoring function to perform dynamic scheduling decision-making, and guiding the UAV to the target area if the comprehensive scheduling score is greater than the dispatch threshold, includes: Establish the ratio relationship between the spatial volume parameters of the spatial search envelope region and the reference volume constant, and calculate the uncertainty penalty term based on the logarithmic mapping function; Collect the risk level score of abnormal events, the confidence parameter of target recognition, and the availability score of UAV resources, and calculate the comprehensive scheduling score by combining the normalized weight coefficient and the uncertainty penalty term; The comprehensive scheduling score is compared with the dispatch threshold. If the comprehensive scheduling score is less than or equal to the dispatch threshold, the scheduling task is stored in the task deferred queue. If the comprehensive scheduling score is greater than the dispatch threshold, the boundary vertex coordinates of the spatial search envelope are extracted and converted into waypoint instructions. A flight path is then issued to guide the UAV to the target area.

5. The UAV video linkage processing method according to claim 4, characterized in that, The steps for collecting the drone resource availability score include: Extract the current remaining battery power of the drone and the estimated flight time to reach the target area; The flight time is converted into an estimated power consumption, and the difference between the current remaining power of the drone and the estimated power consumption is compared. The difference is converted into an availability score within the closed interval [0,1] by a preset normalization function, and the drone resource availability score is output.

6. The UAV video linkage processing method according to claim 1, characterized in that, The steps of generating a secondary observation line-of-sight vector by the UAV, calculating the common perpendicular between the initial line-of-sight vector and the secondary observation line-of-sight vector, and obtaining the midpoint coordinates of the common perpendicular as the converged precise three-dimensional geographic coordinates include: The three-dimensional world coordinates of the UAV are obtained, abnormal targets are locked by visual target matching algorithm, and secondary unit direction vectors are calculated by combining real-time fuselage and gimbal attitude data to construct the secondary observation line-of-sight vector; Based on the spatial analytical geometry solution algorithm, the spatial common perpendicular line of the initial line of sight and the secondary observation line of sight is calculated, and the intersection parameters of the common perpendicular line on the initial line of sight and the intersection parameters on the secondary observation line of sight are obtained. The coordinate positions of the two ends of the common perpendicular are calculated based on the intersection parameters. The coordinates of the midpoint of the line connecting the two ends of the common perpendicular are extracted, and the midpoint coordinates are used as the final converged precise three-dimensional geographic coordinates.

7. The UAV video linkage processing method according to claim 1, characterized in that, The step of using the precise three-dimensional geographic coordinates to inversely determine the target attitude angle to perform dual-field-of-view cooperative orientation control includes: The precise three-dimensional geographic coordinates and the real-time three-dimensional world coordinates of the UAV are obtained. Based on the spatial relative coordinate system, the target yaw angle and target pitch angle of the UAV gimbal are calculated by the arctangent function, and the gimbal attitude adjustment command is issued. The target focal length of the drone zoom camera is determined based on the preset spatial distance and optical focal length correspondence, and a focal length adjustment command is issued to drive the drone zoom camera to acquire the target image. Extract the static 3D world coordinates of the fixed camera, calculate the true yaw angle and true pitch angle of the fixed camera pointing to the precise 3D geographic coordinates based on the spatial coordinate inverse logic, and encapsulate them into a gimbal driving command to drive the fixed camera's field of view center to align with the precise 3D geographic coordinates.

8. The UAV video linkage processing method according to claim 7, characterized in that, When the arctangent function is used to solve for the target pitch angle, the horizontal distance component between the precise three-dimensional geographic coordinates and the three-dimensional world coordinates of the UAV is calculated. When the horizontal distance component is less than a preset denominator threshold and the precise three-dimensional geographic coordinates are located below the UAV, the target pitch angle is set to -90°.

9. The UAV video linkage processing method according to claim 1, characterized in that, The step of simultaneously extracting multi-source data to generate a data digest to complete the evidence chain loop includes: Simultaneously extract video stream segments from the fixed camera, target images acquired by the UAV, and coordinate convergence calculation logs; The video stream segments from the fixed camera, the target images acquired by the UAV, and the coordinate convergence calculation logs are concatenated into binary sequences. A one-way hash function is used to perform a concatenation operation on the concatenated binary sequence to generate a fixed-length hash string as the data digest. The data digest and the source data file are packaged together to form a business work order and uploaded to the system database.

10. A drone video linkage processing system, applied to the drone video linkage processing method according to any one of claims 1-9, characterized in that, include: The perception calibration module is used to fix the camera's perception of abnormal events and construct the initial line-of-sight vector from the fixed camera coordinate system to the world coordinate system; The envelope calculation module is used to set the elevation tolerance range along the initial line-of-sight vector in combination with the digital elevation model, use the fixed camera calibration error covariance matrix to expand the boundary to generate the spatial search envelope region, and calculate the spatial volume parameters of the spatial search envelope region. The scheduling decision module is used to substitute the spatial volume parameter of the spatial search envelope as an uncertainty penalty term into the comprehensive scoring function to perform dynamic scheduling decisions, and guide the UAV to the target area when the comprehensive scheduling score is greater than the dispatch threshold. The coordinate convergence module is used to calculate the common perpendicular line between the initial line-of-sight vector and the secondary line-of-sight vector after the UAV generates the secondary line-of-sight vector, and obtain the midpoint coordinates of the common perpendicular line as the converged precise three-dimensional geographic coordinates. The directional evidence storage module is used to inversely solve the target attitude angle using the precise three-dimensional geographic coordinates to perform dual-field-of-view collaborative orientation control, and simultaneously extract multi-source data to generate data summaries to complete the evidence chain loop.

Images