A high-precision positioning and landing system for unmanned aerial vehicles based on visual guidance
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUANGSHAN UNIV
- Filing Date
- 2026-04-20
- Publication Date
- 2026-06-19
Smart Images
Figure CN122239792A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of unmanned aerial vehicle (UAV) collaborative control technology, and discloses a high-precision positioning and landing system for unmanned shipborne UAVs based on vision guidance. Background Technology
[0002] With the widespread application of cross-domain collaborative technologies between UAVs and unmanned surface vessels (USVs), autonomous landing of UAVs on dynamic shipboard platforms has become a core factor restricting the operational capabilities of such systems. Traditional landing systems mostly rely on single visible light visual guidance, which, while offering high resolution under ideal lighting conditions, is highly susceptible to target loss or identification deviation under complex weather conditions at sea. Even when some systems incorporate infrared assistance, they often remain at a rudimentary stage of image switching or simple weighting, lacking the means to perform deep dynamic fusion of multimodal features. This results in severely insufficient perception accuracy and anti-interference capability of the system for landing markers in harsh environments.
[0003] On the other hand, unmanned surface vessels (USVs) exhibit complex six-degree-of-freedom random motion on the water surface due to wave influence. Existing guidance algorithms often treat the landing platform as a quasi-static or uniform motion model, ignoring the impact of real-time attitude fluctuations of the hull on landing accuracy. More seriously, the system generally suffers from multiple system delays, including image acquisition, data link transmission, algorithm logic operations, and motor mechanical responses. This results in a time lag between the target's perceived pose by the UAV and the actual state of the hull. In dynamic sea conditions, this "perception-execution" time delay deviation can cause oscillations in the UAV's approach trajectory, and even lead to collisions due to pose inaccuracies at the moment of contact with the vessel.
[0004] Existing ship-machine collaborative solutions still lack in terms of time reference unification and system robustness. The lack of high-precision time synchronization between UAVs and unmanned surface vessels (USVs) makes it difficult to align motion feedback data with visual perception data in terms of timing. Furthermore, most systems lack robust degradation and position estimation mechanisms when vision is briefly obstructed or enters a perception blind zone; once vision fails, the system faces the risk of loss of control. In summary, existing technologies still have significant deficiencies in multimodal perception depth, dynamic latency feedforward compensation, and system fault tolerance under extreme sea conditions, making it difficult to meet the all-weather, high-precision autonomous landing requirements of UAVs in highly dynamic water environments. Summary of the Invention
[0005] The purpose of this invention is to provide a high-precision positioning and landing system for unmanned surface vessels (USVs) based on vision guidance. Its core innovation lies in establishing a coupling mechanism of full-link time delay compensation and multimodal dynamic feature fusion. It enhances all-weather perception capabilities by dynamically adjusting the weights of infrared and visible light features using information entropy and edge gradients. It innovatively incorporates the time delay of the entire process of "perception-computation-communication-mechanical response" into the compensation system. Combined with nanosecond-level time synchronization between the ship and the aircraft based on GNSS second pulses, it uses autoregressive moving average and discrete Kalman filter models to perform feedforward prediction of the six degrees of freedom motion of the hull. This fundamentally eliminates the attitude drift caused by system time lag in dynamic environments and realizes high-precision landing control of the entire process of "attitude prediction-follow-approach-crest contact" under highly dynamic sea conditions.
[0006] The objective of this invention can be achieved through the following technical solutions: A vision-guided high-precision positioning and landing system for unmanned surface vessels (USVs) includes the following modules: The multimodal visual perception and fusion module, set on the UAV, includes a visible light camera and an infrared thermal imaging camera. It is used to acquire the target's visible light image and infrared thermal imaging image at the same time stamp, and uses a cross-modal feature fusion network to align and extract features from the two images, outputting the fused physical feature points of the UAV landing guidance mark. The shipborne dynamic status feedback module is set on the unmanned vessel and is used to collect the six degrees of freedom motion data of the unmanned vessel on the water surface caused by the wave effect in real time, and to send the six degrees of freedom motion data and the corresponding timestamp to the unmanned vessel through a wireless communication link. The latency compensation and pose feedforward prediction module is located on the UAV and is communicatively connected to the multimodal visual perception and fusion module and the shipborne dynamic state feedback module, respectively. It is used to calculate the UAV's current initial relative pose based on the fused physical feature points, calculate the total system latency from image acquisition to control command activation, and input the total system latency and the received six-degree-of-freedom motion data into a preset ship motion prediction model. It calculates the expected motion offset of the landing guidance marker at the moment the command takes effect, and adds the expected motion offset to the initial relative pose to generate a high-precision target pose after dynamic compensation. A high-precision landing control module is installed on the UAV and connected to the time delay compensation and pose feedforward prediction module. It is used to receive the high-precision target pose, generate dynamic follow-approach commands, and control the UAV to land on the unmanned vessel.
[0007] Preferably, the multimodal visual perception and fusion module includes a feature-level dynamic fusion network; the module is specifically used for: The first information entropy value and the first edge gradient value of the visible light image are calculated in real time, and the second information entropy value and the second edge gradient value of the infrared thermal imaging image are calculated. The product of the first information entropy value and the first edge gradient value is used as the visible light reference value, and the product of the second information entropy value and the second edge gradient value is used as the infrared reference value. Calculate the proportions of the visible light reference value and the infrared reference value in the sum of the two values respectively, and use the corresponding proportions as the network weight coefficients of the visible light feature branch and the infrared feature branch. Then, according to the network weight coefficients, the feature maps extracted by the feature-level dynamic fusion network from the two images are spliced and weighted to output the fused feature map.
[0008] Preferably, the landing guidance marker includes a visual geometric pattern and an active infrared radiation array; the multimodal visual perception and fusion module extracts physical feature points specifically including: The corner pixel coordinates corresponding to the visual geometric pattern are extracted from the visible light channel data of the fused feature map, and the centroid pixel coordinates corresponding to the active infrared radiation array are extracted from the infrared channel data of the fused feature map. The deviation between all extracted pixel coordinates and the corresponding positions in the pre-stored landing guidance mark physical size topology map is calculated. Mismatched pixel coordinates with deviations greater than a preset deviation threshold are removed, and the remaining pixel coordinates are output as the fused physical feature points.
[0009] Preferably, both the shipborne dynamic status feedback module and the multimodal visual perception and fusion module are equipped with independent timing clocks that receive the same second pulse signal from the same global navigation satellite system. The multimodal visual perception and fusion module records the first absolute timestamp of the local time clock and appends it to the image data stream at the instant the image is exposed and acquired. The shipborne dynamic state feedback module records the second absolute timestamp of the local time clock and appends it to the six-degree-of-freedom motion data at the instant the sensor samples. The time delay compensation and pose feedforward prediction module aligns the image data and the ship motion data by comparing and matching the first absolute timestamp and the second absolute timestamp.
[0010] Preferably, the total system delay calculated by the delay compensation and pose feedforward prediction module is the sum of all time intervals experienced by the system during a single closed-loop control process; The specific time intervals include: the acquisition duration for the multimodal camera to complete the exposure of a single frame image and read it into memory; the execution duration of the algorithm for image fusion and initial pose calculation; the transmission duration for the unmanned vessel to send data frames to the UAV via the wireless communication link; and the mechanical and physical response duration for the high-precision landing control module to issue approach control commands to the UAV rotor motor to generate corresponding lift changes.
[0011] Preferably, the preset hull motion prediction model includes an autoregressive moving average model and a discrete Kalman filter. The time delay compensation and pose feedforward prediction module inputs the six-degree-of-freedom motion data continuously received within a set time window as historical samples into the autoregressive moving average model to fit the motion change curves of the unmanned vessel caused by the water surface waves, resulting in periodic roll, pitch, and heave. The motion change curves are then input into the discrete Kalman filter as state transition parameters to recursively calculate the expected spatial coordinates and expected attitude angles of the unmanned vessel deck in the world coordinate system at future time nodes after adding the total system time delay to the current time node.
[0012] Preferably, the specific steps of the time delay compensation and pose feedforward prediction module in generating the high-precision target pose are as follows: A reference coordinate system is established with the center of the unmanned vessel deck at the current moment as the origin, and the initial relative pose is mapped to the reference coordinate system. Based on the calculated expected spatial coordinates and expected attitude angles at the future time node, the three-dimensional translation vector and three-dimensional rotation matrix generated by the unmanned vessel deck from the current time to the future time node are calculated as the expected motion offset. The three-dimensional translation vector and three-dimensional rotation matrix are applied as spatial transformation parameters to the initial relative pose to generate the final relative spatial coordinates and yaw angle that compensate for the physical displacement error of the unmanned vessel, which are then used as the high-precision target pose.
[0013] Preferably, the high-precision landing control module divides the landing process into an approach phase, a follow-up hovering phase, and a touchdown phase. During the hovering phase, the high-precision landing control module continuously analyzes the vertical velocity component and vertical acceleration component in the received six-degree-of-freedom motion data. When it detects that the absolute value of the vertical velocity component is less than a preset zero velocity threshold and the value of the vertical acceleration component is negative, it determines that the unmanned vessel is at the crest of the wave motion and immediately triggers the touch-down phase at that moment, controlling the UAV to descend vertically.
[0014] Preferably, the delay compensation and pose feedforward prediction module is equipped with fault-tolerant degradation evaluation logic. When the number of fused physical feature points extracted by the multimodal visual perception and fusion module is less than a preset matching number threshold, and the number of consecutive frames in which this state occurs reaches a preset frame number threshold, visual perception is determined to fail. In the case of visual perception failure, the time delay compensation and pose feedforward prediction module locks the initial relative pose successfully calculated in the last frame before the visual perception failure. Based on this, and combined with the self-motion data output by the UAV's onboard inertial measurement unit and the six-degree-of-freedom motion data continuously transmitted back by the shipboard dynamic state feedback module, the relative position relationship between the UAV and the unmanned ship is calculated frame by frame using the dead reckoning algorithm, and then output to the high-precision landing control module.
[0015] Preferably, the shipborne dynamic state feedback module includes a shipborne inertial measurement unit, a dual-antenna real-time dynamic differential positioning receiver, and an ultra-wideband communication transmitter. The inertial measurement unit outputs inertial data including three-axis acceleration and three-axis angular velocity, and the differential positioning receiver outputs three-dimensional absolute coordinates and heading angle data. The shipborne dynamic state feedback module performs multi-sensor Kalman filtering fusion on the inertial data and absolute coordinates to generate the six-degree-of-freedom motion data, which is then transmitted to the UAV via the ultra-wideband communication transmitter.
[0016] The beneficial effects of this invention are: This invention, through dynamic weighted fusion of multimodal features, enables the system to effectively overcome the impact of extreme operating environments such as strong light reflection from the sea surface, low illumination, dense fog, and complex background interference on a single visual source. This ensures that the UAV can maintain stable and high-precision locking on landing markers under various weather conditions and lighting environments, significantly expanding the operating window of the UAV-ship collaborative system.
[0017] By performing in-depth calculation and feedforward prediction of the time delay of the entire system, this invention completely eliminates the problem of perception data failure caused by image processing and communication lag. This enables the UAV to accurately predict and compensate for the hull's motion deviation in the future, significantly reducing the dynamic tracking error of the relative motion between the UAV and the ship, and ensuring the smoothness of the approach process and the consistency of the landing position.
[0018] This invention intelligently captures the "crest" of wave motion to trigger the touchdown command, effectively avoiding the collision risk caused by violent sea surface fluctuations. Combined with the dead reckoning downgrade protection mechanism after visual failure, it minimizes the probability of damage to precision equipment at the moment of landing, ensuring the system's survival rate and landing success rate in complex sea conditions and under abnormal perception conditions. Attached Figure Description
[0019] Figure 1 This is a schematic diagram of the structure of a vision-guided unmanned shipborne drone high-precision positioning and landing system according to the present invention. Detailed Implementation
[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0021] Example: Figure 1 As shown, the present invention provides a vision-guided high-precision positioning and landing system for unmanned surface vessels, which mainly consists of two parts: an airborne system for the unmanned aerial vehicle (UAV) and a shipborne system for the unmanned surface vessel.
[0022] The UAV is equipped with a high-performance embedded computing platform, a high-frame-rate visible light camera, an infrared thermal imaging camera, and a wireless ultra-wideband receiver module. The unmanned surface vessel is equipped with a high-precision inertial measurement unit, a dual-antenna RTK-GNSS positioning system, an ultra-wideband communication transmitter, and a composite landing guidance marker located in the center of the landing deck.
[0023] The multimodal visual perception and fusion module is located on the UAV and is equipped with a visible light camera and an infrared thermal imaging camera that have undergone hardware-level clock synchronization and spatial calibration. In complex marine environments, strong light reflection from the sea surface can easily lead to overexposure of visible light images, while sea fog reduces visible light contrast; although infrared images have strong penetrating power and are unaffected by lighting conditions, they lack geometric texture details. This invention employs a feature-level dynamic fusion algorithm based on real-time image quality assessment.
[0024] When two cameras are at the same exposure timestamp The resolutions obtained are all Visible light image matrix and infrared image matrix Subsequently, the heterogeneous computing units within the system first perform real-time quantitative evaluation of the effective information content of the two images. The ingenuity of this invention lies in the introduction of a two-factor evaluation model of "information entropy" and "edge gradient".
[0025] The first information entropy value of a visible light image is defined as: The calculation formula is as follows: Where k represents the gray level of the image; L represents the total number of quantization levels of the image; Representing visible light images The probability distribution of the number of pixels with a gray value of k relative to the total number of pixels in the entire image; To prevent the appearance of a tiny positive real constant with a zero value inside the logarithmic function.
[0026] Similarly, calculate the second information entropy value of the infrared thermal imaging image. : ,in This represents the grayscale probability distribution of an infrared image. Information entropy. and From a statistical perspective, it objectively reflects the richness of detail and texture complexity contained in each modal image.
[0027] To further evaluate image sharpness and high-frequency contour intensity, the system uses an edge operator with smoothing effect to calculate the edge gradients of the two images separately. The first edge gradient value of the visible light image is defined as... : , where x and y represent the horizontal and vertical spatial coordinates of the pixel; Represents a two-dimensional discrete convolution operation; and These represent the edge detection convolution kernel matrices in the horizontal and vertical directions, respectively.
[0028] Similarly, calculate the second edge gradient value of the infrared thermal imaging image. : .
[0029] After obtaining the above four quality assessment parameters, the system calculates the visible light reference value used for fusion feedforward. Compared with infrared reference value : , .
[0030] Based on this, the network weight coefficients of the visible light feature branch in the feature-level dynamic fusion network are calculated. Network weight coefficients of infrared feature branches This embodiment uses a normalized exponential function with nonlinear excitation: , In the above formula, The preset nonlinear excitation gain coefficient, It is a natural exponential function. The innovative advantage of this weight allocation formula is that when a certain image is severely disturbed, the nonlinear amplification effect of the exponential term can quickly and smoothly transfer almost all the network weights to the unaffected high-quality image branch, ensuring the robustness of feature extraction.
[0031] In a deep learning cross-modal feature fusion network, let the high-dimensional feature tensor extracted by the visible light backbone network at a deep layer be... The high-dimensional feature tensor extracted by the infrared backbone network is Where C is the number of channels, and H and W are the feature map space dimensions, the final output of the system is the fused feature map. for: in, This indicates a splicing operation at the channel level. Will generate The characteristic tensor; This represents a mapping function that uses a 1×1 convolution kernel for dimensionality reduction and summation, ultimately outputting a fused feature map containing high-level semantic information from both modalities.
[0032] In this embodiment, the landing guidance marker is designed as a nested composite physical array: it includes a high-contrast visual geometric pattern in the visible light band and an active infrared radiation dot array arranged at key nodes of the geometric pattern.
[0033] The multimodal visual perception and fusion module obtains information from the detection head network. Features are extracted to obtain the target 2D pixel coordinate set. Let the side length matrix of the polygon formed by the extracted candidate pixel coordinate set be... K represents the number of lines connecting the extracted feature points, and the standard side length matrix corresponding to the physical size topology diagram of the landing guidance markers pre-stored in the onboard computer memory is: Define a spatial topological deviation metric function. : , where s is the overall scaling factor of the target image caused by perspective projection; This is the two-dimensional affine rotation matrix resulting from the tilt of the target plane relative to the camera plane; The Frobenius norm of a matrix is the square root of the sum of the squares of its elements.
[0034] The system executes the RANSAC iterative elimination logic, and when a certain feature point is calculated, it causes a local bias. Greater than the preset deviation threshold When a point is identified as a mismatched pixel coordinate caused by water reflection or noise, it is discarded. The remaining set of pixel coordinates with high spatial topological confidence is then output as the set of fused physical feature points. Ultimately, the system utilizes the PnP nonlinear optimization algorithm, combined with the camera intrinsic parameter matrix, based on... The initial relative pose matrix of the UAV with respect to the deck of the unmanned vessel at the current moment is calculated. It includes translation vectors and rotation matrices.
[0035] The shipborne dynamic status feedback module, located on the unmanned surface vessel (USV), mainly consists of a dual-antenna RTK receiver, a high-frequency shipborne IMU, and a UWB communication transmitter. In traditional landing systems, due to the randomness of the time consumed by UAV image processing and the data transmission time of the USV, the data from both ends are often misaligned on the timeline. This can lead to catastrophic calculation errors on dynamic sea surfaces with rough seas. To address this, this invention creatively introduces a hardware-level nanosecond timing synchronization mechanism based on the second pulse signal of the Global Navigation Satellite System (GNSS).
[0036] Both ends of the vessel are equipped with independent local time synchronization clocks, typically temperature-controlled crystal oscillators (OCXOs) or temperature-compensated crystal oscillators (TCXOs). Let the local crystal oscillator time at the moment the module sensor generates a sampling interruption be... The international standard absolute time of the latest PPS pulse output by the GNSS receiver is 1. Let the hardware clock counter's count value be the value received when the rising edge of the PPS pulse is reached. At the moment the sensor takes a sample, the hardware counter's count value is... Let the nominal rated frequency of the crystal oscillator be... Then the highly accurate absolute timestamp of that sampling moment. The calculation formula is: in, This is a constant for compensating for fixed transmission delay and interrupt response delay in hardware circuits calibrated by an oscilloscope.
[0037] Based on this, the multimodal visual perception and fusion module records the first absolute timestamp from the airborne device at the moment when the multimodal camera completes the global exposure center. And append it to the header of the image frame data stream.
[0038] The shipborne dynamic status feedback module records the second absolute timestamp at the moment of IMU and RTK data fusion sampling. This is then appended to the header of the six-DOF motion data packet. This allows data distributed across two heterogeneous physical platforms to be unified on the same absolute timeline with microsecond-level precision.
[0039] The unmanned ship terminal outputs the three-axis accelerometer from its internal high-frequency IMU. and triaxial angular velocity Three-dimensional absolute coordinates of low-frequency RTK output and heading angle Perform error state Kalman filtering fusion. Output a high-precision six-DOF motion data vector. : These represent the northward, eastward, and vertical positions of the ship's landing center in the world coordinate system, as well as the roll, pitch, and yaw angles, respectively. This vector, along with the timestamp... The latency compensation and pose feedforward prediction module of the UAV is transmitted at high frequency via an ultra-wideband low-latency wireless link.
[0040] The time delay compensation and pose feedforward prediction module is the core component for solving the problem of UAV crashes or misses caused by "perception-execution" lag in dynamic sea conditions. After receiving visual pose and hull motion data, the UAV must first analyze the time consumption in the closed-loop control link.
[0041] This invention reduces the total time delay of complex asynchronous closed-loop control systems. Quantified as a linear superposition of delays in four main processes: ,in, The time required for a multi-modal camera to complete single-frame image exposure, A / D conversion, and read the image into the onboard computer memory via DMA transfer through MIPI or USB bus; Perform multimodal fusion network inference, feature point extraction, mismatch removal, and PnP initial pose for the airborne computing platform. The time required for the solution; The time consumed by the unmanned vessel to send data frames to the drone via UWB over the air interface, and the queuing delay of the onboard computer sending control commands to the flight control unit via CAN / serial port; The hysteresis time is the combined time from the change in PWM command from the flight controller to the actual change in the speed of the UAV's brushless motor, resulting in a corresponding change in aerodynamic lift / thrust on the rotor and ultimately causing physical displacement of the airframe. This parameter can be obtained as an empirical constant through system frequency response or step response identification experiments.
[0042] Therefore, if the first absolute timestamp of the current image is So, what is the absolute moment when the control commands calculated for this frame of image actually take effect in the physical world to produce a flight effect? for: .
[0043] Since the ship's motion caused by waves is a typical non-stationary random process, at the moment the command takes effect... The actual spatial position and attitude of the unmanned vessel's deck compared to the moment of image acquisition. Dramatic changes have occurred. Flying directly to the old position would inevitably lead to trajectory oscillations or a collision. This invention creatively employs an autoregressive moving average model combined with a discrete Kalman filter for feedforward prediction of motion trends.
[0044] For the six degrees of freedom of the hull, construct Model fits wave-driven equations: Where k represents the current discrete time node; is the sequence value of rise and fall positions at time k; p is the order of the autoregressive model; q is the order of the moving average model; Let be the autoregressive correlation coefficient of order i; Let be the correlation coefficient of the j-th moving average; To meet the condition that the mean is zero and the variance is The system uses a Gaussian white noise sequence (representing random, uncertain excitation from sea surface winds and waves). It utilizes continuously received six-DOF motion data over a set time window as historical samples, and employs the least squares method to dynamically identify and update the coefficient set online. and This allows for the fitting of a motion change curve model of the unmanned vessel hull subjected to periodic fluctuations caused by wave action.
[0045] The identified curve model parameters are input as state transition parameters into the discrete Kalman filter. The system state vector of the Kalman filter is defined. It includes 6 degrees of freedom for position / attitude, velocity, and acceleration: Construct the discrete state-space transition equations: ,in, The system state transition matrix is filled with the first and second derivative relationships between state variables by conventional kinematic constants, while the higher-order dynamic response elements along the main diagonal and in its vicinity are coefficients fitted by the aforementioned ARMA model. , Dynamically populate and update; This is the process environmental noise vector; This is a noise-driven control matrix.
[0046] The observation equation is constructed as follows: ,in H represents the latest sequence of UWB measurements of the ship's hull, where H is the observation mapping matrix. It is a Gaussian observation noise with a measurement noise covariance matrix R.
[0047] The optimal state estimate at the current time is obtained in the Kalman standard filtering step. Then, the module uses the power product of the state transition matrix to perform N-step multi-step feedforward prediction (where the number of prediction steps is...). , (Sampling period), calculate the future time when the instruction takes effect. Expected state : .
[0048] Extract the expected spatial translation coordinate vector of the unmanned vessel deck in the world coordinate system at future moments from the predicted state vector. and expected attitude angle vector .
[0049] After obtaining the accurate attitude of the unmanned surface vessel at a future moment, it is necessary to reconstruct the initial relative pose at a "past moment" calculated by the vision module through spatial transformation. Let the exposure time calculated by PnP be... The initial relative translation vector of the UAV with respect to the ship is The rotation matrix is .
[0050] by Establish a local reference coordinate system with the center of the unmanned vessel deck at the current moment as the origin, and calculate the time from the current moment to the time when the future command takes effect. The absolute expected motion offset of the unmanned surface vessel in the world coordinate system. Expected 3D translation offset vector. : ,in This represents the actual position of the ship in the world coordinate system at the moment of exposure.
[0051] The expected three-dimensional rotation offset can be obtained by converting the attitude angles into direction cosine matrices and then calculating the relative relationships. Let the three-dimensional rotation matrix of the hull at the future moment be... The predicted Euler angles are generated through a standard ZYX order transformation: .
[0052] Similarly, the ship rotation matrix at the current moment is calculated based on the current Euler angles. .
[0053] Finally, the expected translation and rotation are used as spatial transformation parameters to perform inverse mapping compensation of the initial relative pose in a homogeneous coordinate system. This generates a high-precision target pose. Includes the final relative space coordinate vector Rotation matrix with final target The calculation formula is as follows: , The target pose represents: if the drone flies towards When it reaches that relative spatial point and passes through Time later, the unmanned vessel moved to the corresponding position under the action of the waves, achieving "pre-aiming and anticipation" in both space and time, and completely eliminating the tracking deviation caused by dynamic lag from the fundamental physical laws.
[0054] In extreme situations such as dense sea fog, splashing waves, or even temporary obstruction of landing markers by shipboard facilities, the UAV's vision module may temporarily lose target. The latency compensation and pose feedforward prediction module of this invention incorporates highly reliable fault-tolerant degradation evaluation logic.
[0055] The system's internal state machine is set under the following conditions: when the physical feature points extracted by multimodal fusion... The number of non-collinear points is less than the preset threshold for the minimum degrees of freedom matching, typically 4, and the number of consecutive video frames in which this "feature loss" state occurs reaches the preset tolerance frame count threshold. At that time, the system immediately determined that it was in a state of "visual perception failure".
[0056] Once in this state, the time delay compensation module locks the initial relative pose that was successfully solved and has the highest quality in the last frame before the visual perception failed, and records it as the reference anchor point pose. Subsequently, dead reckoning was performed solely based on the communication link between the airborne IMU and the ship. At any time t within the visual failure period, the three-axis specific force vectors output by the UAV's airborne high-frequency IMU in the body coordinate system were extracted and transformed to the world coordinate system. According to Newton's second law of motion, by performing a second integral on the net acceleration after removing the gravitational acceleration g, the absolute displacement increment vector of the UAV is obtained. : ,in Let be the attitude rotation matrix from the body coordinate system to the world coordinate system at time step integration. Simultaneously, using the ship's six degrees of freedom data continuously transmitted via UWB from the onboard dynamic state feedback module, the absolute displacement increment of the ship is recorded. .
[0057] Using the motion increments of the two absolute spaces mentioned above, the relative positional relationship vectors of the UAV and the unmanned surface vessel in the blind zone under the condition of no vision are calculated frame by frame. : This calculated pose replaces the visual pose output to the high-precision landing control module. Although this logic accumulates drift errors over time, it is sufficient to ensure that the UAV can maintain relative hovering and smooth movement with the ship in the short visual blind spot of a few seconds, avoiding loss of reference and crash, and greatly enhancing the system's resilience and fault tolerance.
[0058] The high-precision landing control module is located in the flight controller at the bottom layer of the UAV, and it is based on the high-precision target pose generated by feedforward prediction. The approach trajectory was planned, dividing the entire landing process into three physically distinct continuous phases: the approach phase, the hovering phase, and the touchdown phase.
[0059] In traditional drone landing logic, a constant descent rate is often used for direct landing. However, in rough sea conditions, if the ship happens to be in a rapid ascent phase being lifted by waves at the moment of drone contact, the combined relative collision velocities of the two will generate a huge physical impact force, which can easily break the drone's landing gear or damage the onboard electro-optical pod.
[0060] To address this, the present invention introduces intelligent kinematic discrimination logic during the hovering phase. The control module controls the UAV to hover at a preset safe height directly above the landing deck of the unmanned vessel, maintaining relative stillness while following the roll and pitch movements of the deck. Simultaneously, the control processor continuously analyzes the heave motion in the received six-degree-of-freedom motion data using high-frequency differentiation. The heave and displacement of ocean waves are approximately described as a periodic function. Where A is the amplitude, Let ω be the angular frequency. Solve for the first and second time derivatives of the displacement function to extract the vertical velocity component in real time. Vertical acceleration component : , .
[0061] According to the classical physical law of extreme values in simple harmonic motion, when the ship is pushed by the wave to its highest point, i.e., the crest of the wave, the displacement function reaches its maximum value. At this point: The velocity component must pass through zero: that is, theoretically... In the discrete sampling data of actual engineering projects, module detection... ,in The preset zero-speed tolerance threshold is extremely small.
[0062] The acceleration component must be downward and reach a minimum (maximum negative value): that is When downward is defined as positive, the opposite is true. In this system, when vertical upward is defined as positive, a negative acceleration value indicates a downward tendency.
[0063] The triggering logic of the high-precision landing control module of this system is as follows: when the condition is detected at the servo hovering height, the current condition is met. At that time, it was determined that the unmanned vessel was accurately located at the crest of the wave, and that the vessel would begin to sink along the wave trajectory within the next few seconds.
[0064] At that instant, the control module utilizes a rapid-triggered interrupt mechanism to immediately switch to the touchdown phase, controlling the drone's rotors to instantly reduce lift and rapidly descend vertically to touch down. This "wave-crest-following descent" technique, utilizing wave motion principles, ensures that the drone's descent direction is perfectly aligned with the ship's sinking direction, minimizing the relative velocity limit of the collision between the drone and the ship at the moment of impact, achieving a perfect "soft landing" on the dynamic base. This significantly enhances the drone's physical safety and impact resistance at the moment of touchdown from a mechanical perspective.
[0065] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0066] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A high-precision positioning and landing system for unmanned surface vessels based on vision guidance, characterized in that, Includes the following modules: The multimodal visual perception and fusion module, set on the UAV, includes a visible light camera and an infrared thermal imaging camera. It is used to acquire the target's visible light image and infrared thermal imaging image at the same time stamp, and uses a cross-modal feature fusion network to align and extract features from the two images, outputting the fused physical feature points of the UAV landing guidance mark. The shipborne dynamic status feedback module is set on the unmanned vessel and is used to collect the six degrees of freedom motion data of the unmanned vessel on the water surface caused by the wave effect in real time, and to send the six degrees of freedom motion data and the corresponding timestamp to the unmanned vessel through a wireless communication link. The time delay compensation and pose feedforward prediction module is set on the UAV and is communicatively connected to the multimodal visual perception and fusion module and the shipborne dynamic state feedback module, respectively. The system is used to calculate the current initial relative pose of the UAV based on the fused physical feature points, calculate the total system delay from image acquisition to the effective control command, and input the total system delay and the received six-degree-of-freedom motion data into a preset hull motion prediction model. The system calculates the expected motion offset of the landing guidance mark at the moment the command takes effect, and adds the expected motion offset to the initial relative pose to generate a high-precision target pose with dynamic compensation. A high-precision landing control module is installed on the UAV and connected to the time delay compensation and pose feedforward prediction module. It is used to receive the high-precision target pose, generate dynamic follow-approach commands, and control the UAV to land on the unmanned vessel.
2. The vision-guided high-precision positioning and landing system for unmanned surface vessels (USVs) according to claim 1, characterized in that, The multimodal visual perception and fusion module contains a feature-level dynamic fusion network; specifically, the module is used for image fusion as follows: The first information entropy value and the first edge gradient value of the visible light image are calculated in real time, and the second information entropy value and the second edge gradient value of the infrared thermal imaging image are calculated. The product of the first information entropy value and the first edge gradient value is used as the visible light reference value, and the product of the second information entropy value and the second edge gradient value is used as the infrared reference value. Calculate the proportions of the visible light reference value and the infrared reference value in the sum of the two values respectively, and use the corresponding proportions as the network weight coefficients of the visible light feature branch and the infrared feature branch. Then, according to the network weight coefficients, the feature maps extracted by the feature-level dynamic fusion network from the two images are spliced and weighted to output the fused feature map.
3. The vision-guided high-precision positioning and landing system for unmanned surface vessels (USVs) according to claim 2, characterized in that, The landing guidance markers include visual geometric patterns and an active infrared radiation array; the multimodal visual perception and fusion module extracts physical feature points specifically including: The corner pixel coordinates corresponding to the visual geometric pattern are extracted from the visible light channel data of the fused feature map, and the centroid pixel coordinates corresponding to the active infrared radiation array are extracted from the infrared channel data of the fused feature map. The deviation between all extracted pixel coordinates and the corresponding positions in the pre-stored landing guidance mark physical size topology map is calculated. Mismatched pixel coordinates with deviations greater than a preset deviation threshold are removed, and the remaining pixel coordinates are output as the fused physical feature points.
4. The vision-guided high-precision positioning and landing system for unmanned surface vessels based on unmanned aerial vehicles according to claim 1, characterized in that, Both the shipborne dynamic status feedback module and the multimodal visual perception and fusion module are equipped with independent timing clocks that receive the same second pulse signal from the same global navigation satellite system. The multimodal visual perception and fusion module records the first absolute timestamp of the local time clock and appends it to the image data stream at the instant the image is exposed and acquired. The shipborne dynamic state feedback module records the second absolute timestamp of the local time clock and appends it to the six-degree-of-freedom motion data at the instant the sensor samples. The time delay compensation and pose feedforward prediction module aligns the image data and the ship motion data by comparing and matching the first absolute timestamp and the second absolute timestamp.
5. The vision-guided high-precision positioning and landing system for unmanned surface vessels (USVs) according to claim 1, characterized in that, The total system delay calculated by the delay compensation and pose feedforward prediction module is the sum of all time intervals experienced by the system during a single closed-loop control process. The specific time intervals include: the acquisition duration for the multimodal camera to complete the exposure of a single frame image and read it into memory; the execution duration of the algorithm for image fusion and initial pose calculation; the transmission duration for the unmanned vessel to send data frames to the UAV via the wireless communication link; and the mechanical and physical response duration for the high-precision landing control module to issue approach control commands to the UAV rotor motor to generate corresponding lift changes.
6. The vision-guided high-precision positioning and landing system for unmanned surface vessels based on unmanned aerial vehicles according to claim 1, characterized in that, The preset ship motion prediction model includes an autoregressive moving average model and a discrete Kalman filter. The time delay compensation and pose feedforward prediction module inputs the six-degree-of-freedom motion data continuously received within a set time window as historical samples into the autoregressive moving average model to fit the motion change curves of the unmanned vessel caused by the water surface waves, resulting in periodic roll, pitch, and heave. The motion change curves are then input into the discrete Kalman filter as state transition parameters to recursively calculate the expected spatial coordinates and expected attitude angles of the unmanned vessel deck in the world coordinate system at future time nodes after adding the total system time delay to the current time node.
7. The vision-guided high-precision positioning and landing system for unmanned surface vessels based on unmanned aerial vehicles according to claim 1, characterized in that, The specific steps for the delay compensation and pose feedforward prediction module to generate the high-precision target pose are as follows: A reference coordinate system is established with the center of the unmanned vessel deck at the current moment as the origin, and the initial relative pose is mapped to the reference coordinate system. Based on the calculated expected spatial coordinates and expected attitude angles at the future time node, the three-dimensional translation vector and three-dimensional rotation matrix generated by the unmanned vessel deck from the current time to the future time node are calculated as the expected motion offset. The three-dimensional translation vector and three-dimensional rotation matrix are applied as spatial transformation parameters to the initial relative pose to generate the final relative spatial coordinates and yaw angle that compensate for the physical displacement error of the unmanned vessel, which are then used as the high-precision target pose.
8. The vision-guided high-precision positioning and landing system for unmanned surface vessels based on unmanned aerial vehicles according to claim 1, characterized in that, The high-precision landing control module divides the landing process into the approach phase, the follow-up hovering phase, and the touchdown phase. During the hovering phase, the high-precision landing control module continuously analyzes the vertical velocity component and vertical acceleration component in the received six-degree-of-freedom motion data. When it detects that the absolute value of the vertical velocity component is less than a preset zero velocity threshold and the value of the vertical acceleration component is negative, it determines that the unmanned vessel is at the crest of the wave motion and immediately triggers the touch-down phase at that moment, controlling the UAV to descend vertically.
9. The vision-guided high-precision positioning and landing system for unmanned surface vessels based on unmanned aerial vehicles according to claim 1, characterized in that, The delay compensation and pose feedforward prediction module has a fault-tolerant degradation evaluation logic inside; When the number of fused physical feature points extracted by the multimodal visual perception and fusion module is less than a preset matching number threshold, and the number of consecutive frames in which this state occurs reaches a preset frame number threshold, visual perception is determined to fail. In the case of visual perception failure, the time delay compensation and pose feedforward prediction module locks the initial relative pose successfully calculated in the last frame before the visual perception failure. Based on this, and combined with the self-motion data output by the UAV's onboard inertial measurement unit and the six-degree-of-freedom motion data continuously transmitted back by the shipboard dynamic state feedback module, the relative position relationship between the UAV and the unmanned ship is calculated frame by frame using the dead reckoning algorithm, and then output to the high-precision landing control module.
10. The vision-guided high-precision positioning and landing system for unmanned surface vessels based on unmanned aerial vehicles according to claim 1, characterized in that, The shipborne dynamic status feedback module includes a shipborne inertial measurement unit, a dual-antenna real-time dynamic differential positioning receiver, and an ultra-wideband communication transmitter. The inertial measurement unit outputs inertial data including three-axis acceleration and three-axis angular velocity, and the differential positioning receiver outputs three-dimensional absolute coordinates and heading angle data. The shipborne dynamic state feedback module performs multi-sensor Kalman filtering fusion on the inertial data and absolute coordinates to generate the six-degree-of-freedom motion data, which is then transmitted to the UAV via the ultra-wideband communication transmitter.