Method for reconstructing ocean wave surface in full scale based on two unmanned aerial vehicles for mobile platform observation

Abstract

This application discloses a method for real-scale reconstruction of ocean wavefronts using dual unmanned aerial vehicles (UAVs) for mobile platform observation, relating to the field of ocean wavefront reconstruction technology. The method includes: acquiring real-time images from two UAVs to form a synchronous image pair and obtaining multi-source motion state data; extracting features from each real-time image and establishing feature matching relationships; determining the relative rotation and real-scale relative translation between the two UAVs based on the feature matching relationships and the multi-source motion state data; estimating the initial values ​​of the zero-plane parameters by performing zero-plane parameter estimation on the reconstructed three-dimensional point set of the wavefront from the synchronous image pair, establishing a joint optimization model for the zero-plane parameters, and solving for the zero-plane parameters; and mapping the three-dimensional point set of the wavefront to a real-scale reference frame using the zero-plane parameters to form a real-scale three-dimensional ocean wavefront. This method can achieve real-scale restoration of the ocean wavefront under mobile platform conditions, improving the stability and engineering usability of wavefront reconstruction.

Description

Technical Field

[0001] This application relates to the field of ocean wavefront reconstruction technology, and in particular to a dual-UAV ocean wavefront real-scale reconstruction method for mobile platform observation. Background Technology

[0002] The spatial morphology and evolution of ocean waves directly affect ship navigation safety, seakeeping design of offshore structures, and the assessment of operational windows at sea. Therefore, achieving full-scale reconstruction of ocean wave surfaces is of great significance for practical engineering applications.

[0003] While traditional contact buoys, wave height meters, and fixed wave measuring radars can provide wave observation results for specific points or local areas, they suffer from drawbacks such as high deployment costs, limited coverage, and difficulty in adapting to mobile observations. With the development of UAV platforms, industrial cameras, GNSS, UWB, IMU, and real-time positioning technologies, using UAVs for ocean wave observation has become a new solution that combines mobility and low deployment costs.

[0004] Existing technologies have proposed solutions for 3D wave reconstruction based on single UAV viewpoint images for fixed platform scenarios. However, in observation scenarios involving mobile platforms such as ships, achieving stable reconstruction of the ocean wave surface at the true physical scale still faces several coupled challenges that cannot be directly solved using traditional methods from fixed platforms or weak motion scenarios. Summary of the Invention

[0005] To address the aforementioned problems and technical requirements, this application proposes a method for real-scale reconstruction of ocean wavefronts using dual unmanned aerial vehicles (UAVs) for mobile platform observation. The technical solution of this application is as follows: A method for real-scale reconstruction of ocean wavefronts using dual unmanned aerial vehicles (UAVs) for mobile platform observation includes the following steps: Two UAVs are controlled to conduct collaborative observation of the target sea area according to a preset relative spatial relationship. Real-time images collected by the cameras on the two UAVs are simultaneously acquired to form a synchronous image pair. Multi-source motion state data collected by the sensors on the two UAVs are also simultaneously acquired. The multi-source motion state data is consistent with the timing of the synchronous image pair, including ranging data, positioning data, and inertial measurement data used to characterize the relative position and motion state of the two UAVs. Based on positioning data and inertial measurement data, features are extracted from each real-time image in the synchronized image pair, and a feature matching relationship is established between the two real-time images. According to the feature matching relationship, the relative rotation and normalized translation direction between the two UAVs are determined. A real-scale distance constraint model is constructed by fusing multi-source motion state data, and the real-scale distance between the two UAVs is obtained by solving the model. The real-scale distance is combined with the normalized translation direction to obtain the real-scale relative translation. Based on relative rotation and real-scale relative translation, three-dimensional wavefront reconstruction is performed on the synchronous image pair to obtain a three-dimensional point set of the wavefront. Based on linear wave theory, zero-plane parameter estimation is performed on the three-dimensional point set of the wavefront to obtain the initial value of the zero-plane parameter. Inertial measurement data is introduced to establish a joint optimization model of the zero-plane parameter. Solving the joint optimization model of the zero-plane parameter yields the zero-plane parameter at the current moment. The zero plane is a reference surface that characterizes the mean sea level. The three-dimensional point set of the wavefront is mapped to a real-scale reference system using the zero-plane parameter to form a real-scale three-dimensional ocean wavefront. The real-scale reference system is a reference coordinate system with the zero-plane as the elevation reference and coordinate units consistent with the real physical space scale.

[0006] A further technical solution is to use a neural network model to extract features from each real-time image in the synchronized image pair and establish a feature matching relationship between the two real-time images. The neural network model includes a convolutional backbone network, a keypoint detection head, and a descriptor generation head, which are cascaded in sequence. The convolutional backbone network encodes each real-time image in the synchronous image pair to obtain a feature tensor corresponding to each real-time image. The keypoint detection head performs keypoint detection on the feature tensor to obtain candidate keypoints. The descriptor generation head extracts local features from the candidate keypoints to generate feature descriptors. Establishing feature matching relationships between two real-time images includes: The inertial measurement data and positioning data are fused and calculated to obtain the initial value of the relative rotation between the two UAVs, and the local epipolar neighborhood of the candidate keypoints corresponding to each real-time image in the synchronized image pair is determined. The candidate keypoints corresponding to one real-time image in the synchronized image pair are projected into the local epipolar neighborhood of the candidate keypoints corresponding to the other real-time image, and feature descriptor correlation matching is performed to establish the feature matching relationship between the two real-time images. The local epipolar neighborhood is a local search region in the real-time image.

[0007] The further technical solution involves determining the relative rotation and normalized translation directions between the two UAVs based on feature matching relationships, including: By fusing inertial measurement data and positioning data, the initial values ​​of the relative rotation and normalized translation directions between the two UAVs are obtained, so as to construct an objective function for optimizing the relative geometric relationship, which includes reprojection error terms and attitude prior constraints. The relative geometric relationship optimization objective function is solved by using the bundle adjustment method within a preset neighborhood of the initial value to obtain the relative rotation and normalized translation direction between the two UAVs.

[0008] A further technical solution is that the relative geometric relationship optimization objective function for:

[0009] in, For camera projection function, For robust kernel functions, For the attitude prior weights, It is the set of keypoint matching points between two real-time images and , It is the i-th candidate keypoint corresponding to a real-time image in a synchronized image pair. It is the i-th candidate keypoint corresponding to another real-time image in the synchronized image pair. It is a camera rotation parameter. It's the rotation parameter of another camera. It is a camera translation parameter. It is the translation parameter of another camera. It is the 3D wavefront point corresponding to the i-th candidate keypoint. It is the initial value of the relative rotation. It is the relative rotation at the current time t. This represents a logarithmic mapping over the rotation group, used to measure the deviation between the pose prior and the visually optimized pose. This indicates the matrix transpose.

[0010] A further technical solution is that the neural network model optimizes network parameters using a joint loss function, wherein the joint loss function... , in, It is the key point detection loss weight coefficient. It is a key point detection loss and , It is a set of pixel or feature grid locations in a real-time image. The key point detection head is in position Output keypoint probabilities, It is a key point monitoring label. and These are the weights of positive samples at key points and negative samples in the background, respectively. These are the descriptor contrast loss weight coefficients. It is a descriptor contrast loss and , and These are the normalized descriptors for candidate keypoints corresponding to two real-time images. It is a set of positive matching descriptor pairs. It is a set of negative matching descriptor pairs. It is the negative sample interval threshold; It is the geometric consistency loss weighting coefficient. Geometric consistency loss and, and These are the homogeneous pixel coordinates of the candidate keypoints in the two real-time images. The initial values ​​of the fundamental matrix are determined by the initial values ​​of the relative rotation and the normalized translation direction. Represents the robust penalty function. This indicates the calculation of the Euclidean norm. This represents the first element in the vector. This represents the second element in the vector.

[0011] A further technical solution involves using ranging data, positioning data, and inertial measurement data as observation factors to construct the objective function of the real-scale distance constraint model. The objective function of the real-scale distance constraint model is solved using the factor graph optimization method to obtain the real-scale distance between the two UAVs at the current time t. ; to measure the actual distance As a scaling factor, it acts on the normalized translation direction. The real-scale relative translation is obtained. .

[0012] A further technical solution is that the objective function of the real-scale distance constraint model is... for:

[0013] in, For residuals and , This refers to the measured distance in the ranging data. For real-scale distances, To locate the residual, For relative positioning residuals, For motion residuals, subscript , , , These are the covariance matrices of the ranging residual, positioning residual, relative positioning residual, and motion residual, respectively. Denotes the inverse of the covariance matrix. This represents the weighted 2-norm for calculating the inverse of the covariance matrix.

[0014] A further technical solution involves obtaining the zero-plane parameters at the current time t, including: The zero plane is parameterized as a combination of normal vector and bias: , The zero plane normal vector, Zero plane offset, It is a three-dimensional point set of the wavefront and = , It is the first point in the three-dimensional point set of the wavefront j One point, It is the total number of points in the three-dimensional point set of the wavefront; the zero plane offset is used to characterize the positional relationship of the zero plane relative to the camera coordinate system; Based on the physical property in linear wave theory that the statistical mean of wavefront undulation is zero, the initial value of the zero plane offset is determined. , It is the initial value of the zero plane normal vector; the objective function of the joint optimization model of the zero plane parameters is constructed by using inertial measurement data as motion constraints, and the initial value of the zero plane bias is used as the basis. The objective function of the joint optimization model of the zero plane parameters is solved using the factor graph optimization method, and the zero plane normal vector at the current time t is obtained. and zero plane bias .

[0015] A further technical solution is that the objective function of the zero-plane parameter joint optimization model is... for:

[0016] in, The normal prior obtained from inertial measurement data. As a weight for time continuity, This represents the zero-plane offset at the previous time t-1. This represents the variance of the zero-plane bias calculation. This represents the weighted 2-norm of the covariance inverse matrix of the normal vector.

[0017] A further technical solution involves performing three-dimensional wavefront reconstruction on the synchronized image pair to obtain a three-dimensional wavefront point set, including: Stereo correction is performed on the synchronized image pairs based on relative rotation and real-scale relative translation; A dense disparity map is generated by using feature matching relationships as disparity priors and combining them with local cost volumes. Based on the principle of triangulation and combined with real-scale distance, the three-dimensional coordinates of each pixel in the dense disparity map are calculated to obtain the three-dimensional point set of the wavefront.

[0018] The beneficial technical effects of this application are: This application discloses a method for real-scale reconstruction of ocean wave surfaces using dual unmanned aerial vehicles (UAVs) for mobile platform observation. It constructs a complete technology chain from synchronous acquisition of multi-source data, estimation of dynamic relative geometric relationship between the two UAVs, estimation of dynamic zero plane, to three-dimensional reconstruction of ocean wave surfaces at real scale. This method can not only accurately obtain real-scale spatial morphology information of ocean wave surfaces, but also provide wave parameter inputs with real physical meaning for subsequent ship motion forecasting and marine engineering analysis, thus possessing significant engineering application value.

[0019] An improved neural network model is used to extract and match features from real-time images. By introducing inertial measurement data as a priori constraint on the search range of candidate matching points, and combining geometric consistency loss to optimize network training, the mismatch problem in weak texture, repetitive texture and local reflection scenes on the sea surface is effectively suppressed. This provides a more reliable sparse feature correspondence for dynamic relative geometric relationship estimation, significantly improves the accuracy and stability of binocular geometric estimation, and lays a key foundation for subsequent real-scale wavefront reconstruction.

[0020] By combining simultaneous observations from two UAVs with dynamic relative geometric relationship estimation, the problem of unstable acquisition of binocular geometry caused by the continuous relative motion of two mobile platforms is solved. The relative rotation and normalized translation directions are obtained by using bundle adjustment, and the scale is recovered by using multi-source sensor constraints. Time-varying real-scale binocular geometric relationships can be formed for each pair of synchronized images, which significantly improves the stability and consistency of 3D reconstruction in mobile observation scenarios.

[0021] By fusing and solving multi-source motion state data acquired from multiple sensors, the true physical scale of visual reconstruction results relying solely on real-time images is restored, avoiding the limitation of existing single-view schemes that must rely on external reference objects to convert real dimensions. Moreover, by jointly optimizing the initial estimation of the zero plane and the constraints of inertial measurement data to establish a unified reference datum, the problem of the difficulty in stably acquiring the dynamic zero plane under moving platform conditions is solved. This ensures that the obtained three-dimensional ocean wave surface not only has a relative undulation shape but also has absolute scale information that can be used in engineering, and can be directly used for wave parameter calculation and ship motion prediction model input. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of a dual-UAV collaborative observation system.

[0023] Figure 2 This is a flowchart of a method for real-scale reconstruction of ocean wave surfaces using dual unmanned aerial vehicles (UAVs). Detailed Implementation

[0024] The specific embodiments of this application will be further described below with reference to the accompanying drawings.

[0025] This application constructs a dual-UAV cooperative observation system for mobile platforms. Please refer to [reference needed]. Figure 1 The diagram shows the system's structure. It includes two UAVs, two image acquisition units, two attitude and positioning measurement units, and a data processing unit. Each UAV is equipped with a camera and various sensors, including GNSS (Global Positioning System), UWB (Ultra-Wideband) ranging equipment, mobile RTK (Real-Time Differential Positioning), and IMU (Inertial Measurement Unit).

[0026] The image acquisition unit is used to acquire real-time images of the ocean wave surface. It can employ an industrial camera and a fixed-focus lens, or other imaging equipment with synchronous exposure capabilities. Specific platform and camera models are not limited in this application. GNSS provides absolute position and timing information, UWB measures the distance between the two UAVs, IMU provides attitude and motion information, and mobile RTK improves relative positioning accuracy. The figure shows a schematic diagram of a dual-UAV collaborative observation system with a ship as the mobile platform. The image acquisition unit and each sensor have synchronous sampling capabilities to achieve simultaneous acquisition of multi-source data. The synchronously acquired multi-source data is input into the data processing unit for analysis and calculation to reconstruct a real-scale three-dimensional ocean wave surface.

[0027] This application discloses a method for real-scale reconstruction of ocean wavefronts using dual unmanned aerial vehicles (UAVs) for mobile platform observation. This method is deployed in the data processing unit of a dual UAV collaborative observation system. Please refer to [reference needed]. Figure 2 The flowchart shown illustrates the specific steps of this method as follows: Step 1: Control two UAVs to conduct collaborative observation of the target sea area according to a preset relative spatial relationship. Simultaneously acquire real-time images collected by the cameras on the two UAVs to form a synchronous image pair, and simultaneously acquire multi-source motion state data collected by the sensors on the two UAVs. The multi-source motion state data is consistent with the timing of the synchronous image pair, including ranging data, positioning data, and inertial measurement data used to characterize the relative position and motion state of the two UAVs.

[0028] During the observation process, two UAVs are controlled to fly collaboratively according to a preset relative spatial relationship, ensuring that the fields of view of both cameras simultaneously cover the target sea area. The two UAVs can maintain preset tracks or dynamically adjust their relative positions based on the observation target and safety conditions, as long as they can form an effective binocular observation baseline at the same observation time. The cameras on both UAVs undergo pre-calibration of their intrinsic parameters; the specific calibration method can be customized, such as placing checkerboard calibration boards on the ship's deck or other areas during takeoff or preparation to check or compensate for system errors. The preset relative spatial relationship and preset track of the two UAVs are pre-set according to the actual observation task. Figure 1The two UAVs shown are symmetrically arranged on the left and right sides of the mobile platform. The common observation area formed by the two UAVs along their respective observation directions covers the target sea area in the direction of the mobile platform's movement. This application is particularly suitable for forward observation scenarios of the mobile platform, that is, two UAVs simultaneously observe the incoming wave area in the direction of the mobile platform's movement to obtain wave information of the target sea area in advance. However, it can also be applied to other observation angles. This application method is a preferred embodiment and is used to illustrate the engineering value of this application, and does not constitute an essential limitation of the method of this application. In one example, the baseline distance between the two UAVs is controlled within the range of 5-20 m to balance common field of view, stereo geometric stability, and flight safety. The image acquisition frequency is 1 Hz, and the continuous observation time is about 30 minutes to obtain sufficient wave timing information.

[0029] It should be noted that, to ensure the consistency of the transient wave morphology, a GNSS pulse-per-second (PPS) trigger is used to trigger simultaneous exposure of the two cameras, thereby acquiring synchronized image pairs of the wave surface at the same moment. Simultaneously with image acquisition, multi-source motion state data of the two UAVs at corresponding moments are recorded, including ranging data acquired by UWB, positioning data acquired by GNSS, high-precision positioning data acquired by mobile RTK, and inertial measurement data acquired by IMU. This provides time-consistent observational input for subsequent relative geometric relationship estimation and zero-plane solution. The synchronous acquisition frequency of the above multi-source data is an arbitrary fixed frequency from 1 Hz to 8 Hz, which can be set according to the actual application requirements.

[0030] Step 2: Based on positioning data and inertial measurement data, feature extraction is performed on each real-time image in the synchronized image pair, and a feature matching relationship is established between the two real-time images. According to the feature matching relationship, the relative rotation and normalized translation direction between the two UAVs are determined. A real-scale distance constraint model is constructed by fusing multi-source motion state data, and the real-scale distance between the two UAVs is obtained by solving the model. The real-scale distance is combined with the normalized translation direction to obtain the real-scale relative translation.

[0031] In dual-mobile drone scenarios, the dynamic relative geometry between cameras is difficult to obtain stably. During flight, the two drones continuously undergo relative translation, pitch, roll, and yaw changes, resulting in a non-fixed relative geometric relationship between the two cameras. If only the calibration results of a fixed binocular system are used, or if estimation is based solely on images from a single camera, it is difficult to obtain reliable binocular geometric relationships for each synchronized image pair, thus affecting the accuracy of subsequent stereo reconstruction.

[0032] To address this issue, for each pair of synchronized images, identifiable texture features or other stable features on the surface of the waves are first extracted to establish a feature matching relationship between the two real-time images.

[0033] In one embodiment, a neural network model is used to extract features from each real-time image in a synchronized image pair and establish a feature matching relationship between the two real-time images.

[0034] The neural network model comprises a cascaded convolutional backbone network, a keypoint detection head, and a descriptor generation head. The convolutional backbone network encodes each real-time image in the synchronized image pair, obtaining a feature tensor corresponding to each real-time image. The convolutional backbone networks encoding the two real-time images share weights. The keypoint detection head detects keypoints on the feature tensors, obtaining candidate keypoints. The descriptor generation head extracts local features from the candidate keypoints, generating feature descriptors. The keypoints in the real-time images correspond to recognizable local regions such as wave textures and wave crests. These pixels maintain recognizability under different viewing angles and lighting conditions. The feature descriptor is a digital encoding representation of the local image features surrounding the keypoint, transforming information such as texture, grayscale variations, and edge directions near the keypoint into standardized numerical features. The feature descriptors generated for keypoints at the same physical location in different real-time images show high similarity, thus establishing a feature matching relationship between two real-time images.

[0035] Specifically, establishing feature matching relationships between two real-time images includes: The inertial measurement data and positioning data are fused and calculated to obtain the initial value of the relative rotation between the two UAVs, and the local epipolar neighborhood of the candidate keypoints corresponding to each real-time image in the synchronized image pair is determined. The candidate keypoints corresponding to one real-time image in the synchronized image pair are projected into the local epipolar neighborhood of the candidate keypoints corresponding to the other real-time image, and feature descriptor correlation matching is performed to establish the feature matching relationship between the two real-time images. The local epipolar neighborhood is a local search region in the real-time image.

[0036] Specifically, the fusion solution method for inertial measurement data and positioning data can adopt existing technologies, such as extended Kalman filtering; the local epipolar neighborhood constructs an essential matrix based on the initial values ​​of relative rotation and normalized translation direction, maps the candidate key points corresponding to one real-time image in the synchronized image pair to the epipolar line of another real-time image, and sets the epipolar bandwidth and parallax search range based on sensor uncertainties such as camera calibration error, time synchronization error, IMU attitude error and positioning error.

[0037] The specific method for feature descriptor relevance matching is as follows: based on the key points and feature descriptors output by the neural network model, cosine similarity is calculated, and the relevance matching relationship is obtained by combining bidirectional consistency check and threshold screening.

[0038] Compared to traditional SuperPoint-like neural network methods, this application introduces attitude data acquired by sensors as a priori to constrain the search area of ​​candidate matching points, which can suppress mismatches under conditions such as weak texture, repetitive texture, and local reflection on the sea surface.

[0039] Furthermore, when training the neural network model, the network parameters are optimized using a joint loss function. , in, It is the key point detection loss weight coefficient. It is a key point detection loss and , It is a set of pixel or feature grid locations in a real-time image. The key point detection head is in position Output keypoint probabilities, It is a key point monitoring label. and These are the weights of positive samples at key points and negative samples in the background, respectively. These are the descriptor contrast loss weight coefficients. It is a descriptor contrast loss and ,in, and These are the normalized descriptors for candidate keypoints corresponding to two real-time images. It is a set of positive matching descriptor pairs. It is a set of negative matching descriptor pairs. It is the negative sample interval threshold; It is the geometric consistency loss weighting coefficient. It is a geometric consistency loss and ,in, and These are the homogeneous pixel coordinates of the candidate keypoints in the two real-time images. The initial values ​​of the fundamental matrix are determined by the initial values ​​of the relative rotation and the normalized translation direction. Represents the robust penalty function. This indicates the calculation of the Euclidean norm. This represents the first element in the vector. This represents the second element in the vector. Geometric consistency loss is used to constrain keypoint matching pairs between two real-time images output by the neural network model to satisfy local epipolar neighborhood constraints.

[0040] By using a neural network model improved through attitude priors for feature extraction and matching, a keypoint matching set more suitable for ocean wave scenarios can be obtained. .

[0041] Based on this, the reprojection error between the two real-time images is further optimized to obtain the relative rotation and normalized translation directions between the two UAVs. Specifically, based on feature matching relationships, the relative rotation and normalized translation directions between the two UAVs are determined as follows: By fusing inertial measurement data and positioning data, the initial values ​​of the relative rotation and normalized translation directions between the two UAVs are obtained, so as to construct an objective function for optimizing the relative geometric relationship, which includes reprojection error terms and attitude prior constraints. The relative geometric relationship optimization objective function is solved using the bundle adjustment method within a preset neighborhood of the initial values ​​of the relative rotation and normalized translation direction, yielding the relative rotation and normalized translation direction between the two UAVs. The normalized translation direction represents the binocular geometric description where the attitude relationship is determined but the scale has not yet been recovered. The preset neighborhood is customized based on actual application requirements and experience, typically determined by IMU, GNSS, RTK, UWB data measurement errors, camera calibration errors, and historical observation residuals.

[0042] Unlike the traditional method of first estimating the essential matrix using global feature matching and then decomposing the pose, this embodiment adopts a "pose prior initialization + local neighborhood optimization" Bundle Adjustment solution.

[0043] Specifically, the relative pose at the current time t is represented as... ,in, It is the relative rotation at the current time t. It is the normalized translation direction at the current time t; initial value Results derived from the fusion of inertial measurement data and positioning data.

[0044] In one embodiment, the relative geometric relationship optimization objective function for:

[0045] in, For camera projection function, For robust kernel functions, For the attitude prior weights, It is the set of keypoint matching points between two real-time images and , It is the i-th candidate keypoint corresponding to a real-time image in a synchronized image pair. It is the i-th candidate keypoint corresponding to another real-time image in the synchronized image pair. It is a camera rotation parameter. It's the rotation parameter of another camera. It is a camera translation parameter. It is the translation parameter of another camera. It is the 3D wavefront point corresponding to the i-th candidate keypoint. It is the initial value of the relative rotation. It is the relative rotation at the current time t. This represents a logarithmic mapping over the rotation group, used to measure the deviation between the pose prior and the visually optimized pose. This indicates the matrix transpose.

[0046] Furthermore, by adjusting the normalized translation direction... Only within its preset neighborhood range Internal search: This avoids falling into erroneous extreme values ​​caused by global mismatches in low-texture sea surface scenes, and obtains more stable normalized relative geometric relationships.

[0047] Since visual geometry relying solely on the analysis of real-time images typically yields only a normalized translation direction with uncertain scale, this application utilizes multi-source motion state data to construct a real-scale distance constraint model to further recover the scale of the normalized translation direction.

[0048] In one embodiment, the real-scale distance constraint model is a graphical model. Ranging data, positioning data, and inertial measurement data from multi-source motion state data are used as observation factors to construct the objective function of the real-scale distance constraint model. The objective function of the real-scale distance constraint model is solved using a factor graph optimization method to obtain the real-scale distance between the two UAVs at the current time t. ; to measure the actual distance As a scaling factor, it acts on the normalized translation direction. The real-scale relative translation is obtained. .

[0049] Ranging data provides direct ranging information between the two UAVs, positioning data provides global position and relative positioning constraints, and inertial measurement data provides high-frequency attitude and motion information. The system state of the two UAVs at the current time t is defined as... , , These represent the positions of the two drones, , These represent the speeds of the two drones, , Let represent the attitude quaternions of the two UAVs, respectively. By constructing the above observations into a unified graphical model and performing factor graph optimization, the real-scale distance at each time step can be estimated. Combining the visual geometry results with the real-scale distance yields the real-scale relative translation at the corresponding time step.

[0050] Specifically, the objective function of the constructed real-scale distance-constrained model for:

[0051] in, For the distance measurement residual, To locate the residual, For relative positioning residuals, For motion residuals, subscript , , , These are the covariance matrices of the ranging residual, positioning residual, relative positioning residual, and motion residual, respectively. Denotes the inverse of the covariance matrix. This represents the weighted 2-norm for calculating the inverse of the covariance matrix.

[0052] , This refers to the measured distance in the ranging data. For real-scale distance and , , Indicate the positions of the two drones respectively; It is a 6-dimensional column vector. and It is the absolute position observation value output by GNSS; ,in It is a high-precision position observation value output by mobile RTK; , in, It is the position component of the motion residual. It is the velocity component of the motion residual. It is the attitude component of the motion residual. It is the change in relative position. It is the change in velocity. It is the change in attitude. The gravity vector It is a change over time. Let be the rotation matrix at time t-1. Let be the vector formed by the velocities of the two UAVs at time t-1. Let be the vector formed by the velocities of the two UAVs at time t. Let be the vector formed by the attitudes of the two UAVs at time t-1. Let be the vector formed by the attitudes of the two UAVs at time t. To represent quaternion multiplication, This indicates taking the imaginary part of the quaternion.

[0053] Multi-time joint optimization can utilize the motion continuity constraints of the IMU and the state correlation between adjacent times to reduce the impact of single-time UWB, GNSS or RTK noise on the results, making the obtained real-scale distances smoother and more consistent in time.

[0054] The objective function of the real-scale distance constraint model is solved using the factor graph optimization method to obtain the real-scale distance between the two UAVs at the current time t. .

[0055] The real-scale binocular geometry relationship includes real-scale relative translation and relative rotation, which is the binocular extrinsic parameter of the two cameras. This relationship reflects the dynamic relative attitude changes under dual-moving platform conditions and restores the true physical scale, providing a reliable foundation for subsequent three-dimensional wavefront reconstruction of the ocean.

[0056] Compared with existing technologies, this application achieves dynamic real-scale binocular geometry construction by jointly solving visual geometry and multi-source scale constraints, which is more in line with real marine observation conditions.

[0057] Step 3: Based on relative rotation and real-scale relative translation, perform three-dimensional wavefront reconstruction on the synchronized image pair to obtain a three-dimensional point set of the wavefront; based on linear wave theory, perform zero-plane parameter estimation on the three-dimensional point set of the wavefront to obtain the initial value of the zero-plane parameter, and introduce inertial measurement data to establish a joint optimization model of the zero-plane parameter. Solve the joint optimization model of the zero-plane parameter to obtain the zero-plane parameter at the current time. The zero plane is a reference surface that characterizes the mean sea level.

[0058] In fixed-platform scenarios, the mean sea level can be estimated by averaging multiple real-time images or using a static reference frame. However, for mobile platform observation scenarios, the UAV itself undergoes heave, pitch, and roll motions, and the ocean waves themselves are also in a state of spatiotemporal variation, making the zero plane unsuitable as a simple fixed reference surface. The zero plane should not be directly obtained using the multi-frame averaging method under fixed-platform conditions because the UAV's own attitude changes and the random fluctuations of the ocean waves will simultaneously affect the image observation results.

[0059] To address this issue, this application estimates a dynamically changing zero plane. In one embodiment, obtaining the zero plane parameters at the current time t includes: The zero plane is parameterized as a combination of normal vector and bias: , The zero plane normal vector, Zero plane offset, It is a three-dimensional point set of the wavefront and = , It is the first point in the three-dimensional point set of the wavefront j One point, It is the total number of points in the three-dimensional point set of the wavefront; the zero plane offset is used to characterize the positional relationship of the zero plane relative to the camera coordinate system; based on this parameterized representation, the zero plane parameters include the zero plane normal vector and the zero plane offset.

[0060] Based on the physical property in linear wave theory that the statistical mean of wavefront undulation is zero, the initial value of the zero plane offset is determined. ,in, The initial value of the zero-plane normal vector comes from prior inertial measurement data. The objective function of the joint optimization model for the zero-plane parameters is constructed using the inertial measurement data as motion constraints, and is based on the initial value of the zero-plane bias. The objective function of the joint optimization model of the zero plane parameters is solved using the factor graph optimization method, and the zero plane normal vector at the current time t is obtained. and zero plane bias .

[0061] The initial value of the zero-plane offset is inferred from the current wavefront texture, camera viewpoint relationship, and wave statistical laws, providing a reasonable starting point for subsequent optimization. Then, information such as platform attitude changes, angular velocity, and acceleration obtained by the inertial measurement unit is introduced as motion constraints into the zero-plane solution process to establish a joint optimization model for the zero-plane parameters.

[0062] In this model, the zero plane parameters at each time step are constrained by both single-frame observations and the continuity of motion of the mobile platform. Therefore, they can be solved together through factor graph optimization to obtain the zero plane parameters corresponding to the synchronous image pairs at each time step, thereby achieving a stable estimation of the ocean zero plane under the mobile platform condition.

[0063] In one embodiment, the objective function of the zero-plane parameter joint optimization model is... for:

[0064] in, The normal prior obtained from inertial measurement data. As a weight for time continuity, This represents the zero-plane offset at the previous time t-1. This represents the variance of the zero-plane bias calculation. This represents the weighted 2-norm of the covariance inverse matrix of the normal vector.

[0065] This method avoids the simplistic assumption of treating the zero plane as a fixed constant, so that the subsequent three-dimensional wavefront reconstruction not only has relative shape information, but also has a unified absolute reference plane, which can support the calculation of parameters such as true wave height and meaningful wave height.

[0066] Step 4: Map the three-dimensional point set of the wavefront to a real-scale reference system using the zero-plane parameter to form a real-scale three-dimensional ocean wavefront. The real-scale reference system is a reference coordinate system with the zero plane as the elevation reference and coordinate units consistent with the real physical space scale. The zero-plane parameter is used to determine the elevation zero reference in the reference system, and the distance of the reconstructed point relative to the zero plane represents the physical wave height.

[0067] After obtaining the real-scale binocular geometric relationships and zero-plane parameters, stereo reconstruction is first performed on the synchronized image pairs. Specifically, three-dimensional wavefront reconstruction is performed on the synchronized image pairs to obtain a three-dimensional wavefront representation including: Stereo correction is performed on synchronized image pairs based on relative rotation and real-scale relative translation. A dense disparity map is generated by using feature matching relationships as disparity priors and combining them with local cost volumes. Based on the principle of triangulation and combined with real-scale distance, the 3D coordinates of each pixel in the dense disparity map are calculated to obtain a 3D representation of the wavefront. The local cost volume is derived from two real-time images after stereo correction. First, the candidate disparity range for each pixel is determined based on sparse matching points, disparity priors, and local epipolar constraints. Then, under each candidate disparity, the matching cost between a local window in one real-time image and the corresponding window in the other real-time image is calculated. Finally, the costs under different spatial locations and different disparities are stacked to form the local cost volume.

[0068] In the equivalent parallel binocular coordinate system after stereo correction, the pixel disparity of any pixel in the dense disparity map. Its three-dimensional coordinates It can be recovered from the real-scale binocular geometry as follows:

[0069] in, f Indicates focal length. The focal length and principal point coordinates are indicated by the camera intrinsic parameter calibration. u The x-axis coordinates of the image coordinate system. v The coordinates are the vertical axis coordinates of the image coordinate system.

[0070] Then, based on the zero plane parameters, the three-dimensional coordinates of each reconstructed pixel are mapped to the real-scale reference system. The physical wave height of each reconstructed point relative to the zero plane is calculated with the zero plane as the reference, resulting in a real-scale three-dimensional wavefront with real physical scale and absolute wavefront elevation.

[0071] For any reconstructed point in the camera coordinate system Its physical wave height relative to the zero plane can be expressed as: .

[0072] The real-scale reference frame is a coordinate space with real physical units. Its origin O is the intersection of the zero plane normal passing through the optical center of one of the two cameras and the zero plane. The XOY plane coincides with the zero plane, and the Z-axis is perpendicular to the zero plane. The coordinate values ​​of each point in the real-scale reference frame correspond to the real space size.

[0073] Furthermore, based on this real-scale three-dimensional wavefront, at least one wave parameter, such as meaningful wave height, wave height time series, wave direction, wavelength, and period, can be extracted and output as results for use in marine environmental observation, wave analysis, and ship motion forecasting.

[0074] To further verify the practical effectiveness of the method proposed in this application, experiments were conducted using two multi-rotor UAVs, each equipped with an industrial camera, GNSS, UWB, IMU, and related recording devices, to conduct collaborative observation of the target sea area. The camera used was a global shutter industrial camera with a fixed-focus lens to ensure imaging stability in moving scenes. The specific models of the UAVs, cameras, and lenses are merely examples; adjustments can be made according to actual needs in practical use, and this application does not impose any limitations.

[0075] In this experiment, two UAVs flew collaboratively along a pre-set flight path, maintaining a baseline range of approximately 5-20 m, and acquired synchronized images at a frequency of 1 Hz. Each observation session lasted approximately 30 minutes, simultaneously recording UWB, GNSS, mobile RTK, and IMU data. After the observations, offline processing was performed to complete a real-scale reconstruction of the ocean wave surface. Offline processing is only one feasible implementation method; this application can also be extended to other processing architectures and is not limited to offline processing.

[0076] In this experimental verification, external ground truth equipment such as wave rider buoys were used to verify the reconstruction results. The results show that the method proposed in this application can stably recover the true-scale three-dimensional morphology of the ocean wave surface under dual-moving platform conditions and output key parameters such as meaningful wave height. The error between the reconstructed meaningful wave height and the buoy results is controlled within the order of 0.1 m, and the zero-plane parameter estimation error is better than 5%.

[0077] The experimental results show that the method proposed in this application can not only obtain spatial morphological information of ocean waves, but also provide wave parameter inputs with real physical meaning for subsequent ship motion forecasting and marine engineering analysis, thus demonstrating significant engineering application value.

[0078] The above descriptions are merely preferred embodiments of this application, and this application is not limited to the above embodiments. It is understood that other improvements and variations that can be directly derived or conceived by those skilled in the art without departing from the spirit and concept of this application should be considered to be included within the protection scope of this application.

Claims

1. A method for real-scale reconstruction of ocean wavefronts using dual unmanned aerial vehicles (UAVs) for mobile platform observation, characterized in that, The dual-UAV ocean wavefront real-scale reconstruction method includes: Two drones are controlled to conduct collaborative observation of the target sea area according to a preset relative spatial relationship. Real-time images collected by the cameras on the two drones are simultaneously acquired to form a synchronous image pair. Multi-source motion state data collected by the sensors on the two drones are also simultaneously acquired. The multi-source motion state data is consistent with the timing of the synchronous image pair and includes ranging data, positioning data, and inertial measurement data used to characterize the relative position and motion state of the two drones. Based on positioning data and inertial measurement data, features are extracted from each real-time image in the synchronized image pair, and a feature matching relationship is established between the two real-time images. According to the feature matching relationship, the relative rotation and normalized translation direction between the two UAVs are determined. A real-scale distance constraint model is constructed by fusing multi-source motion state data, and the real-scale distance between the two UAVs is obtained by solving the model. The real-scale distance is combined with the normalized translation direction to obtain the real-scale relative translation. Based on relative rotation and real-scale relative translation, three-dimensional wavefront reconstruction is performed on the synchronized image pair to obtain a three-dimensional point set of the wavefront; based on linear wave theory, zero-plane parameter estimation is performed on the three-dimensional point set of the wavefront to obtain the initial value of the zero-plane parameter, and inertial measurement data is introduced to establish a joint optimization model of the zero-plane parameter. Solving the joint optimization model of the zero-plane parameter yields the zero-plane parameter at the current moment. The zero-plane is a reference surface characterizing the mean sea level. The three-dimensional point set of the wavefront is mapped to a real-scale reference system using the zero-plane parameters to form a real-scale three-dimensional ocean wavefront; the real-scale reference system is a reference coordinate system with the zero-plane as the elevation reference and coordinate units consistent with the real physical space scale.

2. The method for real-scale reconstruction of ocean wave surfaces using dual unmanned aerial vehicles according to claim 1, characterized in that, The neural network model is used to extract features from each real-time image in the synchronized image pair and establish a feature matching relationship between the two real-time images. The neural network model includes a convolutional backbone network, a keypoint detection head, and a descriptor generation head, which are cascaded in sequence. The convolutional backbone network encodes each real-time image in the synchronous image pair to obtain a feature tensor corresponding to each real-time image. The keypoint detection head performs keypoint detection on the feature tensor to obtain candidate keypoints. The descriptor generation head extracts local features from candidate keypoints to generate feature descriptors; Establishing feature matching relationships between two real-time images includes: The inertial measurement data and positioning data are fused and calculated to obtain the initial value of the relative rotation between the two UAVs, and the local epipolar neighborhood of the candidate keypoints corresponding to each real-time image in the synchronized image pair is determined. The candidate keypoints corresponding to one real-time image in the synchronized image pair are projected into the local epipolar neighborhood of the candidate keypoints corresponding to the other real-time image, and feature descriptor correlation matching is performed to establish the feature matching relationship between the two real-time images. The local epipolar neighborhood is a local search region in the real-time image.

3. The method for real-scale reconstruction of ocean wave surfaces using dual unmanned aerial vehicles according to claim 1, characterized in that, Based on feature matching relationships, determining the relative rotation and normalized translation directions between the two UAVs includes: By fusing inertial measurement data and positioning data, the initial values ​​of the relative rotation and normalized translation directions between the two UAVs are obtained, so as to construct an objective function for optimizing the relative geometric relationship, which includes reprojection error terms and attitude prior constraints. The relative geometric relationship optimization objective function is solved by using the bundle adjustment method within a preset neighborhood of the initial value to obtain the relative rotation and normalized translation direction between the two UAVs.

4. The method for real-scale reconstruction of ocean wave surfaces using dual unmanned aerial vehicles according to claim 3, characterized in that, The relative geometric relationship optimization objective function for: in, For camera projection function, For robust kernel functions, For the attitude prior weights, It is the set of keypoint matching points between two real-time images and , It is the i-th candidate keypoint corresponding to a real-time image in a synchronized image pair. It is the i-th candidate keypoint corresponding to another real-time image in the synchronized image pair. It is a camera rotation parameter. It's the rotation parameter of another camera. It is a camera translation parameter. It is the translation parameter of another camera. It is the 3D wavefront point corresponding to the i-th candidate keypoint. It is the initial value of the relative rotation. It is the relative rotation at the current time t. This represents a logarithmic mapping over the rotation group, used to measure the deviation between the pose prior and the visually optimized pose. This indicates the matrix transpose.

5. The method for real-scale reconstruction of ocean wave surfaces using dual unmanned aerial vehicles according to claim 2, characterized in that, The neural network model optimizes network parameters using a joint loss function. , in, It is the key point detection loss weight coefficient. It is a key point detection loss and , It is a set of pixel or feature grid locations in a real-time image. The key point detection head is in position Output keypoint probabilities, It is a key point monitoring label. and These are the weights of positive samples at key points and negative samples in the background, respectively. These are the descriptor contrast loss weight coefficients. It is a descriptor contrast loss and , and These are the normalized descriptors for candidate keypoints corresponding to two real-time images. It is a set of positive matching descriptor pairs. It is a set of negative matching descriptor pairs. It is the negative sample interval threshold; It is the geometric consistency loss weighting coefficient. It is a geometric consistency loss and , and These are the homogeneous pixel coordinates of the candidate keypoints in the two real-time images. The initial values ​​of the fundamental matrix are determined by the initial values ​​of the relative rotation and the normalized translation direction. Represents the robust penalty function. This indicates the calculation of the Euclidean norm. This represents the first element in the vector. This represents the second element in the vector.

6. The method for real-scale reconstruction of ocean wave surfaces using dual unmanned aerial vehicles according to claim 1, characterized in that, The objective function of the real-scale distance constraint model is constructed by using ranging data, positioning data, and inertial measurement data as observation factors. The objective function of the real-scale distance constraint model is solved using the factor graph optimization method to obtain the real-scale distance between the two UAVs at the current time t. ; to measure the actual distance As a scaling factor, it acts on the normalized translation direction. The real-scale relative translation is obtained. .

7. The method for real-scale reconstruction of ocean wave surfaces using dual unmanned aerial vehicles according to claim 6, characterized in that, The objective function of the real-scale distance constraint model for: in, For residuals and , This refers to the measured distance in the ranging data. For real-scale distances, To locate the residual, For relative positioning residuals, For motion residuals, subscript , , , These are the covariance matrices of the ranging residual, positioning residual, relative positioning residual, and motion residual, respectively. Denotes the inverse of the covariance matrix. This represents the weighted 2-norm for calculating the inverse of the covariance matrix.

8. The method for real-scale reconstruction of ocean wave surfaces by dual unmanned aerial vehicles according to claim 1, characterized in that, The zero plane parameters at the current time t include: The zero plane is parameterized as a combination of normal vector and bias: , The zero plane normal vector, Zero plane offset, It is a three-dimensional point set of the wavefront and = , It is the first point in the three-dimensional point set of the wavefront j One point, It is the total number of points in the three-dimensional point set of the wavefront; the zero plane offset is used to characterize the positional relationship of the zero plane relative to the camera coordinate system; Based on the physical property in linear wave theory that the statistical mean of wavefront undulation is zero, the initial value of the zero plane offset is determined. , It is the initial value of the zero plane normal vector; the objective function of the joint optimization model of the zero plane parameters is constructed by using inertial measurement data as motion constraints, and the initial value of the zero plane bias is used as the basis. The objective function of the joint optimization model of the zero plane parameters is solved using the factor graph optimization method, and the zero plane normal vector at the current time t is obtained. and zero plane bias .

9. The method for real-scale reconstruction of ocean wave surfaces by dual unmanned aerial vehicles according to claim 8, characterized in that, The objective function of the zero-plane parameter joint optimization model for: in, The normal prior obtained from inertial measurement data. As a weight for time continuity, This represents the zero-plane offset at the previous time t-1. This represents the variance of the zero-plane bias calculation. This represents the weighted 2-norm of the covariance inverse matrix of the normal vector.

10. The method for real-scale reconstruction of ocean wave surfaces by dual unmanned aerial vehicles according to claim 1, characterized in that, Three-dimensional wavefront reconstruction is performed on the synchronized image pair to obtain a three-dimensional wavefront point set including: Stereo correction is performed on the synchronized image pairs based on relative rotation and real-scale relative translation; A dense disparity map is generated by using feature matching relationships as disparity priors and combining them with local cost volumes. Based on the principle of triangulation and combined with real-scale distance, the three-dimensional coordinates of each pixel in the dense disparity map are calculated to obtain the three-dimensional point set of the wavefront.

Images