Ocean wave field reconstruction method and device based on binocular vision

Abstract

The invention belongs to the technical field of ocean observation, and discloses an ocean wave field reconstruction method and device based on binocular vision. An ocean wave field reconstruction method based on binocular vision comprises the steps that internal parameters of a camera are calibrated through a Bouguet calibration algorithm, a binocular camera provided with an IMU sensor is used for collecting images and pose information, feature points are extracted through an SURF algorithm, feature matching is completed under a game theory framework, and abnormal points are removed through an RANSAC algorithm. Then, camera parameters are calculated based on an eight-point method, and the camera parameters and three-dimensional point coordinates are optimized through an SBA algorithm; thirdly, generating a disparity map by adopting epipolar correction and an SGM algorithm, and performing morphological processing on the disparity map; and the processed image is subjected to triangulation to generate a three-dimensional point cloud, and filtering is carried out. And finally, estimating a sea level position through an RANSAC algorithm, and optimizing a result through a least square method to complete three-dimensional reconstruction of ocean waves.

Description

Technical Field

[0001] The present application belongs to the field of computer vision and ocean engineering, and relates to a method and device for reconstructing ocean wave fields based on binocular vision. Background Art

[0002] Accurate measurement of ocean wave fields is an important topic in marine science and engineering research. Traditional wave measurement methods, such as the use of wave buoys, radars and other equipment, can provide local or global wave information, but are usually limited by equipment deployment and data acquisition costs, making it difficult to comprehensively and accurately measure the three-dimensional structure of the wave field. In addition, existing measurement methods are difficult to cope with dynamically changing wave characteristics in complex marine environments, and their measurement accuracy and real-time performance are insufficient.

[0003] With the development of computer vision technology, wave measurement methods based on binocular vision have gradually attracted the attention of scholars. Binocular vision simulates the working principle of the human visual system and uses two cameras to capture different perspectives of the same scene, thereby extracting three-dimensional information. This method is not only low-cost, but also capable of non-contact measurement of a large range of wave surfaces with high accuracy and real-time performance. However, the current binocular vision wave measurement technology still has some challenges, such as the robustness of feature point matching, the accuracy of parallax calculation, and the stability of the wave three-dimensional reconstruction model.

[0004] Therefore, there is an urgent need for an ocean wave field reconstruction method that can combine advanced image processing algorithms with binocular vision technology to improve the accuracy and efficiency of wave measurement and expand its application in marine engineering. Summary of the invention

[0005] In view of this, it is necessary to provide a method and device for ocean wave field reconstruction based on binocular vision to solve the existing problems of low measurement accuracy, insufficient real-time performance and difficulty in coping with complex ocean environments in the process of three-dimensional reconstruction of wave fields.

[0006] A method for reconstructing ocean wave fields based on binocular vision, the steps are as follows:

[0007] Step 1: Use a binocular camera to shoot at least 15 sets of calibration plate images at different positions and viewing angles, each set of calibration plate images includes a left view and a right view; use the Bouguet calibration algorithm based on Zhang Zhengyou's calibration method to estimate the binocular camera intrinsic parameter matrix and perform distortion correction;

[0008] Step 2: Use a binocular camera equipped with an IMU sensor to collect the wavefront image of the target area and the three-dimensional pose information of the binocular camera, extract the feature points of the left view and the right view according to the SURF algorithm, and generate feature descriptors; perform feature matching on the feature points based on the feature matching of the game theory framework, and eliminate outliers through the RANSAC algorithm to obtain matching points;

[0009] Step 3: Use the eight-point method to calculate the basic matrix of the matching points, and then calculate the essential matrix; decompose the essential matrix through SVD to obtain the camera extrinsic matrix, so as to determine the relative position of the two cameras, and optimize the camera parameters and 3D point coordinates as a whole according to the SBA algorithm;

[0010] Step 4: Perform epipolar correction on the left and right views according to the epipolar correction principle of the Bouguet algorithm to ensure that the epipolar lines in the image are parallel to the camera baseline. Crop the corrected image to eliminate the unmatched area, and reduce the search for corresponding matching points from two dimensions to one dimension. Perform stereo matching and morphological processing on the corrected image according to the SGM algorithm to obtain a disparity map.

[0011] Step 5. Generate a 3D point cloud using triangulation based on the disparity map, and filter the 3D point cloud in combination with the application scenario. Estimate the average sea level position using the RANSAC algorithm, and perform least squares fitting on all internal points after obtaining the initial estimated sea level position. Introduce the distance from the point to the camera as a weight coefficient to improve the initial sea level estimation, and finally generate a wave 3D reconstruction model at the estimated sea level position.

[0012] Further, in step 2, the SURF algorithm includes:

[0013] Constructing the Hessian matrix

[0014] in, G(x,y,σ) is the Gaussian convolution kernel, I(x,y) is the grayscale value of the input image;

[0015] Use a box filter instead to process the Gaussian second-order filter, i.e. in, Representation Matrix The approximate value of the determinant, w = 0.9 is the weight value of the matrix; D xx , D yy , D xyThey are box filter templates used to calculate the gradient in the x, y, and oblique directions respectively; box filter templates of different sizes are used to convolve with the calibration plate image collected by the binocular camera to replace the Gaussian filter to construct the scale space; the Hessian matrix determinant is used to distinguish the positive and negative and non-maximum suppression methods of the maximum and minimum values, and the extreme points are detected in the 3×3×3 range of the stereo neighborhood of each scale layer pixel point except the top and bottom layers of the scale space, and the point with the largest absolute value is the candidate feature point;

[0016] The Harr wavelet responses of the candidate feature points in the x and y directions are counted. The vector with the largest component in the x and y directions is the main direction of the feature point. Finally, the feature descriptor is generated in the square area around the feature point. The size of the square area is 20S cale ×20S cale , S cale It is the characteristic scale of the layer where the feature point is located, and its direction is consistent with the main direction.

[0017] Further, in step 2, the feature matching based on the game theory framework combines affine transformation consistency and epipolar geometric constraints to achieve matching of feature points;

[0018] Initial stage of feature matching: Each pair of candidate feature points that may be matched is regarded as a matching strategy in game theory, and all matching strategies constitute a strategy set O = {1, ..., n}; the matching pairs of multiple candidate feature points compete through the game process, and the benefit of each matching strategy depends on its geometric consistency with other matching pairs. The benefit matrix C = (c ij ), where c ij is the profit obtained by the individual using matching strategy i against matching strategy j; the mixed strategy is the random initialization of the available matching strategies, that is, the probability distribution x on the strategy set O = (x1, x2, ..., x n ) T , restricted to the n-dimensional standard simplex In, x i is the probability of selecting the corresponding pure matching strategy, x i ≥0,

[0019] Matching strategy game selection stage: The selected matching strategy x should first satisfy the Nash equilibrium. For any mixed strategy y∈Δ, x T Cx≥y T Cy; select an evolutionary method to calculate Nash equilibrium, that is, evolutionary stable strategy, matching strategy x satisfies Nash equilibrium and meets the following conditions: Considering the efficiency and simplicity of selection, the selection process is driven by the imitator dynamic equation. i tis the number of iterations; different matches are allowed to compete with each other in this game theory framework, and each match is supported by compatible associations while bearing the competitive pressure from all other matches; the selection process is simulated by running the imitator dynamic equation, and at equilibrium, only mutually compatible matches are retained and regarded as interior points;

[0020] Combined with the consistency of affine transformation, the matching game stage is defined as follows: given a set of feature points M in the source image and a set of feature points D in the target image, any feature point set consisting of a g ∈M, g=1,2,...,m and b h ∈D, h=1,2,...,n' composed of matching strategy pairs (a g ,b h ) is the matching strategy, and the set of all matching strategies is Since the feature point extractor extracts thousands of feature points from the source image, for each feature point of the source image, k matching strategies are generated according to the descriptor distance, and connected to the k nearest features in the target image to reasonably screen and limit the scale, k = 3~4; define a profit function Π: In order to utilize some pairwise information available before estimating the stereo camera and scene parameters, the pairwise information is the similarity between the matching strategies, thereby enforcing consistency globally; since the position, scale and rotation information recorded by the feature descriptor is associated with each feature point, each match (a g ,b h ) is associated with a similarity transformation T(a g ,b h ), mapping the position, orientation and scale in the source image to the target image; under the assumption of small-scale motion, it is expected that these similarity transformations are similar locally; when defining the benefit function Π, one-to-one matching is achieved based on the concept of reprojection and similarity. The two matching strategies (a w1 ,b w2 ) and (a z1 ,b z2 ) and its similarity transformation T(a w1 ,b w2 ) and T(a z1 ,b z2 ), where w1 and z1 are any integers in the interval [1, m]; w2 and z2 are any integers in the interval [1, n']; define a virtual point b w2 '=T(a z1 ,b z2 ) w1 ; b z2 '=T(a w1 ,b w2 ) z1 ,according to

[0021] Measure (a w1 ,b w2 ) and (a z1 ,b z2 ), where λ is a selectivity parameter; when λ is small, the similarity function has a strong tolerance to the deviation in the similarity transformation, and the larger λ is, the stronger the selectivity is; the benefit function Π is defined as:

[0022]

[0023] A game setting based on epipolar constraints is proposed to further optimize the matching strategy. In this game, the set S of matching strategies corresponds to paired feature groups extracted from the affine matching game, i.e., the matching game defined in combination with affine transformation consistency, and the benefits between these feature groups are related to their conformity to the common epipolar geometry. Specifically, given two pairs of matching groups and Estimate the epipolar geometry from the set a∪b and define the payoff between them as:

[0024]

[0025] Among them, l(s) is a function that obtains the epipolar line l in the target image according to the feature point t in the source image; d(t,l) represents the distance between the feature point t and the epipolar line l; the above operation optimizes the selection of matching pairs and finally realizes the feature matching operation.

[0026] Further, in step 3, the eight-point method refers to the projection space coordinate relationship x' T Fx=0 is transformed into Af=0, where x' and x are the projective space coordinates of the feature matching point pair; F is the basic matrix; f contains the eight unknown parameters to be solved. 11 ~F 32 , use singular value decomposition A = U'S'V' on the A matrix T , where U' and V' are orthogonal matrices, S' is a diagonal matrix containing singular values; after obtaining the basic matrix, add a singularity constraint; let the final solution be F', under the condition of detF'=0, the final solution is to obtain F' with the smallest Frobenius norm ||F-F'||; the solution process also uses SVD decomposition: if F=U"DV" T , where U" and V" are orthogonal matrices; the diagonal matrix D = diag(r,s,t'), satisfies r≥s≥t', and t' is replaced by 0, then F' = U"diag(r,s,0)V" TIt is the solution that minimizes the norm ||F-F'||, that is, the final solution; then the calculated three-dimensional point coordinates are further iteratively optimized according to the bundle adjustment.

[0027] Further, the SBA algorithm is a method for reducing the amount of calculation and improving efficiency by constructing a sparse matrix based on bundle adjustment;

[0028] The solution optimization of sparse bundle adjustment relies on the Levenberg-Marquette algorithm, whose normal equation is: J T Jδ P +μδ P =(J T J+μI)δ P =J T ∈;

[0029] Where J is the Jacobian equation; δ P Update the camera parameter increment for each iteration; μ is the damping coefficient; I is the identity matrix; ∈ represents the residual;

[0030] If the covariance matrix Σ of the uncertainty of the observed quantity is available, the normal equation is transformed into:

[0031]

[0032] The binocular camera observes q points, and the sparse form of its Jacobian matrix J is:

[0033]

[0034] in Represented as x i'j' The corresponding estimated measurement vector; defined as a parameter vector by 2 projection matrices and all parameters of q 3D points This defines a j' 、b i' ;

[0035] Due to the sparsity of J, the normal equations are themselves sparse; let the covariance matrix of the complete measurement vector X be a block diagonal matrix

[0036] Therefore, the normal equation obtained based on J is also sparse:

[0037]

[0038] in, in * represents the augmentation of the diagonal elements of U and V; is the update vector of camera parameters; represents the update vector of the structural parameters, The residual matrix representing the camera parameters; Residual matrix representing the structural parameters;

[0039] In the iteration according to (U * -WV *-1 W T )δ a =∈ a -WV *-1 b Update parameter δ a , use Cholesky decomposition to solve, and solve for δ a After that, according to V * δ b =∈ b -W T δ a Solving for δ b , after multiple iterations, the optimized result is obtained.

[0040] Further, in step 4, the Bouguet epipolar correction algorithm is a correction method that, based on the relative posture relationship between the left image and the right image, rotates the right image through a rotation matrix so that the right image is parallel to the corresponding pixel coordinates of the left image, so that the poles of the left image and the right image are both at infinity, and the epipolar line is parallel to the baseline; the correction method comprises the following steps:

[0041] (1) Decompose the rotation matrix R of the right view relative to the left view into two matrices R l and R r :

[0042] R l =R 1 / 2

[0043] R r =R 1 / 2

[0044] Its function is to rotate the left and right cameras by half the angle between the optical axes of the two cameras, so that the imaging planes of the left and right cameras are parallel;

[0045] (2) Constructing the matrix R rect Make the baseline parallel to the imaging plane, which is accomplished by the offset matrix T of the right camera relative to the left camera;

[0046]

[0047] in, T x , T yRespectively represent the offset of the right camera relative to the left camera in the x-axis and y-axis directions; e3 = e1 × e2;

[0048] (3) Construct the overall correction matrix of the left and right cameras:

[0049] R l '=R rect R l

[0050] R r '=R rect R r

[0051] (4) The correction matrix is ​​multiplied by the respective camera coordinate systems to obtain the corrected matrix. At this point, the image is distorted and needs to be cropped to deal with the areas that cannot be matched; a new image center and edge are selected to maximize the overlap between the left and right images, completing the post-processing of the epipolar corrected image.

[0052] Further, in step 5, the point cloud is filtered in combination with the application scenario, and all points with a depth less than 1m and points with a depth greater than a set threshold are filtered out from the unfiltered three-dimensional point cloud, wherein the threshold is adjusted according to the application scenario and the baseline distance of the camera; all points with an angle of less than 20 degrees between two intersecting camera rays are filtered out;

[0053] Since the SGM algorithm is designed for a general scenario, the continuous and smooth characteristics of the ocean surface are used to further filter the point cloud. First, a graph with three-dimensional points as vertices is constructed, and each vertex is connected to its four adjacent points in the image lattice topology. If two three-dimensional points come from adjacent pixels in the right view, they are connected by an edge. The weight of each edge is calculated, and the weight is defined as the absolute value of the coordinate difference between the two connected points in the z-axis direction of the camera. Edges with weights greater than the 98th percentile of the weight distribution are trimmed, and those points that have mutations in the z-axis direction relative to neighboring points are disconnected. Finally, discrete points that do not belong to the maximum connected component after pruning are filtered out, thereby completing the cleaning and filtering operations of the point cloud.

[0054] A device for reconstructing ocean wave fields based on binocular vision, comprising:

[0055] Preprocessing and data acquisition module: pre-calibrate the camera, estimate the camera intrinsic parameters and perform de-distortion processing according to the Bouguet calibration algorithm, and use a binocular camera equipped with an IMU sensor to collect wavefront images and the camera's 3D pose information;

[0056] The feature matching module extracts feature points based on the SURF algorithm, performs feature matching based on the game theory framework, and finally uses the RANSAC algorithm to eliminate outliers;

[0057] The parameter optimization module uses the eight-point method to calculate the basic matrix and essential matrix for the feature matching point pairs, and uses SVD decomposition to obtain the external parameters of the camera; the camera parameters and 3D point coordinates are further optimized based on the sparse bundle adjustment;

[0058] The stereo matching module performs epipolar correction and post-processing of the left and right views according to the correction principle of the Bouguet algorithm, obtains the disparity map through stereo matching using the SGM algorithm, and optimizes the disparity map through morphological operations;

[0059] The point cloud processing module uses triangulation method to generate unfiltered 3D point cloud based on the disparity map. After reasonable filtering operation, the point cloud is further filtered by introducing the application scenario where the ocean surface is continuous and smooth everywhere.

[0060] The wave reconstruction module estimates the average sea level position according to the RANSAC algorithm. After obtaining the initial estimated position, it fits all internal points according to the least squares method and introduces weight coefficients to improve the initial estimated sea level position, and finally generates a wave three-dimensional reconstruction model at the estimated sea level position.

[0061] Furthermore, the disparity map processed by SGM in the stereo matching module tends to show errors at the boundary of the matching area, and morphological operations need to be performed to process the disparity map; first, a dilation operation is performed, and the initial dilation operation can close many small holes on the observed sea surface caused by sun glare or other artifacts; the subsequent erosion operation will eliminate the disparity value at the boundary of the disparity map; and the disparity map before and after morphological processing is compared to determine the appropriate number of dilation and erosion operations.

[0062] Furthermore, after the point cloud filtering process, the wave reconstruction module estimates the average sea level position according to the RANSAC algorithm, sets the appropriate inlier threshold and number of iterations according to the required accuracy and computer performance, and obtains the initial estimated sea level position; then, the least squares fitting is performed on all the inliers of this model, and the distance from the point to the camera is introduced as the weight coefficient to improve the initial sea level estimation, and finally a wave three-dimensional reconstruction model is generated at the estimated sea level position.

[0063] The present invention is based on binocular vision technology, combined with a number of advanced algorithms and image processing technologies, to achieve efficient and accurate three-dimensional reconstruction of ocean wave fields. By using the SURF algorithm for feature extraction and creatively introducing a feature matching method based on the game theory framework, the robustness and reliability of feature matching are greatly improved; the SBA algorithm is used to optimize the camera parameters and three-dimensional point coordinates as a whole, which significantly improves the calculation efficiency; the left and right views are corrected according to the epipolar correction principle of the Bouguet algorithm, and the matching dimension of the feature points is converted from two-dimensional to one-dimensional, which greatly shortens the stereo matching time. In addition, the present invention optimizes and filters the disparity map generated by the SGM algorithm in combination with specific application scenarios, further suppresses the interference of noise and artifacts, and ensures the accuracy and reliability of the measurement results.

[0064] Compared with traditional wave measurement equipment (such as buoys, radars, etc.), the present invention uses industrial binocular cameras and efficient image processing algorithms, which not only reduces the equipment deployment cost, but also realizes non-contact three-dimensional measurement of wave fields. This method has great adaptability to complex marine environments and is suitable for a wide range of marine monitoring scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 The figure is a flow chart of a binocular vision-based ocean wave field reconstruction method in one embodiment;

[0066] Figure 2 is a schematic structural diagram of an ocean wave field reconstruction device based on binocular vision in one embodiment;

[0067] Figure 3 The original images taken by the left and right cameras of a method and device for reconstructing ocean wave fields based on binocular vision in one embodiment; wherein (a) is the original image taken by the left camera; (b) is the original image taken by the right camera;

[0068] Figure 4 A disparity map of a method and device for reconstructing an ocean wave field based on binocular vision in one embodiment;

[0069] Figure 5 A three-dimensional point cloud of a binocular vision-based ocean wave field reconstruction method and device in one embodiment;

[0070] Figure 6 The present invention is a wave reconstruction model of a method and device for reconstructing ocean wave fields based on binocular vision in one embodiment.

[0071] Figure 7 The present invention is a schematic diagram of the application structure of a method and device for reconstructing an ocean wave field based on binocular vision in one embodiment. DETAILED DESCRIPTION

[0072] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0073] In one embodiment, Figure 1 As shown, a method for reconstructing ocean wave fields based on binocular vision is provided, comprising the following steps:

[0074] Step 101, use the same model of left and right industrial cameras to shoot 15 groups of complete calibration plate images at different positions and viewing angles in advance, estimate the camera intrinsic parameter matrices K1 and K2 according to the Bouguet calibration algorithm based on Zhang Zhengyou calibration method, and perform distortion correction.

[0075] Step 102: A binocular camera equipped with an IMU sensor collects a wavefront image of the target area and the three-dimensional position information of the camera, such as Figure 3 As shown in the figure, the feature points of the left and right views are extracted according to the SURF algorithm to generate feature descriptors. The matching operation is completed according to the feature matching based on the game theory framework, and finally the outliers are eliminated through the RANSAC algorithm.

[0076] Step 103, the basic matrix and the essential matrix are calculated according to the matching points using the eight-point method. The essential matrix is ​​decomposed by SVD to obtain the rotation matrix R and the translation matrix T, so the relative position of the two cameras is also determined.

[0077] For the above steps, the eight-point method is used to calculate the basic matrix F according to the following formula based on the matching points:

[0078]

[0079] After determining F, according to Derive the essential matrix E.

[0080] Step 104: The camera parameters determined in step 103 still need to be further optimized. The camera parameters and 3D point coordinates are optimized as a whole according to the SBA algorithm. The objective function is: After multiple iterations, the optimized result is obtained.

[0081] Step 105, epipolar correction is performed on the left and right views according to the epipolar correction principle of the Bouguet algorithm to ensure that the epipolar lines in the image are parallel to the camera baseline, thereby reducing the search for corresponding matching points from two dimensions to one dimension.

[0082] For the above steps, construct the overall correction matrix for the left and right cameras:

[0083] R l '=R rect R l

[0084] R r '=Rrect R r

[0085] The correction matrix is ​​multiplied by the respective camera coordinate system to obtain the corrected matrix. At this time, the image contains areas that cannot be matched, and the image needs to be cropped. Reselect an image center and edge to maximize the overlap between the left and right images, and complete the post-processing of the epipolar correction image.

[0086] Step 106: Perform stereo matching according to the SGM algorithm and select the minimum cost disparity min d S(p,d) performs disparity estimation, and performs sub-pixel estimation through the quadratic curve fitted by the adjacent cost; according to the left and right consistency check, by judging d L (x) and d R Whether (x′) is consistent determines whether the match is correct or not; due to low texture, reflection, noise and other reasons, the disparity image may also contain erroneous points. These points usually show small areas with large disparity differences from the surrounding areas. The predefined threshold is 16 to filter these smaller areas; the disparity map processed by SGM tends to show errors at the boundaries of the matching areas, so morphological processing operations are performed. First, two dilation operations are performed. The initial dilation operation can close many small holes on the observed sea surface caused by sun glare or other artifacts. Then two erosion operations are performed to eliminate the disparity values ​​at the boundaries of the disparity map. The final disparity map is as follows Figure 4 .

[0087] Step 107, generate an unfiltered 3D point cloud using triangulation based on the disparity map, filter out all points with a depth less than 1m and points with a depth greater than a threshold of 3m, and filter out all points where the angle between two intersecting camera rays is less than 20 degrees; construct vertex topological connecting lines, and trim connecting lines with weights greater than the 98th percentile of the weight distribution, in order to filter out points that have abrupt changes in the z-axis direction relative to neighboring points. Finally, filter these discrete points to complete the point cloud cleaning and filtering operations.

[0088] Step 108, estimate the average sea level position according to the RANSAC algorithm, set the inlier threshold to 3, iterate 200 rounds, and obtain the initial estimated sea level position; then perform least squares fitting on all the inliers of this model, introduce the distance from the point to the camera as the weight coefficient, improve the initial sea level estimation, and finally generate a wave 3D reconstruction model at the estimated sea level position.

[0089] In one embodiment, Figure 2 As shown, a device for reconstructing ocean wave fields based on binocular vision is provided, comprising:

[0090] The preprocessing and data acquisition module 201 pre-calibrates the camera, obtains the camera intrinsic parameter matrix by minimizing the reprojection error, and uses a binocular camera equipped with an IMU sensor to collect the wavefront image and the camera's three-dimensional pose information.

[0091] The feature matching module 202 extracts feature points according to the SURF algorithm, performs feature matching on the extracted feature points according to the game theory framework, and finally uses the RANSAC algorithm to eliminate outliers.

[0092] The parameter optimization module 203 uses the eight-point method to calculate the basic matrix and the essential matrix for the feature matching point pairs, and uses SVD decomposition to obtain the external parameters of the camera. The camera parameters and the three-dimensional point coordinates are further optimized according to the sparse bundle adjustment.

[0093] The stereo matching module 204 performs epipolar correction and post-processing of the corrected images on the left and right views according to the correction principle of the Bouguet algorithm, obtains a disparity map by stereo matching through the SGM algorithm, and optimizes the disparity map through appropriate morphological operations (dilation, erosion, etc.).

[0094] The point cloud processing module 205 generates an unfiltered three-dimensional point cloud using a triangulation method according to the disparity map, and further filters the point cloud by introducing an application scenario in which the ocean surface is continuous and smooth everywhere after a reasonable filtering operation.

[0095] The wave reconstruction module 206 estimates the average sea level position according to the RANSAC algorithm, and after obtaining the initial estimated position, fits all the internal points according to the least squares method and introduces weight coefficients to improve the initial estimated sea level position, and finally generates a wave three-dimensional reconstruction model at the estimated sea level position.

[0096] It should be noted that the terms "first\second\third" involved in the embodiments of the present invention are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that the specific order or sequence of "first\second\third" can be interchanged where permitted. It should be understood that the objects distinguished by "first\second\third" can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0097] The terms "including" and "having" and any variations thereof in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or (module) units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0098] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0099] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0100] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0101] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for reconstructing ocean wave fields based on binocular vision, characterized in that: Here are the steps: Step 1: Use a binocular camera to shoot at least 15 sets of calibration plate images at different positions and viewing angles, each set of calibration plate images includes a left view and a right view; use the Bouguet calibration algorithm based on Zhang Zhengyou's calibration method to estimate the binocular camera intrinsic parameter matrix and perform distortion correction; Step 2: Use a binocular camera equipped with an IMU sensor to collect the wavefront image of the target area and the three-dimensional pose information of the binocular camera, extract the feature points of the left view and the right view according to the SURF algorithm, and generate feature descriptors; perform feature matching on the feature points based on the feature matching of the game theory framework, and eliminate outliers through the RANSAC algorithm to obtain matching points; Step 3: Use the eight-point method to calculate the basic matrix of the matching points, and then calculate the essential matrix; The camera extrinsic matrix is ​​obtained by decomposing the essential matrix through SVD, so as to determine the relative position of the two cameras, and the camera parameters and 3D point coordinates are optimized as a whole according to the SBA algorithm; Step 4: Perform epipolar correction on the left and right views according to the epipolar correction principle of the Bouguet algorithm to ensure that the epipolar lines in the image are parallel to the camera baseline. Crop the corrected image to eliminate the unmatched area, and reduce the search for corresponding matching points from two dimensions to one dimension. Perform stereo matching and morphological processing on the corrected image according to the SGM algorithm to obtain a disparity map. Step 5. Generate a 3D point cloud using triangulation based on the disparity map, and filter the 3D point cloud in combination with the application scenario. Estimate the average sea level position using the RANSAC algorithm, and perform least squares fitting on all internal points after obtaining the initial estimated sea level position. Introduce the distance from the point to the camera as a weight coefficient to improve the initial sea level estimation, and finally generate a wave 3D reconstruction model at the estimated sea level position.

2. The method for reconstructing ocean wave fields based on binocular vision according to claim 1, characterized in that: In step 2, the SURF algorithm includes: Constructing the Hessian matrix in, G(x,y,σ) is the Gaussian convolution kernel, I(x,y) is the grayscale value of the input image; Use a box filter instead to process the Gaussian second-order filter, i.e. and in, Representation Matrix The approximate value of the determinant, w = 0.9 is the weight value of the matrix; D xx , D yy , D xy They are box filter templates used to calculate the gradient in the x, y, and oblique directions respectively; box filter templates of different sizes are used to convolve with the calibration plate image collected by the binocular camera to replace the Gaussian filter to construct the scale space; the Hessian matrix determinant is used to distinguish the positive and negative and non-maximum suppression methods of the maximum and minimum values, and the extreme points are detected in the 3×3×3 range of the stereo neighborhood of each scale layer pixel point except the top and bottom layers of the scale space, and the point with the largest absolute value is the candidate feature point; The Harr wavelet responses of the candidate feature points in the x and y directions are counted. The vector with the largest component in the x and y directions is the main direction of the feature point. Finally, the feature descriptor is generated in the square area around the feature point. The size of the square area is 20S cale ×20S cale , S cale It is the characteristic scale of the layer where the feature point is located, and its direction is consistent with the main direction.

3. The method for reconstructing ocean wave fields based on binocular vision according to claim 1, characterized in that: In step 2, the feature matching based on the game theory framework combines affine transformation consistency and epipolar geometric constraints to achieve feature point matching; Initial stage of feature matching: Each pair of candidate feature points that may be matched is regarded as a matching strategy in game theory, and all matching strategies constitute a strategy set O = {1, ..., n}; the matching pairs of multiple candidate feature points compete through the game process, and the benefit of each matching strategy depends on its geometric consistency with other matching pairs. The benefit matrix C = (c ij ), where c ij is the profit obtained by the individual using matching strategy i against matching strategy j; the mixed strategy is the random initialization of the available matching strategies, that is, the probability distribution x on the strategy set O = (x1, x2, ..., x n ) T , restricted to the n-dimensional standard simplex In, x i is the probability of selecting the corresponding pure matching strategy, x i ≥0, i∈1,…,n and Matching strategy game selection stage: The selected matching strategy x should first satisfy the Nash equilibrium. For any mixed strategy y∈Δ, x T Cx≥y T Cy; select an evolutionary method to calculate Nash equilibrium, that is, evolutionary stable strategy, matching strategy x satisfies Nash equilibrium and meets the following conditions: Considering the efficiency and simplicity of selection, the selection process is driven by the imitator dynamic equation. i t is the number of iterations; different matches are allowed to compete with each other in this game theory framework, and each match is supported by compatible associations while bearing the competitive pressure from all other matches; the selection process is simulated by running the imitator dynamic equation, and at equilibrium, only mutually compatible matches are retained and regarded as interior points; Combined with the consistency of affine transformation, the matching game stage is defined as follows: given a set of feature points M in the source image and a set of feature points D in the target image, any feature point set consisting of a g ∈M, g=1,2,...,m and b h ∈D, h=1,2,...,n' composed of matching strategy pairs (a g ,b h ) is the matching strategy, and the set of all matching strategies is Since the feature point extractor extracts thousands of feature points from the source image, for each feature point of the source image, k matching strategies are generated according to the descriptor distance, and connected to the k nearest features in the target image to reasonably screen and limit the scale, k = 3~4; define a profit function Π: In order to utilize some pairwise information available before estimating the stereo camera and scene parameters, the pairwise information is the similarity between the matching strategies, thereby enforcing consistency globally; since the position, scale and rotation information recorded by the feature descriptor is associated with each feature point, each match (a g ,b h ) is associated with a similarity transformation T(a g ,b h ), mapping the position, orientation and scale in the source image to the target image; under the assumption of small-scale motion, it is expected that these similarity transformations are similar locally; when defining the benefit function Π, one-to-one matching is achieved based on the concept of reprojection and similarity. The two matching strategies (a w1 ,b w2 ) and (a z1 ,b z2 ) and its similarity transformation T(a w1 ,b w2 ) and T(a z1 ,b z2 ), where w1 and z1 are any integers in the interval [1, m]; w2 and z2 are any integers in the interval [1, n']; define a virtual point b w2 '=T(a z1 ,b z2 ) w1 ; b z2 '=T(a w1 ,b w2 ) z1 ,according to Measure (a w1 ,b w2 ) and (a z1 ,b z2 ), where λ is a selectivity parameter; when λ is small, the similarity function has a strong tolerance to the deviation in the similarity transformation, and the larger λ is, the stronger the selectivity is; the benefit function Π is defined as: A game setting based on epipolar constraints is proposed to further optimize the matching strategy. In this game, the set S of matching strategies corresponds to paired feature groups extracted from the affine matching game, i.e., the matching game defined in combination with affine transformation consistency, and the benefits between these feature groups are related to their conformity to the common epipolar geometry. Specifically, given two pairs of matching groups and Estimate the epipolar geometry from the set a∪b and define the payoff between them as: Among them, l(s) is a function that obtains the epipolar line l in the target image according to the feature point t in the source image; d(t,l) represents the distance between the feature point t and the epipolar line l; the above operation optimizes the selection of matching pairs and finally realizes the feature matching operation.

4. The method for reconstructing ocean wave fields based on binocular vision according to claim 1, characterized in that: In step 3, the eight-point method refers to the projection space coordinate relationship x' T Fx=0 is transformed into Af=0, where x' and x are the projective space coordinates of the feature matching point pair; F is the basic matrix; f contains the eight unknown parameters to be solved. 11 ~F 32 , use singular value decomposition A = U'S'V' on the A matrix T , where U' and V' are orthogonal matrices, S' is a diagonal matrix containing singular values; after obtaining the basic matrix, add a singularity constraint; let the final solution be F', under the condition of detF'=0, the final solution is to obtain F' with the smallest Frobenius norm ||F-F'||; The solution process also uses SVD decomposition: if F = U"DV" T , where U" and V" are orthogonal matrices; the diagonal matrix D = diag(r,s,t'), satisfies r≥s≥t', and t' is replaced by 0, then F' = U"diag(r,s,0)V" T is the solution that minimizes the norm ||F-F'||, that is, the final solution; Then the calculated three-dimensional point coordinates are iteratively optimized based on bundle adjustment.

5. The method for reconstructing ocean wave fields based on binocular vision according to claim 1, characterized in that: The SBA algorithm is a method for reducing the amount of calculation and improving efficiency by constructing a sparse matrix based on bundle adjustment; The solution optimization of sparse bundle adjustment relies on the Levenberg-Marquette algorithm, whose normal equation is: J T Jδ P +μδ P =(J T J+μI)δ P =J T ∈; Where J is the Jacobian equation; δ P Update the camera parameter increment for each iteration; μ is the damping coefficient; I is the identity matrix; ∈ represents the residual; If the covariance matrix Σ of the uncertainty of the observed quantity is available, the normal equation is transformed into: The binocular camera observes q points, and the sparse form of its Jacobian matrix J is: in Represented as x i'j' The corresponding estimated measurement vector; defined as a parameter vector by 2 projection matrices and all parameters of q 3D points This defines a j' 、b i' ; Due to the sparsity of J, the normal equations are themselves sparse; let the covariance matrix of the complete measurement vector X be a block diagonal matrix Therefore, the normal equation obtained based on J is also sparse: in, in * represents the augmentation of the diagonal elements of U and V; is the update vector of camera parameters; represents the update vector of the structural parameters, The residual matrix representing the camera parameters; Residual matrix representing the structural parameters; In the iteration according to (U * -WV *-1 W T )δ a =∈ a -WV *-1 b Update parameter δ a , use Cholesky decomposition to solve, and solve for δ a After that, according to V * δ b =∈ b -W T δ a Solving for δ b , after multiple iterations, the optimized result is obtained.

6. The method for reconstructing ocean wave fields based on binocular vision according to claim 1, characterized in that: In step 4, the Bouguet epipolar correction algorithm is a correction method that, based on the relative posture relationship between the left image and the right image, rotates the right image through a rotation matrix so that the right image is parallel to the corresponding pixel coordinates of the left image, so that the poles of the left image and the right image are both at infinity, and the epipolar lines are parallel to the baseline; the correction method comprises the following steps: (1) Decompose the rotation matrix R of the right view relative to the left view into two matrices R l and R r : R l =R 1 / 2 R r =R 1 / 2 Its function is to rotate the left and right cameras by half the angle between the optical axes of the two cameras, so that the imaging planes of the left and right cameras are parallel; (2) Constructing the matrix R rect Make the baseline parallel to the imaging plane, which is accomplished by the offset matrix T of the right camera relative to the left camera; in, T x , T y Respectively represent the offset of the right camera relative to the left camera in the x-axis and y-axis directions; e3 = e1 × e2; (3) Construct the overall correction matrix of the left and right cameras: R l '=R rect R l R r '=R rect R r (4) The correction matrix is ​​multiplied by the respective camera coordinate systems to obtain the corrected matrix. At this point, the image is distorted and needs to be cropped to deal with the areas that cannot be matched; a new image center and edge are selected to maximize the overlap between the left and right images, completing the post-processing of the epipolar corrected image.

7. The method for reconstructing ocean wave fields based on binocular vision according to claim 1, characterized in that: In step 5, the point cloud is filtered in combination with the application scenario, and all points with a depth less than 1m and points with a depth greater than a set threshold are filtered out from the unfiltered three-dimensional point cloud, wherein the threshold is adjusted according to the application scenario and the baseline distance of the camera; all points with an angle of less than 20 degrees between two intersecting camera rays are filtered out; Since the SGM algorithm is designed for a general scenario, the continuous and smooth characteristics of the ocean surface are used to further filter the point cloud. First, a graph with three-dimensional points as vertices is constructed, and each vertex is connected to its four adjacent points in the image lattice topology. If two three-dimensional points come from adjacent pixels in the right view, they are connected by an edge. The weight of each edge is calculated, and the weight is defined as the absolute value of the coordinate difference between the two connected points in the z-axis direction of the camera. Edges with weights greater than the 98th percentile of the weight distribution are trimmed, and those points that have mutations in the z-axis direction relative to neighboring points are disconnected. Finally, discrete points that do not belong to the maximum connected component after pruning are filtered out, thereby completing the cleaning and filtering operations of the point cloud.

8. A device for reconstructing ocean wave fields based on binocular vision, characterized in that: include: Preprocessing and data acquisition module: pre-calibrate the camera, estimate the camera intrinsic parameters and perform de-distortion processing according to the Bouguet calibration algorithm, and use a binocular camera equipped with an IMU sensor to collect wavefront images and the camera's 3D pose information; The feature matching module extracts feature points based on the SURF algorithm, performs feature matching based on the game theory framework, and finally uses the RANSAC algorithm to eliminate outliers; The parameter optimization module uses the eight-point method to calculate the basic matrix and essential matrix for the feature matching point pairs, and uses SVD decomposition to obtain the external parameters of the camera; the camera parameters and 3D point coordinates are further optimized based on the sparse bundle adjustment; The stereo matching module performs epipolar correction and post-processing of the left and right views according to the correction principle of the Bouguet algorithm, obtains the disparity map through stereo matching using the SGM algorithm, and optimizes the disparity map through morphological operations; The point cloud processing module uses triangulation method to generate unfiltered 3D point cloud based on the disparity map. After reasonable filtering operation, the point cloud is further filtered by introducing the application scenario where the ocean surface is continuous and smooth everywhere. The wave reconstruction module estimates the average sea level position according to the RANSAC algorithm. After obtaining the initial estimated position, it fits all internal points according to the least squares method and introduces weight coefficients to improve the initial estimated sea level position, and finally generates a wave three-dimensional reconstruction model at the estimated sea level position.

9. The binocular vision-based ocean wave field reconstruction device according to claim 8, characterized in that: The disparity map processed by SGM in the stereo matching module tends to show errors at the boundary of the matching area, and morphological operations need to be performed to process the disparity map; first, a dilation operation is performed, and the initial dilation operation can close many small holes on the observed sea surface caused by sun glare or other artifacts; the subsequent erosion operation eliminates the disparity value at the boundary of the disparity map; and the disparity map before and after morphological processing is compared to determine the appropriate number of dilation and erosion operations.

10. The device for reconstructing ocean wave fields based on binocular vision according to claim 8, characterized in that: After point cloud filtering, the wave reconstruction module estimates the average sea level position according to the RANSAC algorithm, sets the appropriate inlier threshold and number of iterations according to the required accuracy and computer performance, and obtains the initial estimated sea level position; then, the least squares fitting is performed on all the inliers of the model, and the distance from the point to the camera is introduced as the weight coefficient to improve the initial sea level estimation, and finally a wave three-dimensional reconstruction model is generated at the estimated sea level position.

Images