Sea wave measurement method based on binocular vision and laser radar

Abstract

The invention discloses an offshore wave measurement method based on binocular vision and a laser radar. The method comprises the following steps: detecting waves in a ship operation area through a laser radar to obtain laser point cloud wave data, shooting the waves through a first camera to obtain a first wave map, and shooting the waves through a second camera to obtain a second wave map; performing subtraction processing on the position data of the corresponding pixel points in the first wave map and the second wave map to obtain a wave disparity map; calculating depth information of each pixel point in the wave disparity map according to the internal reference of the binocular camera, determining point cloud data corresponding to the wave disparity map based on the depth information of each pixel point, and obtaining visual point cloud wave data; fusing the visual point cloud wave data and the laser point cloud wave data to obtain fused point cloud wave data; and performing three-dimensional modeling on waves based on the fused point cloud wave data to obtain a wave model. According to the technical scheme, the accuracy of the determined wave model is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of ships, and in particular to a method for measuring ocean waves based on binocular vision and lidar. Background Art

[0002] Ocean waves are one of the important dynamic factors in the ocean environment. The accurate measurement of their relevant parameters is of crucial significance for many fields such as ocean engineering, navigation safety, and ocean scientific research.

[0003] Currently, the methods for measuring ocean waves mainly include buoy measurement and acoustic wave measurement. However, buoy measurement is restricted by the deployment method and has limited spatial resolution, making it difficult to accurately capture the dynamic wave changes in the ship operation area; acoustic wave measurement is easily affected by turbulence and complex environments, and its universality is poor. Therefore, the accuracy of the above two measurement methods cannot meet the actual needs.

[0004] Therefore, there is an urgent need to propose a new method to solve the above problems. Summary of the Invention

[0005] The present invention provides a method for measuring ocean waves based on binocular vision and lidar, which improves the accuracy of wave measurement and further improves the accuracy of the determined wave model.

[0006] In a first aspect, an embodiment of the present invention provides a method for measuring ocean waves based on binocular vision and lidar, which is applied to a ship. A binocular camera and a lidar are installed on the ship. The binocular camera includes a first camera and a second camera. The method includes:

[0007] Detecting the waves in the ship operation area by the lidar to obtain lidar point cloud wave data, photographing the waves by the first camera to obtain a first wave image, and photographing the waves by the second camera to obtain a second wave image;

[0008] Subtracting the position data of the corresponding pixel points in the first wave image and the second wave image to obtain a wave disparity map;

[0009] Calculating the depth information of each pixel point in the wave disparity map according to the internal parameters of the binocular camera, and determining the point cloud data corresponding to the wave disparity map based on the depth information of each pixel point to obtain visual point cloud wave data;

[0010] Fusing the visual point cloud wave data and the lidar point cloud wave data to obtain fused point cloud wave data;

[0011] Performing three-dimensional modeling on the waves based on the fused point cloud wave data to obtain a wave model.

[0012] In the technical solution of the embodiment of the present invention, first, a lidar is used to detect the waves in the ship operation area to obtain lidar point cloud wave data, a first camera is used to photograph the waves to obtain a first wave image, and a second camera is used to photograph the waves to obtain a second wave image. The dynamic wave changes in the ship operation area can be accurately captured, and the interference caused by turbulence and complex environments can be effectively reduced, providing a reliable data basis for constructing an accurate wave model subsequently. Then, the position data of the corresponding pixel points in the first wave image and the second wave image are subtracted to obtain a wave disparity map, laying a data foundation for obtaining the depth information of each pixel point. After that, the depth information of each pixel point in the wave disparity map is calculated according to the internal parameters of the binocular camera, and the point cloud data corresponding to the wave disparity map is determined based on the depth information of each pixel point to obtain visual point cloud wave data, which can accurately restore the fine features of the waves (such as foam and ripples) and provide a data basis for subsequent fusion with the lidar point cloud wave data. The visual point cloud wave data and the lidar point cloud wave data are fused to obtain fused point cloud wave data, integrating the high-density texture details and dynamic features of the visual point cloud wave data with the high-precision three-dimensional coordinates and spatial positioning of the lidar point cloud wave data, effectively making up for the limitations of single data and obtaining more comprehensive wave information. This not only provides a solid data basis for constructing a wave model subsequently but also improves the accuracy of the wave model. Based on the fused point cloud wave data, a three-dimensional model of the waves is constructed to obtain a wave model, making the wave model more accurate in morphological restoration and capable of truly reflecting the dynamic features of the waves such as undulation, breaking, and droplets, providing a scientific basis for the safety assessment of offshore operations. In addition, the fused data effectively reduces the influence of noise and outliers, improves the stability and robustness of the overall data, and further ensures the reliability of the wave model under complex sea conditions. Therefore, the technical solution of the present invention solves the problem that the accuracy of the existing measurement method is difficult to meet the actual requirements.

[0013] In a second aspect, the embodiment of the present invention further provides a marine wave measurement device based on binocular vision and lidar, which is applied to a ship. A binocular camera and a lidar are installed on the ship. The binocular camera includes a first camera and a second camera. The device includes:

[0014] An acquisition module, configured to detect the waves in the ship operation area by using the lidar to obtain lidar point cloud wave data, photograph the waves by using the first camera to obtain a first wave image, and photograph the waves by using the second camera to obtain a second wave image;

[0015] A first calculation module, configured to subtract the position data of the corresponding pixel points in the first wave image and the second wave image to obtain a wave disparity map;

[0016] A second calculation module, configured to calculate the depth information of each pixel point in the wave disparity map according to the internal parameters of the binocular camera, determine the point cloud data corresponding to the wave disparity map based on the depth information of each pixel point, and obtain visual point cloud wave data;

[0017] A fusion module, configured to fuse the visual point cloud wave data and the lidar point cloud wave data to obtain fused point cloud wave data;

[0018] A modeling module, configured to perform three-dimensional modeling on the wave based on the fused point cloud wave data to obtain a wave model.

[0019] In a third aspect, an embodiment of the present invention further provides a ship, which includes:

[0020] At least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute any one of the methods for measuring sea waves based on binocular vision and lidar in the first aspect.

[0021] In a fourth aspect, an embodiment of the present invention further provides a storage medium containing computer-executable instructions, and the computer-executable instructions implement any one of the methods for measuring sea waves based on binocular vision and lidar in the first aspect when executed by a computer processor.

[0022] It should be noted that the above computer instructions can be stored in whole or in part on a computer-readable storage medium. Among them, the computer-readable storage medium can be packaged together with the processor of the device for measuring sea waves based on binocular vision and lidar, or can be packaged separately from the processor of the device for measuring sea waves based on binocular vision and lidar. This application does not make any limitation in this regard.

[0023] The descriptions of the second aspect, the third aspect, and the fourth aspect in this application can refer to the detailed description of the first aspect; and, for the beneficial effects of the descriptions of the second aspect, the third aspect, and the fourth aspect, reference can be made to the analysis of the beneficial effects of the first aspect, which will not be elaborated here.

[0024] In this application, the names of the above-mentioned device for measuring sea waves based on binocular vision and lidar do not constitute a limitation to the device or functional module itself. In actual implementation, these devices or functional modules may appear under other names. As long as the functions of each device or functional module are similar to those of this application and fall within the scope of the claims of this application and their equivalent technologies.

[0025] These aspects or other aspects of this application will be more clearly understood in the following description. Description of the Drawings

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0027] Figure 1a It is a flowchart of a method for measuring sea waves based on binocular vision and lidar provided by an embodiment of the present invention;

[0028] Figure 1b It is an example diagram of the viewing angle of a binocular camera provided by an embodiment of the present invention;

[0029] Figure 1c It is a schematic diagram of an indoor experiment of a method for measuring sea waves based on binocular vision and lidar provided by an embodiment of the present invention;

[0030] Figure 2 It is a flowchart of another method for measuring sea waves based on binocular vision and lidar provided by an embodiment of the present invention;

[0031] Figure 3 It is a schematic structural diagram of a device for measuring sea waves based on binocular vision and lidar provided by an embodiment of the present invention;

[0032] Figure 4 It is a schematic structural diagram of a ship provided by an embodiment of the present invention. Detailed Embodiments

[0033] The following will further elaborate on the present invention in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. Additionally, it should be noted that for the sake of description, only the parts related to the present invention rather than all the structures are shown in the drawings.

[0034] The term "and / or" in this document is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone.

[0035] The terms "first" and "second" in the specification and drawings of this application are used to distinguish different objects or different processes for the same object, rather than to describe the specific order of the objects.

[0036] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally further include other steps or units not listed, or may optionally further include other steps or units inherent to these processes, methods, products, or devices.

[0037] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts depict the operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operations are completed, but it can also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc. In addition, in the case of no conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0038] It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0039] In the description of this application, unless otherwise specified, the meaning of "a plurality of" refers to two or more.

[0040] Figure 1a The flowchart of a method for measuring sea waves based on binocular vision and lidar provided for the embodiments of the present invention is applicable to the situation where it is necessary to measure the waves in the ship operation area. This method can be applied to a ship equipped with a binocular camera and a lidar. The binocular camera includes a first camera and a second camera. This method can be executed by a device for measuring sea waves based on binocular vision and lidar, and the device for measuring sea waves based on binocular vision and lidar can be implemented in a hardware / software manner. Exemplarily, the device can be integrated into the ship. Refer to Figure 1a , the method for measuring sea waves based on binocular vision and lidar in this embodiment specifically includes the following steps:

[0041] Step 110: Detect the sea waves in the ship operation area through the lidar to obtain lidar point cloud wave data, take a first wave image through the first camera, and take a second wave image through the second camera.

[0042] Specifically, lidar refers to a device that measures information such as the distance, speed, and angle of a target object by emitting laser beams and receiving reflected light. For example, the lidar can be a 32-line lidar. The ship operation area refers to the sea area or water area where the ship conducts specific operations (such as loading and unloading goods, offshore construction, etc.). The laser point cloud wave data refers to a dataset composed of a large number of three-dimensional space points obtained by detecting waves with lidar. Each point represents a position on the wave surface and contains the three-dimensional coordinate information of the point. A binocular camera refers to a vision system designed based on the principle of bionics, simulating the imaging mechanism of human binoculars. It consists of two cameras with a certain distance (i.e., the baseline distance) in space. For example, the optional range of the baseline distance of the binocular camera is from 1.15 meters to 1.25 meters, the optional range of the pitch angle of the binocular camera is from -10 degrees to +5 degrees, and the relationship between the field of view coverage area and the viewing angle of the binocular camera is where S is the field of view coverage area, r is the distance from the camera to the plane where the observation target is located, and θ is the viewing angle of the binocular camera, that is, the angle that the camera can perceive. For example, as Figure 1b shown, it can be found that the black solid figures are two cameras installed on the bow deck of the ship, namely the "first camera" and the "second camera", which together form a binocular camera. θ is the viewing angle of the binocular camera, the dotted line is the wave, and point P is a point on the wave measured by the binocular camera. In this embodiment, the binocular camera includes a first camera and a second camera. The first wave image refers to the two-dimensional wave image captured by the first camera. The second wave image refers to the two-dimensional wave image captured by the second camera.

[0043] In specific implementation, the waves in the ship operation area can be detected by the lidar installed on the bow deck of the ship to obtain the laser point cloud wave data. At the same time, the first wave image and the second wave image are obtained by using the binocular camera also installed on the bow deck of the ship. Specifically, the first wave image is captured by the first camera, and the second wave image is captured by the second camera.

[0044] In practical applications, to ensure that the lidar and the binocular camera can effectively detect the same wave area, the field of view overlap area of the lidar and the binocular camera installed on the bow deck of the ship needs to be greater than or equal to 85%. At the same time, the binocular camera and the lidar can be jointly calibrated based on a calibration board to determine the external parameter matrix between the two, so that the data they collect can be unified into the same spatial coordinate system, and the spatial registration error during calibration ≤ 2 cm.

[0045] In addition, to ensure the time synchronization of the data obtained by the lidar and the binocular camera, the synchronization error can be calculated using the synchronization error compensation formula first, and then it is determined whether it is within the preset error range. If it is within the preset error range, the data obtained by the two can be directly used; if it exceeds the preset error range, any of the following methods can be used to achieve time synchronization: 1. Achieve microsecond-level synchronization of the timestamps of the two through the Precision Time Protocol (PTP). 2. Use a Field-Programmable Gate Array (FPGA) hardware synchronizer to trigger the synchronized exposure of the lidar and the binocular camera. When acquiring data, the FPGA hardware synchronizer can accurately control the exposure moments of the lidar and the binocular camera. For example, the synchronization error compensation formula can be: △t = ||t1 - t2|| / v, where t1 is the trigger timestamp of the lidar, t2 is the trigger timestamp of the binocular camera, and v is the synchronization signal transmission rate. For example, the synchronization error compensation formula can also be: △t = t2 - t1.

[0046] In this embodiment, through the above steps, the dynamic wave changes in the ship operation area can be accurately captured, and the interference caused by turbulence and complex environments can be effectively reduced, providing a reliable data basis for constructing an accurate wave model subsequently.

[0047] Step 120: Subtract the position data of the corresponding pixel points in the first wave image and the second wave image to obtain a wave disparity map.

[0048] Specifically, the wave disparity map refers to an image obtained by calculating the disparity between corresponding pixel points (i.e., the position difference of the same object point in the two images) based on two wave images captured by a binocular camera. A pixel point is the basic unit of an image, representing a point in the image.

[0049] In specific implementation, after obtaining the first wave image and the second wave image, first, preprocess the first wave image and the second wave image (such as noise reduction, image enhancement, filtering, etc.) to obtain the preprocessed first wave image and the second wave image, and then perform epipolar correction on the preprocessed first wave image and the second wave image to eliminate the pixel position deviation caused by lens distortion, and at the same time constrain the projection points of the same object in the two images to be on the same horizontal epipolar line (i.e., achieve epipolar alignment). Then, a matching algorithm (such as block matching based on grayscale, feature-based matching, or deep learning matching, etc.) can be used to find the corresponding pixel points in the two images.

[0050] Finally, subtract the position data of the corresponding pixel points in the first wave image and the second wave image to obtain a wave disparity map. Specifically, for each pixel point in the first wave image, calculate the horizontal position information of this pixel point and the horizontal position information of the corresponding pixel point (i.e., the homologous point) of this pixel point in the second wave image to obtain the disparity value of this pixel point. Then, arrange the disparity values of each pixel point according to the position information of each pixel point (including horizontal position information and vertical position information) to obtain a wave disparity map.

[0051] In practical applications, when calculating the disparity, the pyramid hierarchical strategy can be used to accelerate the calculation process.

[0052] In this embodiment, through the above steps, a data basis is provided for obtaining the depth information of each pixel point later.

[0053] Step 130: Calculate the depth information of each pixel point in the wave disparity map according to the internal parameters of the binocular camera, and determine the point cloud data corresponding to the wave disparity map based on the depth information of each pixel point to obtain visual point cloud wave data.

[0054] Specifically, the internal parameters of the binocular camera refer to the internal parameters of the camera, including the horizontal focal length, vertical focal length, baseline distance, main point position information, distortion coefficient, etc. of the binocular camera. The depth information of a pixel point refers to the depth value of the surface point of the object in the three-dimensional space corresponding to the pixel point. The visual point cloud wave data refers to the point cloud data calculated based on the wave images captured by the binocular camera.

[0055] In the specific implementation, the product of the horizontal focal length and the baseline distance of the binocular camera can be calculated first to obtain a conversion coefficient, and then the ratio of the conversion coefficient to the disparity value of each pixel point in the wave disparity map can be calculated to obtain the depth information of each pixel point. Then, according to the depth information of each pixel point, the main point position information and the horizontal focal length of the binocular camera, calculate the horizontal information of each pixel point; according to the depth information of each pixel point, the main point position information and the vertical focal length of the binocular camera, calculate the vertical information of each pixel point; finally, determine the point cloud data corresponding to the wave disparity map based on the depth information, horizontal information and vertical information of each pixel point to obtain visual point cloud wave data.

[0056] In this embodiment, through the above steps, the fine features of the wave (such as foam and ripples) can be accurately restored and a data basis is provided for subsequent fusion with the laser point cloud wave data.

[0057] Step 140: Fuse the visual point cloud wave data and the laser point cloud wave data to obtain fused point cloud wave data.

[0058] Specifically, the fused point cloud wave data refers to the point cloud data obtained by fusing the laser point cloud wave data and the visual point cloud wave data.

[0059] In a specific implementation, for each point in the laser point cloud wave data, first, calculate the distance (such as the Euclidean distance) between this point and each point in the visual point cloud wave data. Then, to improve the correspondence accuracy, determine the point in the visual point cloud wave data that is the closest to this point and less than the preset distance threshold as the associated point, and form an associated point pair with these two points. For each associated point pair, the weighted average method can be used to fuse their attributes (such as coordinates, colors, etc.). For example, for the coordinate attribute, different weights can be assigned according to the source of the point (such as the weight of the point in the laser point cloud wave data is set to 0.6, and the weight of the point in the visual point cloud wave data is set to 0.4), and then, the fused coordinates are obtained through weighted calculation.

[0060] It should be noted that for the individual points that do not form associated point pairs, they can be omitted to make the number of point clouds in the laser point cloud wave data equal to that in the visual point cloud wave data.

[0061] In this embodiment, through the above steps, the high-density texture details and dynamic features of the visual point cloud wave data are fused with the high-precision three-dimensional coordinates and spatial positioning of the laser point cloud wave data, effectively making up for the limitations of single data and obtaining more comprehensive wave information, which not only provides a solid data foundation for subsequent construction of the wave model but also improves the accuracy of the wave model.

[0062] Step 150: Perform three-dimensional modeling on the wave based on the fused point cloud wave data to obtain a wave model.

[0063] Specifically, the wave model refers to a three-dimensional wave surface model created based on the fused point cloud wave data.

[0064] In a specific implementation, after obtaining the fused point cloud wave data, it can be first denoised and resampled to avoid grid distortion caused by discrete point errors. Then, based on point cloud meshing methods (such as Poisson reconstruction, moving least squares method, etc.), the discrete fused point cloud wave data is converted into a triangular mesh model. Subsequently, use a mesh optimization algorithm (such as Laplacian smoothing algorithm, Laplace - Beltrami operator, anisotropic smoothing, etc.) to optimize the generated triangular mesh model, adjust the shape and size of the triangles, and finally, map texture features such as colors and details to the optimized triangular mesh model by calculating the correspondence between each triangular face and the texture image to obtain the wave model.

[0065] After obtaining the wave model, the wave height, wave trough, and wave peak of the current wave can be obtained by analyzing the height information of each point in the wave model. Then, the results of the wave model within a preset period are obtained, and based on this data, the wave direction angle, as well as the periods of the wave trough and wave peak within the preset period, are calculated. Specifically, the wave direction angle can be calculated using the continuous frame centroid displacement method, and the periods of the wave trough and wave peak can be calculated using the direct counting method or the Fourier transform method. In addition, fast Fourier transform analysis can be performed on the wave height time series data within the preset period to convert the time-domain wave height data to the frequency domain, and the main frequency period can be extracted from it to assist researchers in analyzing wave characteristics.

[0066] In practical applications, in order to make the wave model more adaptable to hardware with limited resources (such as on-board devices), reduce the occupancy of computing resources, and improve the operation efficiency, after obtaining the wave model, model compression methods (such as replacing standard convolution with depthwise separable convolution, mixed-precision training, pruning techniques, 8-bit integer quantization, etc.) can be used to compress the wave model without significantly reducing the model performance, so that the number of parameters of the compressed model is less than the preset number of parameters (such as 5 megabytes), the inference latency is less than the preset latency (such as 50 milliseconds), and the peak power consumption does not exceed the preset power consumption (such as 15 watts). Then, according to the preset deployment strategy, the compressed model is deployed to the on-board device hardware. Among them, the preset deployment strategy refers to the deployment rules determined in advance according to the actual situation or requirements. For example: the preset deployment strategy can be an adaptive power consumption control strategy, and the relationship between the Graphics Processing Unit (GPU) frequency and the real-time data volume is:

[0067]

[0068] where Q is the real-time data volume, f min and f max are the minimum and maximum operating frequencies of the GPU, Q min and Q max are the minimum and maximum data thresholds that the device can withstand.

[0069] In this embodiment, the wave model obtained by three-dimensional modeling based on the fused point cloud wave data is more accurate in shape restoration and can truly reflect the dynamic characteristics of waves such as undulation, breaking, and spray, providing a scientific basis for the safety assessment of offshore operations. In addition, the fused data effectively reduces the influence of noise and outliers, improves the overall data stability and robustness, and further ensures the reliability of the wave model under complex sea conditions.

[0070] In addition, the method for measuring sea waves based on binocular vision and lidar provided in this embodiment can also verify its feasibility indoors, such as Figure 1cAs shown in the figure, a schematic diagram of an indoor experiment of this method is provided. The figure shows a long water tank used to simulate a wave environment. A wave-making device is installed at one end of the water tank, which can generate regular waves with different parameters (such as wave height, wavelength, frequency, etc.), push the water in the water tank to form waves, and construct an approximate wave scene on the sea. The wave direction is from the end where the wave-making device is installed to the other end of the water tank. Above the water tank, a combined device of a binocular camera and a lidar is installed to collect wave image data and accurately measure wave information. A wave height gauge is arranged in the water tank, which is connected to the binocular camera and the lidar to accurately measure the height of the waves. The measured data can be verified and supplemented with the data collected by the binocular camera and the lidar. The whole set of devices closely cooperate through various components to simulate waves and collect multi-dimensional data, providing reliable data support and experimental basis for the feasibility verification of this method, algorithm optimization, and related ocean research.

[0071] The method for measuring sea waves based on binocular vision and lidar provided by the embodiment of the present invention first detects the waves in the ship operation area through the lidar to obtain lidar point cloud wave data, takes pictures of the waves through the first camera to obtain the first wave image, and takes pictures of the waves through the second camera to obtain the second wave image, which can accurately capture the dynamic wave changes in the ship operation area, effectively reduce the interference caused by turbulence and complex environments, and provide a reliable data basis for subsequent construction of an accurate wave model. Then, the position data of the corresponding pixel points in the first wave image and the second wave image are subtracted to obtain a wave disparity map, laying a data foundation for obtaining the depth information of each pixel point. Then, according to the internal parameters of the binocular camera, the depth information of each pixel point in the wave disparity map is calculated, and the point cloud data corresponding to the wave disparity map is determined based on the depth information of each pixel point to obtain visual point cloud wave data, which can accurately restore the fine features of the waves (such as foam and ripples) and provide a data foundation for subsequent fusion with the lidar point cloud wave data. The visual point cloud wave data and the lidar point cloud wave data are fused to obtain fused point cloud wave data, which combines the high-density texture details and dynamic features of the visual point cloud wave data with the high-precision three-dimensional coordinates and spatial positioning of the lidar point cloud wave data, effectively making up for the limitations of single data, obtaining more comprehensive wave information, not only providing a solid data foundation for subsequent construction of a wave model, but also improving the accuracy of the wave model. Based on the fused point cloud wave data, a three-dimensional model of the waves is constructed to obtain a wave model, making the wave model more accurate in morphological restoration and capable of truly reflecting the dynamic features of the waves such as undulation, breaking, and spray, providing a scientific basis for the safety assessment of maritime operations. In addition, the fused data effectively reduces the influence of noise and outliers, improves the stability and robustness of the overall data, and further ensures the reliability of the wave model under complex sea conditions. Therefore, the technical solution of the present invention solves the problem that the accuracy of the existing measurement method is difficult to meet the actual requirements.

[0072] Figure 2 The figure is a flowchart of another method for measuring ocean waves based on binocular vision and lidar provided by an embodiment of the present invention. This embodiment is a specific implementation based on the above embodiment. In this embodiment, the method may further include:

[0073] Step 210: Detect ocean waves in the ship operation area through lidar to obtain lidar point cloud wave data, capture the waves through a first camera to obtain a first wave image, and capture the waves through a second camera to obtain a second wave image.

[0074] Step 211: Preprocess the first wave image to obtain a first processed wave image, and preprocess the second wave image to obtain a second processed wave image.

[0075] Specifically, the first processed wave image refers to the first wave image after preprocessing. The second processed wave image refers to the second wave image after preprocessing.

[0076] In a specific implementation, after obtaining the first wave image, the first processed wave image can be first denoised based on a denoising algorithm to obtain a denoised image, then filtered based on a filtering algorithm to obtain a filtered image; finally, the filtered image is enhanced based on an image enhancement method to obtain the first processed wave image.

[0077] Similarly, the second wave image can be processed using the same processing method as the first wave image to obtain a second processed wave image.

[0078] In this embodiment, through the above steps, the quality of the images collected by the binocular camera is improved, thereby enhancing the accuracy of subsequent calculations; at the same time, redundant information is reduced, improving the calculation efficiency.

[0079] Further, preprocessing the first wave image to obtain a first processed wave image includes: performing image enhancement processing on the first wave image to obtain a first enhanced wave image; performing filtering processing on the first enhanced wave image to obtain a first filtered wave image; performing denoising processing on the first filtered wave image to obtain a first processed wave image; correspondingly, preprocessing the second wave image to obtain a second processed wave image includes: performing image enhancement processing on the second wave image to obtain a second enhanced wave image; performing filtering processing on the second enhanced wave image to obtain a second filtered wave image; performing denoising processing on the second filtered wave image to obtain a second processed wave image.

[0080] Specifically, the first enhanced wave diagram refers to the first wave diagram after image enhancement processing. The first filtered wave diagram refers to the first wave diagram after image enhancement and filtering processing. The second enhanced wave diagram refers to the second wave diagram after image enhancement processing. The second filtered wave diagram refers to the second wave diagram after image enhancement and filtering processing.

[0081] In specific implementation, after obtaining the first wave diagram, image enhancement processing can be first performed on it based on image enhancement methods (such as MFPD-Net dehazing algorithm, histogram equalization, improved dark channel prior model, etc.) to obtain the first enhanced wave diagram; then, filtering processing is performed on the first enhanced wave diagram based on filtering algorithms (such as bilateral filtering, Gaussian filtering, median filtering, etc.) to obtain the first filtered wave diagram; finally, noise reduction processing is performed on the first filtered wave diagram based on noise reduction algorithms (such as wavelet threshold noise reduction, Fourier transform noise reduction, non-local means noise reduction, etc.) to obtain the first processed wave diagram.

[0082] Similarly, the same processing method as that for the first wave diagram can be used to process the second wave diagram to obtain the second processed wave diagram.

[0083] Exemplarily, if the MFPD-Net dehazing algorithm is used to perform image enhancement processing on the first wave diagram, the first enhanced wave diagram can be expressed by the following formula:

[0084]

[0085] Among them, J(x) is the first processed wave diagram, I(x) is the first wave diagram, A is the global atmospheric light intensity, b(x) is the transmittance, and b0 is the lower threshold of the transmittance.

[0086] Exemplarily, if the improved dark channel prior model is used to perform image enhancement processing on the first wave diagram, the first enhanced wave diagram can be expressed by the following formula:

[0087]

[0088] Among them, a is the weight coefficient, min c is the operation of taking the minimum value of the color channels (such as RGB channels), I c (y) is the intensity value of the image color channel c (such as R, G, B channels) at the pixel point y, and A c is the intensity value of the global atmospheric light on the color channel c.

[0089] Exemplarily, if wavelet threshold noise reduction is used to perform noise reduction processing on the first filtered wave diagram, the wavelet coefficient threshold function is:

[0090]

[0091] Among them, is the wavelet coefficient threshold, w is the original wavelet coefficient, and U is the fixed threshold.

[0092] In this embodiment, first, the image is enhanced to improve characteristics such as the contrast, brightness, and color of the image. After such processing, dynamic features such as the undulation and breaking of the waves will become more prominent. At this time, the distinguishability between noise and signal is relatively improved. Then, the enhanced image is filtered, which can more effectively remove noise, interference, and unwanted frequency components, making the image smoother. After image enhancement and filtering, most of the noise in the image has been weakened or preliminarily removed. On this basis, noise reduction processing is performed, which can further deeply remove the remaining noise and greatly improve the image quality. Moreover, since the signal characteristics are highlighted in the previous processing, the signal and noise can be more accurately distinguished during the noise reduction process, reducing misjudgments caused by noise interference, thereby better retaining the details and textures of the image, further improving the accuracy of the first processed wave diagram and the second processed wave diagram, and ultimately enhancing the accuracy of subsequent calculations.

[0093] Step 212: Construct an intermediate correction matrix based on the binocular translation vector.

[0094] Specifically, the binocular translation vector refers to the translation vector between the optical centers of two cameras in a binocular camera, which reflects the relative position relationship between the two cameras in space. The intermediate correction matrix refers to a matrix constructed based on the binocular translation vector, which is used to perform epipolar correction on the images captured by the binocular camera.

[0095] In specific implementation, the binocular camera can be calibrated by a calibration board first to obtain the binocular translation vector. Then, the binocular translation vector is unitized, and then the horizontal auxiliary vector is calculated based on the unitized binocular translation vector and the reference vector; the vertical auxiliary vector is calculated based on the unitized binocular translation vector and the horizontal auxiliary vector, and finally, the intermediate correction matrix is constructed based on the unitized binocular translation vector, the horizontal auxiliary vector, and the vertical auxiliary vector.

[0096] In this embodiment, the construction of the intermediate correction matrix is realized.

[0097] Further, step 212 can specifically include: unitizing the binocular translation vector to obtain the first unit vector; unitizing the cross product result of the first unit vector and the reference vector to obtain the second unit vector; calculating the cross product of the first unit vector and the second unit vector to obtain the third unit vector; and constructing the intermediate correction matrix based on the first unit vector, the second unit vector, and the third unit vector.

[0098] Specifically, the reference vector refers to a reference vector preset according to the actual situation or requirements, which is used for cross product operation with the binocular translation vector.

[0099] Exemplarily, if the binocular translation vector is and the reference vector is [0, 0, -1] T , let the first unit vector be the second unit vector be the third unit vector be and the intermediate correction matrix be M, then the first unit vector the second unit vector the third unit vector and the intermediate correction matrix

[0100] In this embodiment, through the above steps, not only the accuracy of the intermediate correction matrix is improved, but also its construction complexity is reduced, and the calculation efficiency is improved.

[0101] Step 213: Perform epipolar correction on the first processed wave diagram according to the intermediate correction matrix to obtain an updated first wave diagram, and perform epipolar correction on the second processed wave diagram according to the intermediate correction matrix to obtain an updated second wave diagram.

[0102] Specifically, the updated first wave diagram refers to the first wave diagram after epipolar correction. The updated second wave diagram refers to the second wave diagram after epipolar correction.

[0103] In specific implementation, the binocular camera can be calibrated using a calibration board to obtain the internal parameter matrix of the binocular camera. Subsequently, calculate the product of the internal parameter matrix of the binocular camera and the intermediate correction matrix to obtain the camera correction projection matrix, and then multiply the camera correction projection matrix by the inverse matrix of the internal parameter matrix of the binocular camera to obtain the correction matrix; finally, use this correction matrix to perform epipolar correction on the first processed wave diagram, that is, calculate the product of the correction matrix and the pixel matrix corresponding to the first processed wave diagram to obtain the updated first wave diagram.

[0104] Similarly, using the same processing method, perform epipolar correction on the second processed wave diagram according to the intermediate correction matrix to obtain an updated second wave diagram.

[0105] In this embodiment, through the above steps, the corresponding points in the updated first wave diagram and the second wave diagram are constrained on the same horizontal epipolar line, that is, the calculation amount of determining the corresponding pixel points in the two diagrams is reduced, the calculation time is saved, the wave measurement efficiency is improved, and at the same time, the matching accuracy of the corresponding pixel points is improved through epipolar alignment, thereby improving the accuracy of the subsequent wave disparity map, and finally ensuring the accuracy of the subsequent fusion and modeling.

[0106] Further, perform epipolar correction on the first processed wave image according to the intermediate correction matrix to obtain an updated first wave image, including: determining the square root of the binocular extrinsic parameter matrix as the first initial semi-rotation matrix; calculating the product of the intermediate correction matrix and the first initial semi-rotation matrix to obtain the first target correction matrix; calculating the product of the first target rotation matrix and the first processed wave image to obtain the updated first wave image; correspondingly, perform epipolar correction on the second processed wave image according to the intermediate correction matrix to obtain an updated second wave image, including: determining the square root of the inverse matrix of the binocular extrinsic parameter matrix as the second initial semi-rotation matrix; calculating the product of the intermediate correction matrix and the second initial semi-rotation matrix to obtain the second target correction matrix; calculating the product of the second target rotation matrix and the second processed wave image to obtain the updated second wave image.

[0107] Specifically, the binocular extrinsic parameter matrix refers to a matrix that describes the pose relationship of the second camera relative to the first camera. The first target correction matrix refers to the complete correction matrix for performing epipolar correction on the first wave image. The second target correction matrix refers to the complete correction matrix for performing epipolar correction on the second wave image.

[0108] In specific implementation, when performing epipolar correction on the first processed wave image, the square root of the binocular extrinsic parameter matrix can be first determined as the first initial semi-rotation matrix, and then, calculate the product of the intermediate correction matrix and the first initial semi-rotation matrix to obtain the first target correction matrix. For example: if the binocular extrinsic parameter matrix is R and the intermediate correction matrix is M, then the first initial semi-rotation matrix is R 1 / 2 , and the first target correction matrix is M×R 1 / 2 . Finally, calculate the product of the correction matrix and the pixel matrix corresponding to the first processed wave image to obtain the updated first wave image.

[0109] When performing epipolar correction on the second processed wave image, the square root of the inverse matrix of the binocular extrinsic parameter matrix can be first determined as the second initial semi-rotation matrix, and then calculate the product of the intermediate correction matrix and the second initial semi-rotation matrix to obtain the second target correction matrix. For example: if the binocular extrinsic parameter matrix is R and the intermediate correction matrix is M, then the first initial semi-rotation matrix is R -1 / 2 , and the first target correction matrix is M×R -1 / 2 . Finally, calculate the product of the second target correction matrix and the pixel matrix corresponding to the second processed wave image to obtain the updated second wave image.

[0110] In this embodiment, through the above steps, the efficiency and accuracy of epipolar correction for images are improved.

[0111] Step 214: Subtract the position data of the corresponding pixel points in the first wave image and the second wave image to obtain a wave disparity map.

[0112] Further, before step 214, it also includes: for the current pixel point in the first wave diagram, determining a matching window centered on the current pixel point; calculating the matching cost between the matching window and each pixel point in the same row as the current pixel point in the second wave diagram; and determining the pixel point with the minimum matching cost as the corresponding pixel point of the current pixel point in the second wave diagram.

[0113] Specifically, the matching window refers to a rectangular area centered on the current pixel point in the first wave diagram. The matching cost refers to a numerical index for measuring the corresponding relationship between pixel points in the first wave diagram and the second wave diagram, and the smaller its value, the higher the matching degree.

[0114] In specific implementation, for the current pixel point in the first wave diagram, the window size of the matching window can be first determined based on a matching algorithm (such as semi-global block matching algorithm, dynamic shape window matching algorithm, multi-scale window method, etc.). For example: if the semi-global block matching algorithm is adopted, the window size calculation formula is as follows:

[0115] W = (3 + 2·sigmoid(ρ - 0.35)) × (3 + 2·sigmoid(ρ - 0.35))

[0116] where W is the window size, ρ is the local texture complexity of the image, and sigmoid is the activation function.

[0117] Then, a matching window is delimited centered on the current pixel point according to this size. Subsequently, the matching cost between the matching window and each pixel point in the same row as the current pixel point in the second wave diagram is calculated. Specifically, for each peer pixel point in the same row as the current pixel point in the second wave diagram, the matching cost between each pixel point in the matching window and this peer pixel point can be calculated based on a matching cost algorithm (such as absolute difference cost algorithm, squared difference cost algorithm, normalized cross-correlation cost algorithm, semi-global matching cost algorithm, etc.), and the minimum matching cost is determined as the matching cost between the matching window and this peer pixel point.

[0118] Finally, the peer pixel point with the minimum matching cost is determined as the corresponding pixel point of the current pixel point in the second wave diagram.

[0119] In this embodiment, through the above steps, not only the determination of the corresponding pixel points in the first wave diagram and the second wave diagram is realized, providing a data basis for obtaining the wave disparity map subsequently, but also the accuracy of the determination of the corresponding pixel points is improved, thereby enhancing the accuracy of the wave disparity map.

[0120] Step 215: Calculate the depth information of each pixel point in the wave disparity map according to the internal parameters of the binocular camera, and determine the point cloud data corresponding to the wave disparity map based on the depth information of each pixel point to obtain the visual point cloud wave data.

[0121] Optionally, the internal parameters of the binocular camera include the horizontal focal length and the baseline distance of the binocular camera.

[0122] Further, calculate the depth information of each pixel point in the wave disparity map according to the internal parameters of the binocular camera, including: calculating the product of the horizontal focal length and the baseline distance to obtain a conversion coefficient; calculating the ratio of the conversion coefficient to the disparity value of each pixel point in the wave disparity map to obtain the depth information of each pixel point.

[0123] Specifically, the baseline distance refers to the distance between the optical centers of the first camera and the second camera.

[0124] Exemplarily, for the current pixel point in the wave disparity map, if the horizontal focal length is fx, the baseline distance is L, and the disparity value of the current pixel point is d, then the conversion coefficient is fx×B, and the depth information of the current pixel point is fx×B / d.

[0125] In this embodiment, through the above steps, the accuracy of the depth information of each pixel point obtained is improved.

[0126] Optionally, the internal parameters of the binocular camera further include the vertical focal length and the principal point position information of the binocular camera.

[0127] Further, determine the point cloud data corresponding to the wave disparity map based on the depth information of each pixel point to obtain the visual point cloud wave data, including: calculating the horizontal information of each pixel point according to the principal point position information, the horizontal focal length, and the disparity value of each pixel point; calculating the vertical information of each pixel point according to the principal point position information, the vertical focal length, and the disparity value of each pixel point; determining the point cloud data corresponding to the wave disparity map based on the depth information, horizontal information, and vertical information of each pixel point to obtain the visual point cloud wave data.

[0128] Specifically, the principal point position information includes the principal point horizontal position information and the principal point vertical position information.

[0129] In specific implementation, for the current pixel point in the wave disparity map, the difference between the horizontal position information of the current pixel point and the principal point horizontal position information can be calculated first to obtain the horizontal disparity offset, then the product of the horizontal disparity offset and the depth information of the current pixel point can be calculated to obtain the horizontal scaled disparity, and finally the ratio of the horizontal scaled disparity to the horizontal focal length can be calculated to obtain the horizontal information of the current pixel point.

[0130] Similarly, calculate the difference between the vertical position information of the current pixel point and the principal point vertical position information to obtain the vertical disparity offset, then calculate the product of the vertical disparity offset and the depth information of the current pixel point to obtain the vertical scaled disparity, and finally calculate the ratio of the vertical scaled disparity to the vertical focal length to obtain the vertical information of the current pixel point.

[0131] Finally, based on the depth information, horizontal information, and vertical information of each pixel point, each pixel point is mapped into a three-dimensional space to obtain visual point cloud wave data.

[0132] In this embodiment, through the above steps, the accuracy of the obtained visual point cloud wave data is improved.

[0133] Step 216: Fuse the visual point cloud wave data and the lidar point cloud wave data to obtain fused point cloud wave data.

[0134] Furthermore, before step 216, it also includes: based on the position data of the corresponding points in the visual point cloud wave data and the lidar point cloud wave data, the camera-lidar rotation matrix, and the camera-lidar translation vector, calculate the error between the visual point cloud wave data and the lidar point cloud wave data to obtain the point cloud fusion error; when the point cloud fusion error is less than the preset error threshold, trigger the execution of fusing the visual point cloud wave data and the lidar point cloud wave data to obtain fused point cloud wave data; when the point cloud fusion error is not less than the preset error threshold, return to execute obtaining the lidar point cloud wave data by detecting the waves in the ship operation area with a lidar, obtaining the first wave image by shooting the waves with the first camera, and obtaining the second wave image by shooting the waves with the second camera.

[0135] Specifically, the camera-lidar rotation matrix refers to the matrix that describes the rotation relationship between the camera and the lidar. The camera-lidar translation vector refers to the vector that describes the translation relationship between the camera and the lidar. The point cloud fusion error refers to the index that measures the degree of difference between the visual point cloud wave data and the lidar point cloud wave data. The preset error threshold refers to the threshold preset according to the actual situation or requirements for judging whether the point cloud fusion error is acceptable.

[0136] In specific implementation, after obtaining the visual point cloud wave data, for each point in the lidar point cloud wave data, the distance (such as Euclidean distance) between this point and each point in the visual point cloud wave data can be calculated first. Then, to improve the corresponding accuracy, the point in the visual point cloud wave data that is the closest to this point and less than the preset distance threshold is determined as the corresponding point, and these two points form an associated point pair. At the same time, for the individual points that do not form an associated point pair, they are omitted to make the number of points in the lidar point cloud wave data equal to that in the visual point cloud wave data.

[0137] After that, based on the position data of the corresponding points in the visual point cloud wave data and the lidar point cloud wave data, the camera-lidar rotation matrix, and the camera-lidar translation vector, calculate the error between the visual point cloud wave data and the lidar point cloud wave data to obtain the point cloud fusion error. The specific calculation formula is as follows:

[0138]

[0139] Among them, R1 is the camera-radar rotation matrix, g is the camera-radar translation vector, N is the total number of point clouds, and p i is the i-th point in the visual point cloud wave data, and q i is the i-th corresponding point in the lidar point cloud wave data, corresponding to p i correspondingly.

[0140] Then, determine whether the point cloud fusion error is less than the preset error threshold. If it is less, it means that the matching degree between the visual point cloud wave data and the lidar point cloud wave data is relatively high, and the reliability of the fused result of the two is good. At this time, trigger the execution to fuse the visual point cloud wave data and the lidar point cloud wave data to obtain the fused point cloud wave data; if it is not less, it means that the difference between the visual point cloud wave data and the lidar point cloud wave data is large, and the current data fusion effect is not good. At this time, return to execute to obtain the lidar point cloud wave data by detecting the waves in the ship operation area with the lidar, obtain the first wave image by shooting the waves with the first camera, and obtain the second wave image by shooting the waves with the second camera.

[0141] In this embodiment, through the above steps, on the one hand, it avoids the problems of data chaos and error accumulation that may be caused by blind fusion, and avoids invalid fusion calculations for data with poor matching degree, thereby saving computing resources and time costs; on the other hand, it ensures that the quality of the finally fused data meets the requirements, and further improves the accuracy of the subsequent fused point cloud wave data and enhances the precision of wave measurement.

[0142] Step 217: Calculate the product of the real-time hull rotation matrix and the inverse matrix of the initial sensor offset matrix to obtain the attitude correction matrix.

[0143] Specifically, the real-time hull rotation matrix refers to the matrix reflecting the current rotation state of the hull and is used for attitude correction. The initial sensor offset matrix refers to the matrix describing the offset state of the sensor (such as a camera, lidar) relative to the hull during initial installation. The attitude correction matrix refers to the matrix obtained by calculating the product of the real-time hull rotation matrix and the inverse matrix of the initial sensor offset matrix, and is used for attitude correction of the fused point cloud wave data to eliminate the influence of hull movement on the wave data.

[0144] In specific implementation, the real-time hull rotation matrix and the initial sensor offset matrix can be obtained first through a six-axis anti-shake gimbal bracket installed on the ship, and then the product of the real-time hull rotation matrix and the inverse matrix of the initial sensor offset matrix is calculated to obtain the attitude correction matrix.

[0145] In practical applications, the six-axis anti-shake gimbal bracket can be installed on the bow deck of the ship, with a response frequency ≥ 20 Hz and an attitude angle compensation error < 0.1 degree.

[0146] Exemplarily, if the real-time hull rotation matrix is Ri and the initial sensor offset matrix is Ro, then the attitude correction matrix Rc = Ri × Ro -1 .

[0147] In this embodiment, through the above steps, the real-time attitude correction matrix is obtained, which not only provides a data basis for obtaining the updated fused point cloud wave data subsequently, but also improves the accuracy of the updated fused point cloud wave data.

[0148] Step 218: Correct the fused point cloud wave data based on the attitude correction matrix to obtain the updated fused point cloud wave data.

[0149] Specifically, the updated fused point cloud wave data refers to the fused point cloud wave data after attitude correction, which more accurately reflects the actual shape of the wave.

[0150] In specific implementation, the updated fused point cloud wave data can be obtained according to the product of the attitude correction matrix and the fused point cloud wave data.

[0151] In this embodiment, through the above steps, the attitude correction of the fused point cloud wave data is performed to eliminate the influence of hull movement on the wave data and improve the accuracy of the fused point cloud wave data.

[0152] Step 219: Perform three-dimensional modeling on the wave based on the fused point cloud wave data to obtain a wave model.

[0153] In the embodiments of the present invention, first, lidar is used to detect waves in the ship operation area to obtain laser point cloud wave data, a first camera is used to capture waves to obtain a first wave image, and a second camera is used to capture waves to obtain a second wave image. The dynamic wave changes in the ship operation area can be accurately captured, effectively reducing the interference caused by turbulence and complex environments, and providing a reliable data basis for subsequent construction of an accurate wave model. Then, the first wave image is preprocessed to obtain a first processed wave image, and the second wave image is preprocessed to obtain a second processed wave image, improving the quality of the images collected by the binocular camera, thus enhancing the accuracy of subsequent calculations; at the same time, redundant information is reduced, improving the calculation efficiency. Then, an intermediate correction matrix is constructed based on the binocular translation vector. According to the intermediate correction matrix, the first processed wave image is rectified along the epipolar line to obtain an updated first wave image, and according to the intermediate correction matrix, the second processed wave image is rectified along the epipolar line to obtain an updated second wave image, so that the corresponding points in the updated first wave image and the second wave image are constrained on the same horizontal epipolar line, that is, the calculation amount for determining the corresponding pixel points in the two images is reduced, saving calculation time, improving the wave measurement efficiency, and at the same time improving the matching accuracy of the corresponding pixel points through epipolar alignment, thereby enhancing the accuracy of the subsequent wave disparity map, and finally ensuring the accuracy of subsequent fusion and modeling. Then, the position data of the corresponding pixel points in the first wave image and the second wave image are subtracted to obtain a wave disparity map, laying a data foundation for obtaining the depth information of each pixel point. Then, the depth information of each pixel point in the wave disparity map is calculated based on the internal parameters of the binocular camera, and the point cloud data corresponding to the wave disparity map is determined based on the depth information of each pixel point to obtain visual point cloud wave data, which can accurately restore the fine features of the waves (such as foam, ripples) and provide a data basis for subsequent fusion with the laser point cloud wave data. The visual point cloud wave data and the laser point cloud wave data are fused to obtain fused point cloud wave data, integrating the high-density texture details and dynamic features of the visual point cloud wave data with the high-precision three-dimensional coordinates and spatial positioning of the laser point cloud wave data, effectively making up for the limitations of single data, obtaining more comprehensive wave information, not only providing a solid data basis for subsequent construction of a wave model, but also enhancing the accuracy of the wave model. Then, the product of the real-time hull rotation matrix and the inverse matrix of the initial sensor offset matrix is calculated to obtain an attitude correction matrix. Based on the attitude correction matrix, the fused point cloud wave data is corrected to obtain updated fused point cloud wave data, performing attitude correction on the fused point cloud wave data to eliminate the influence of hull movement on the wave data, and improving the accuracy of the fused point cloud wave data. Finally, based on the fused point cloud wave data, a three-dimensional model of the wave is constructed to obtain a wave model, making the wave model more accurate in shape restoration and capable of truly reflecting the dynamic features of the wave such as undulation, breaking, and spray, providing a scientific basis for the safety assessment of offshore operations.In addition, the fused data effectively reduces the influence of noise and outliers, improves the stability and robustness of the overall data, and further ensures the reliability of the wave model under complex sea conditions. Therefore, the technical solution of the present invention solves the problem that the accuracy of the existing measurement method is difficult to meet the actual requirements.

[0154] Figure 3 FIG. 4 is a schematic structural diagram of a marine wave measurement device based on binocular vision and lidar provided by an embodiment of the present invention. This device and the marine wave measurement method based on binocular vision and lidar in the above embodiments belong to the same inventive concept. For the details not described in detail in the embodiment of the marine wave measurement device based on binocular vision and lidar, reference may be made to the embodiment of the marine wave measurement method based on binocular vision and lidar.

[0155] As Figure 3 shown, the device includes:

[0156] An acquisition module 310, configured to detect waves in the ship operation area through the lidar to obtain lidar point cloud wave data, photograph the waves through the first camera to obtain a first wave image, and photograph the waves through the second camera to obtain a second wave image;

[0157] A first calculation module 320, configured to subtract the position data of corresponding pixel points in the first wave image and the second wave image to obtain a wave disparity map;

[0158] A second calculation module 330, configured to calculate the depth information of each pixel point in the wave disparity map according to the internal parameters of the binocular camera, and determine the point cloud data corresponding to the wave disparity map based on the depth information of each pixel point to obtain visual point cloud wave data;

[0159] A fusion module 340, configured to fuse the visual point cloud wave data and the lidar point cloud wave data to obtain fused point cloud wave data;

[0160] A modeling module 350, configured to perform three-dimensional modeling on the waves based on the fused point cloud wave data to obtain a wave model.

[0161] Based on the above embodiments, the device further includes:

[0162] A calibration module is used to preprocess the first wave graph to obtain a first processed wave graph and preprocess the second wave graph to obtain a second processed wave graph before subtracting the position data of corresponding pixel points in the first wave graph and the second wave graph to obtain a wave parallax graph; construct an intermediate calibration matrix based on the binocular translation vector; perform epipolar calibration on the first processed wave graph according to the intermediate calibration matrix to obtain an updated first wave graph, and perform epipolar calibration on the second processed wave graph according to the intermediate calibration matrix to obtain an updated second wave graph.

[0163] Based on the above embodiments, the calibration module constructs an intermediate calibration matrix based on the binocular translation vector, including:

[0164] Normalize the binocular translation vector to obtain a first unit vector; normalize the cross product result of the first unit vector and a reference vector to obtain a second unit vector; calculate the cross product of the first unit vector and the second unit vector to obtain a third unit vector; construct the intermediate calibration matrix based on the first unit vector, the second unit vector, and the third unit vector.

[0165] Based on the above embodiments, the calibration module performs epipolar calibration on the first processed wave graph according to the intermediate calibration matrix to obtain an updated first wave graph, including:

[0166] Determine the square root of the binocular extrinsic matrix as a first initial semi-rotation matrix; calculate the product of the intermediate calibration matrix and the first initial semi-rotation matrix to obtain a first target calibration matrix; calculate the product of the first target rotation matrix and the first processed wave graph to obtain an updated first wave graph;

[0167] Correspondingly, the calibration module performs epipolar calibration on the second processed wave graph according to the intermediate calibration matrix to obtain an updated second wave graph, including:

[0168] Determine the square root of the inverse matrix of the binocular extrinsic matrix as a second initial semi-rotation matrix; calculate the product of the intermediate calibration matrix and the second initial semi-rotation matrix to obtain a second target calibration matrix; calculate the product of the second target rotation matrix and the second processed wave graph to obtain an updated second wave graph.

[0169] Based on the above embodiments, the calibration module preprocesses the first wave graph to obtain a first processed wave graph, including:

[0170] Perform image enhancement processing on the first wave graph to obtain a first enhanced wave graph; perform filtering processing on the first enhanced wave graph to obtain a first filtered wave graph; perform noise reduction processing on the first filtered wave graph to obtain the first processed wave graph;

[0171] Correspondingly, the correction module preprocesses the second wave graph to obtain a second processed wave graph, including:

[0172] Perform image enhancement processing on the second wave graph to obtain a second enhanced wave graph; perform filtering processing on the second enhanced wave graph to obtain a second filtered wave graph; perform noise reduction processing on the second filtered wave graph to obtain the second processed wave graph.

[0173] Based on the above embodiments, the device further includes:

[0174] A matching module, configured to, before subtracting the position data of corresponding pixel points in the first wave graph and the second wave graph to obtain a wave parallax map, for the current pixel point in the first wave graph, determine a matching window centered on the current pixel point; calculate the matching cost between the matching window and each pixel point in the second wave graph that is in the same row as the current pixel point; and determine the pixel point with the minimum matching cost as the corresponding pixel point of the current pixel point in the second wave graph.

[0175] Based on the above embodiments, the internal parameters of the binocular camera include the horizontal focal length and the baseline distance of the binocular camera. The second calculation module 330 calculates the depth information of each pixel point in the wave parallax map according to the internal parameters of the binocular camera, including:

[0176] Calculate the product of the horizontal focal length and the baseline distance to obtain a conversion coefficient;

[0177] Calculate the ratio of the conversion coefficient to the parallax value of each pixel point in the wave parallax map to obtain the depth information of each pixel point.

[0178] Based on the above embodiments, the internal parameters of the binocular camera further include the vertical focal length and the principal point position information of the binocular camera. The second calculation module 330 determines the point cloud data corresponding to the wave parallax map based on the depth information of each pixel point to obtain visual point cloud wave data, including:

[0179] Calculate the horizontal information of each pixel point according to the principal point position information, the horizontal focal length, and the depth information of each pixel point; calculate the vertical information of each pixel point according to the principal point position information, the vertical focal length, and the depth information of each pixel point; and determine the point cloud data corresponding to the wave parallax map based on the depth information, horizontal information, and vertical information of each pixel point to obtain the visual point cloud wave data.

[0180] Based on the above embodiments, the device further includes:

[0181] An error module, configured to calculate the error between the visual point cloud wave data and the lidar point cloud wave data based on the position data of the corresponding points, the camera-lidar rotation matrix, and the camera-lidar translation vector in the visual point cloud wave data and the lidar point cloud wave data before fusing the visual point cloud wave data and the lidar point cloud wave data to obtain the fused point cloud wave data, so as to obtain the point cloud fusion error; when the point cloud fusion error is less than a preset error threshold, trigger the execution of fusing the visual point cloud wave data and the lidar point cloud wave data to obtain the fused point cloud wave data; when the point cloud fusion error is not less than the preset error threshold, return to execute obtaining the lidar point cloud wave data by detecting the waves in the ship operation area through the lidar, obtaining the first wave image by photographing the waves through the first camera, and obtaining the second wave image by photographing the waves through the second camera.

[0182] Based on the above embodiments, the device further includes:

[0183] A correction module, configured to calculate the product of the real-time hull rotation matrix and the inverse matrix of the initial sensor offset matrix after fusing the visual point cloud wave data and the lidar point cloud wave data to obtain the attitude correction matrix; correct the fused point cloud wave data based on the attitude correction matrix to obtain the updated fused point cloud wave data.

[0184] The marine wave measurement device based on binocular vision and lidar provided by the embodiments of the present invention can execute the marine wave measurement method based on binocular vision and lidar provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method.

[0185] It should be noted that in the embodiments of the above marine wave measurement device based on binocular vision and lidar, the various units and modules included are only divided according to the functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the present invention.

[0186] Figure 4 It is a schematic structural diagram of a ship provided by an embodiment of the present invention. Figure 4 A block diagram of an exemplary ship 4 suitable for implementing the embodiments of the present invention is shown. Figure 4 The ship 4 shown is only an example and should not bring any limitation to the functions and usage scope of the embodiments of the present invention.

[0187] As shown Figure 4 in FIG. 4, the vessel 4 is embodied in the form of a general computing electronic device. The components of the vessel 4 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.

[0188] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus structures. By way of example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MAC) buses, Enhanced ISA buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.

[0189] The vessel 4 typically includes a variety of computer system readable media. Such media may be any available media that can be accessed by the vessel 4, including both volatile and nonvolatile media, removable and non-removable media.

[0190] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The vessel 4 may further include other removable / non-removable, volatile / nonvolatile computer system storage media. By way of example only, a storage system 34 may be provided for reading from and writing to a non-removable, nonvolatile magnetic medium ( Figure 4 not shown and typically referred to as a "hard disk drive"). Although Figure 4 not shown in FIG. 4, a magnetic disk drive may be provided for reading from and writing to a removable nonvolatile disk (such as a "floppy disk"), and an optical disk drive may be provided for reading from and writing to a removable nonvolatile optical disk (such as a CD-ROM, DVD-ROM, or other optical medium). In these cases, each drive may be connected to the bus 18 by one or more data media interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of the embodiments of the present invention.

[0191] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, and an implementation of a network environment may be included in some or all of these examples. The program modules 42 generally carry out the functions and / or methods of the embodiments described herein.

[0192] The ship 4 can also communicate with one or more external devices 14 (such as keyboards, pointing devices, monitors 24, etc.), and can also communicate with one or more devices that enable users to interact with the ship 4, and / or communicate with any device that enables the ship 4 to communicate with one or more other computing devices (such as network cards, modems, etc.). Such communication can be carried out through the input / output (I / O) interface 22. Moreover, the ship 4 can also communicate with one or more networks (such as local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through the network adapter 20. As Figure 4 shown, the network adapter 20 communicates with other modules of the ship 4 through the bus 18. It should be understood that although Figure 4 not shown in the figure, other hardware and / or software modules can be used in combination with the ship 4, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0193] The processing unit 16 executes various functional applications and page displays by running programs stored in the system memory 28. For example, it implements the method for measuring ocean waves based on binocular vision and lidar provided by the embodiments of the present invention, which is applied to a ship equipped with a binocular camera and a lidar. The binocular camera includes a first camera and a second camera. The method includes:

[0194] Detecting ocean waves in the ship's operation area through the lidar to obtain lidar point cloud wave data, taking a first wave image of the waves through the first camera, and taking a second wave image of the waves through the second camera;

[0195] Subtracting the position data of corresponding pixel points in the first wave image and the second wave image to obtain a wave disparity map;

[0196] Calculating the depth information of each pixel point in the wave disparity map according to the internal parameters of the binocular camera, and determining the point cloud data corresponding to the wave disparity map based on the depth information of each pixel point to obtain visual point cloud wave data;

[0197] Fusing the visual point cloud wave data and the lidar point cloud wave data to obtain fused point cloud wave data;

[0198] Performing three-dimensional modeling on the waves based on the fused point cloud wave data to obtain a wave model.

[0199] Of course, those skilled in the art can understand that the processor can also implement the technical solutions of the method for measuring ocean waves based on binocular vision and lidar provided by any embodiment of the present invention.

[0200] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements, for example, the method for measuring ocean waves based on binocular vision and lidar provided by the embodiment of the present invention, which is applied to a ship. A binocular camera and a lidar are installed on the ship. The binocular camera includes a first camera and a second camera. The method includes:

[0201] Detect ocean waves in the ship operation area through the lidar to obtain lidar point cloud wave data, capture the waves through the first camera to obtain a first wave image, and capture the waves through the second camera to obtain a second wave image;

[0202] Subtract the position data of corresponding pixel points in the first wave image and the second wave image to obtain a wave disparity map;

[0203] Calculate the depth information of each pixel point in the wave disparity map according to the internal parameters of the binocular camera, and determine the point cloud data corresponding to the wave disparity map based on the depth information of each pixel point to obtain visual point cloud wave data;

[0204] Fuse the visual point cloud wave data and the lidar point cloud wave data to obtain fused point cloud wave data;

[0205] Perform three-dimensional modeling on the waves based on the fused point cloud wave data to obtain a wave model.

[0206] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device.

[0207] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0208] The program code contained on a computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0209] The computer program code for performing the operations of the present invention can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., by using an Internet service provider to connect through the Internet).

[0210] Those of ordinary skill in the art should understand that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device. They can be concentrated on a single computing device or distributed over a network composed of multiple computing devices. Optionally, they can be implemented using program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module to be implemented. Thus, the present invention is not limited to any specific combination of hardware and software.

[0211] In addition, in the technical solution of the present invention, the acquisition, storage, use, processing, etc. of data all comply with the relevant regulations of laws and regulations.

[0212] Note that the above is only a preferred embodiment of the present invention and the applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments here, and various obvious changes, re-adjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments only. Without departing from the concept of the present invention, more other equivalent embodiments can be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for measuring sea waves based on binocular vision and lidar, characterized in that, Applied to a ship, where a binocular camera and a lidar are installed on the ship, the binocular camera includes a first camera and a second camera, and the method includes: Detecting waves in the ship operation area by the lidar to obtain lidar point cloud wave data, photographing the waves by the first camera to obtain a first wave image, and photographing the waves by the second camera to obtain a second wave image; Subtracting the position data of corresponding pixel points in the first wave image and the second wave image to obtain a wave disparity map; Calculating the depth information of each pixel point in the wave disparity map according to the internal parameters of the binocular camera, and determining the point cloud data corresponding to the wave disparity map based on the depth information of each pixel point to obtain visual point cloud wave data; Fusing the visual point cloud wave data and the lidar point cloud wave data to obtain fused point cloud wave data; Performing three-dimensional modeling on the waves based on the fused point cloud wave data to obtain a wave model.

2. The method for measuring sea waves based on binocular vision and lidar according to claim 1, wherein Before subtracting the position data of corresponding pixel points in the first wave image and the second wave image to obtain a wave disparity map, it further includes: Preprocessing the first wave image to obtain a first processed wave image, and preprocessing the second wave image to obtain a second processed wave image; Constructing an intermediate correction matrix based on the binocular translation vector; Performing epipolar correction on the first processed wave image according to the intermediate correction matrix to obtain an updated first wave image, and performing epipolar correction on the second processed wave image according to the intermediate correction matrix to obtain an updated second wave image.

3. The method for measuring sea waves based on binocular vision and lidar according to claim 2, wherein Constructing an intermediate correction matrix based on the binocular translation vector includes: Normalizing the binocular translation vector to obtain a first unit vector; Normalizing the cross product result of the first unit vector and the reference vector to obtain a second unit vector; Calculating the cross product of the first unit vector and the second unit vector to obtain a third unit vector; Constructing the intermediate correction matrix based on the first unit vector, the second unit vector, and the third unit vector.

4. The method for measuring sea waves based on binocular vision and lidar according to claim 2, wherein Performing epipolar correction on the first processed wave image according to the intermediate correction matrix to obtain an updated first wave image, including: Determining the square root of the binocular extrinsic matrix as a first initial semi-rotation matrix; Calculating the product of the intermediate correction matrix and the first initial semi-rotation matrix to obtain a first target correction matrix; Calculating the product of the first target rotation matrix and the first processed wave image to obtain an updated first wave image; Correspondingly, performing epipolar correction on the second processed wave image according to the intermediate correction matrix to obtain an updated second wave image, including: Determining the square root of the inverse matrix of the binocular extrinsic matrix as a second initial semi-rotation matrix; Calculating the product of the intermediate correction matrix and the second initial semi-rotation matrix to obtain a second target correction matrix; Calculating the product of the second target rotation matrix and the second processed wave image to obtain an updated second wave image.

5. The method for measuring sea waves based on binocular vision and lidar according to claim 2, characterized in that Preprocessing the first wave image to obtain a first processed wave image, including: Performing image enhancement processing on the first wave image to obtain a first enhanced wave image; Filter the first enhanced wave diagram to obtain a first filtered wave diagram; Perform noise reduction processing on the first filtered wave diagram to obtain the first processed wave diagram; Correspondingly, preprocess the second wave diagram to obtain a second processed wave diagram, including: Perform image enhancement processing on the second wave diagram to obtain a second enhanced wave diagram; Filter the second enhanced wave diagram to obtain a second filtered wave diagram; Perform noise reduction processing on the second filtered wave diagram to obtain the second processed wave diagram.

6. The method for measuring ocean waves based on binocular vision and lidar according to claim 1, wherein Before subtracting the position data of the corresponding pixel points in the first wave diagram and the second wave diagram to obtain a wave parallax diagram, it further includes: For the current pixel point in the first wave diagram, determine a matching window centered on the current pixel point; Calculate the matching cost between the matching window and each pixel point in the same row as the current pixel point in the second wave diagram; Determine the pixel point with the minimum matching cost as the corresponding pixel point of the current pixel point in the second wave diagram.

7. The method for measuring sea waves based on binocular vision and lidar according to claim 1, wherein The internal parameters of the binocular camera include the horizontal focal length and the baseline distance of the binocular camera. Calculating the depth information of each pixel point in the wave parallax diagram according to the internal parameters of the binocular camera includes: Calculate the product of the horizontal focal length and the baseline distance to obtain a conversion coefficient; Calculate the ratio of the conversion coefficient to the parallax value of each pixel point in the wave parallax diagram to obtain the depth information of each pixel point.

8. The method for measuring sea waves based on binocular vision and lidar according to claim 7, characterized in that, The internal parameters of the binocular camera further include the vertical focal length and the principal point position information of the binocular camera. Determining the point cloud data corresponding to the wave parallax diagram based on the depth information of each pixel point to obtain visual point cloud wave data includes: Calculate the horizontal information of each pixel point according to the principal point position information, the horizontal focal length, and the depth information of each pixel point; Calculate the vertical information of each pixel point according to the principal point position information, the vertical focal length, and the depth information of each pixel point; Determine the point cloud data corresponding to the wave parallax diagram based on the depth information, horizontal information, and vertical information of each pixel point to obtain the visual point cloud wave data.

9. The method for measuring sea waves based on binocular vision and lidar according to claim 1, characterized in that Before fusing the visual point cloud wave data and the laser point cloud wave data to obtain fused point cloud wave data, it further includes: Calculate the error between the visual point cloud wave data and the laser point cloud wave data based on the position data of the corresponding points, the camera-lidar rotation matrix, and the camera-lidar translation vector in the visual point cloud wave data and the laser point cloud wave data to obtain a point cloud fusion error; When the point cloud fusion error is less than a preset error threshold, trigger the execution of fusing the visual point cloud wave data and the laser point cloud wave data to obtain fused point cloud wave data; When the point cloud fusion error is not less than the preset error threshold, return to execute obtaining laser point cloud wave data by detecting the waves in the ship operation area through the lidar, obtaining a first wave diagram by shooting the waves with the first camera, and obtaining a second wave diagram by shooting the waves with the second camera.

10. The method for measuring sea waves based on binocular vision and lidar according to claim 1, wherein After fusing the visual point cloud wave data and the laser point cloud wave data to obtain fused point cloud wave data, the following steps are further included: Calculate the product of the real-time hull rotation matrix and the inverse matrix of the initial sensor offset matrix to obtain an attitude correction matrix; Correct the fused point cloud wave data based on the attitude correction matrix to obtain updated fused point cloud wave data.

Images