A sea-sky line detection method based on image processing

Abstract

The application discloses a sea-sky line detection method based on image processing, and relates to the field of image processing. The method can estimate the relative attitude relationship between the camera coordinate system and the sea level and the camera attitude transformation matrix by using the measured motion data provided by a motion sensing unit. The direction of the sea-sky line can be determined according to the relative attitude relationship between the camera coordinate system and the sea level. Then, the search range is narrowed by using the direction constraint to quickly locate the sea-sky line observation result. The predicted position of the sea-sky line is obtained by using the camera attitude transformation matrix, and the predicted position is jointly optimized with the sea-sky line observation result. Even in the scene of fast ship body shaking and image blur, the position of the sea-sky line can be stably estimated. Stable estimation can be provided in the poor visibility or complex sea state, and the precision, stability, robustness and efficiency of the sea-sky line tracking task are comprehensively improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a method for detecting sea surface area based on image processing. Background Technology

[0002] The sea horizon is the boundary between the sky and the sea surface in an image. Sea horizon detection has a wide range of far-reaching applications. It is a fundamental technology in computer vision and marine engineering, and has important applications in promoting scientific research, ensuring navigation safety, and promoting economic development.

[0003] Traditional methods for detecting sea-line features include: (1) Based on grayscale or gradient features, using the grayscale difference or vertical gradient change of the upper and lower regions of the sea-line feature for detection. Its advantages are simple calculation, fast detection speed, and suitability for real-time processing under ideal sea conditions. However, it is less robust when encountering interference with similar gradient features such as waves and clouds. (2) Based on image segmentation, i.e., segmentation based on the grayscale difference between the sky and the sea surface. This method is intuitive and simple to implement, and has good effect on images with obvious differences between the sky and the sea. However, the segmentation accuracy decreases when there is uneven lighting, near-field targets, or noise. (3) Direct line detection, such as detection methods based on Hough transform. This method has good effect on detecting complete lines, but the computation is large and only edge information is considered. It is easily affected by interference with straight-line features such as clouds and waves. (4) Based on polarization imaging, using the difference in polarization characteristics of reflected light from the sky and the sea surface to enhance the features of the sea-line feature. This method can effectively improve the signal-to-noise ratio of the sea-line feature in low-contrast environments. However, it requires dedicated polarization imaging equipment, the system is more complex, and it is greatly affected by noise and sea clutter. (5) Based on deep learning, semantic segmentation of the sea and sky region is performed, and then the boundary line is extracted. This method is usually robust, can adapt to a variety of complex sea and sky backgrounds, and has good accuracy. However, it relies on a large amount of labeled data to train the model, consumes a lot of computing resources, and may be limited in real time. Summary of the Invention

[0004] This application addresses the aforementioned problems and technical requirements by proposing an image processing-based method for detecting sea surface area. The technical solution of this application is as follows:

[0005] A sea-line detection method based on image processing, the sea-line detection method comprising:

[0006] Using a camera to acquire Images of the sea and sky scene at any given moment, and images obtained using a motion sensing unit. Real-time motion data, rigid connection between the motion sensing unit and the camera, integer parameters. ;

[0007] according to Estimation of measured motion data at time points The relative attitude relationship between the camera coordinate system and the sea level at any given time and Camera pose transformation matrix at time step Among them, the world coordinate system and The camera coordinate system coincides with the world coordinate system at any given moment. The plane is parallel to the sea level;

[0008] based on The relative attitude relationship between the camera coordinate system and the sea level at any given time is determined. The direction of the horizon line in the sea-sky scene image frame at any given time, and the direction of the horizon line is used to determine the... The long-line extraction of the sea and sky scene image frame at a given time is obtained by directional long-line extraction. Sea-line observation results at time ;

[0009] according to Camera pose transformation matrix at time step and The position of the horizon at any given time predict Predicted position of the sea-line at a given time And based on the observation results of the sea-sky line Predicted location of the sea-line The error of the camera pose transformation matrix Perform iterative optimization until the iteration termination condition is met to obtain The position of the horizon at any given time ;in, Time is The moment before the moment, The camera coordinate system at any given moment coincides with the world coordinate system.

[0010] Its further technical solution is to adjust the orientation of the sea-line line according to the direction of the sea-line line. The long-line extraction of the sea and sky scene image frame at a given time is obtained by directional long-line extraction. Sea-line observation results at time include:

[0011] calculate The directional curvature spectrum of the sea and sky scene image frame at any given time in the direction of the sea-sky line;

[0012] After binarizing the directional curvature spectrum, directional line segments whose length reaches the length threshold are extracted as a set of directional line segments;

[0013] Perform a merge operation on each oriented line segment in the oriented line segment set to obtain Sea-line observation results at time .

[0014] A further technical solution involves performing a merging operation on each oriented line segment in the oriented line segment set, including:

[0015] For oriented line segments whose shortest distance between segments is less than a distance threshold and whose included angle between segments is less than an angle threshold, merge them. Calculate the average of the midpoints of each oriented line segment before merging as the midpoint of the merged oriented line segment. Calculate the average of the direction vectors of each oriented line segment before merging as the direction vector of the merged oriented line segment. The line segment in the sea and sky scene image frame at a given time that follows the direction vector of the merged directional line segment and passes through the midpoint of the merged directional line segment is taken as the merged directional line segment.

[0016] Its further technical solution is to calculate The orientation curvature spectrum of the sea-sky scene image frame at a given time along the direction of the horizon includes:

[0017] for Any pixel in a seascape image frame at any given moment ,extract The current pixel in the sea and sky scene image frame at any given moment. The image within a local window centered on a pixel. Based on the Facet model, according to the image Calculate pixels Second derivative at , , And calculate the pixel points Along the direction of the sea-line at the current local window scale Reference curvature ;in, yes The direction of the horizon in the seascape image frame at any given moment The angle with the horizontal direction of the image;

[0018] Calculate the pixel points separately In the direction of the sea-line at multiple different local window scales Reference curvature on , to pixel Reference curvature at all local window scales The maximum value is used as the pixel point In the direction of the horizon directional curvature on And obtain the directional curvature spectrum.

[0019] Its further technical solution is a camera coordinate system. With the camera's optical center as the origin , The optical axis aligns with the axial direction and points in the imaging direction. The axis is parallel to the horizontal direction of the image plane. The axes are parallel to the perpendicular direction of the image plane and form a right-handed coordinate system; the world coordinate system. of , , The shafts are respectively with Camera coordinate system at time , , Axis coincidence; determine the obtained The relative attitude relationship between the camera coordinate system and the sea level at any given time includes... The camera coordinate system at any given moment revolves around the world coordinate system. Euler angles of rotation of axis ;based on The relative attitude relationship between the camera coordinate system and the sea level at any given time is determined. The direction of the horizon in the sea-sky scene image frame at any given time includes:

[0020] Sure The direction of the horizon in the seascape image frame at any given moment The angle between the pixel coordinate system and the u-axis satisfy , For camera intrinsic parameters, the pixel coordinate system takes the top left corner of the sea and sky scene image frame as the origin, with the u-axis pointing horizontally to the right and the v-axis pointing vertically downwards.

[0021] Its further technical solution is, according to Estimation of measured motion data at time points The relative attitude relationship between the camera coordinate system and the sea level at any given time includes:

[0022] Theoretical acceleration in world coordinate system Perform coordinate transformation to obtain Ideal acceleration in the coordinate system of the motion sensing unit at all times ;in, It is a rotation transformation model from the camera coordinate system to the motion sensing unit coordinate system. Is the world coordinate system to The rotation transformation model of the camera coordinate system at time t and , It is about the world coordinate system Axis rotation Obtain the rotation matrix of the intermediate transition coordinate system. It is around the intermediate transition coordinate system Axis rotation The rotation matrix;

[0023] Solve the objective equation of the LM nonlinear optimization algorithm get Euler angles of time and To determine The relative attitude relationship between the camera coordinate system and the sea level at any given time, where, It is the motion sensing unit in The measured acceleration at any given moment.

[0024] Its further technical solution is, according to Estimation of measured motion data at time points Camera pose transformation matrix at time step include:

[0025] Build Measured angular velocity at time t The corresponding pure quaternion And the fourth-order Runge-Kutta method was used to estimate Quaternion of time , , , , ;in, To represent quaternion multiplication, It is the time step. yes Quaternion at time;

[0026] For the estimated quaternions Normalization processing Then, determine the quaternion. real part and the virtual part And calculate the rotation matrix. As a motion sensing unit Attention transformation matrix at time step ;

[0027] Combining the rotation transformation model from the camera coordinate system to the motion sensing unit coordinate system ,get Camera pose transformation matrix at time step .

[0028] Its further technical solution is, according to Camera pose transformation matrix at time step and The position of the horizon at any given time predict Predicted position of the sea-line at a given time include:

[0029] Prediction using homography matrix Predicted position of the horizon in the seascape image frame at any given time ;in, This is the intrinsic parameter matrix of the camera. , , For camera intrinsic parameters, image principal point It is the intersection of the camera's optical axis and the image frame of the sea and sky scene; yes The scale factor corresponding to the time. yes The scale factor corresponding to the time.

[0030] A further technical solution is that the sea-line detection method also includes:

[0031] Using a camera to acquire Extracting images of the sea and sky scene at any given moment. The image frame passing through the principal point of the sea and sky scene at a certain moment. And the pixels along the straight line in the horizontal direction of the image plane are obtained The position of the horizon at any given time Image principal point It is the intersection of the camera's optical axis and the image frame of the sea and sky scene.

[0032] Its further technical solution is based on the observation results of the sea-sky horizon. Predicted location of the sea-line The error of the camera pose transformation matrix Iterative optimization until the iteration termination condition is met includes:

[0033] Calculate the results of sea-sky observations Predicted location of the sea-line The deviation between them is used as Single-frame observation error at time 1 ;

[0034] The overall objective function is calculated as follows: ,in, Represents the single-frame observation error across all frames. the sum of This represents the attitude error correction amount for all frames. The sum of squares, It is the regularization coefficient. This indicates taking the minimum value;

[0035] The total objective function is determined by solving the nonlinear least squares method. Attitude error correction at time step The attitude error correction amount Superimposed on the camera pose transformation matrix Up, and utilize the updated camera pose transformation matrix Continue iterating until the iteration termination condition is met.

[0036] The beneficial technical effects of this application are:

[0037] This application discloses an image processing-based sea-line detection method. This method uses measured motion data provided by a motion sensing unit to estimate the relative attitude relationship between the camera coordinate system and the sea level, as well as the camera attitude transformation matrix. Based on the relative attitude relationship between the camera coordinate system and the sea level, the direction of the sea-line can be determined. Then, the direction constraint is used to narrow the search range and quickly locate the sea-line observation results. The predicted position of the sea-line is obtained using the camera attitude transformation matrix and jointly optimized with the sea-line observation results. Even in scenarios with rapid ship swaying and blurred images, the position of the sea-line can be stably estimated. It can provide stable estimation in poor visibility or complex sea conditions, bringing a comprehensive improvement in accuracy, stability, robustness, and efficiency to the sea-line tracking task.

[0038] This method employs a curvature-based directional long-line extraction algorithm to extract sea-sky observation results. It integrates directional priors and local curvature analysis, utilizes directional constraints to narrow the search range, and combines curvature consistency to effectively filter irrelevant edges, thereby significantly improving processing speed. Attached Figure Description

[0039] Figure 1 This is a flowchart of a method for detecting the sea surface area according to an embodiment of this application.

[0040] Figure 2 yes Camera coordinate system at time A schematic diagram of the plane being parallel to the sea level.

[0041] Figure 3 In one example, the orientation of the sea-line is used to... The long-line extraction of the sea and sky scene image frame at a given time is obtained by directional long-line extraction. Sea-line observation results at time A schematic diagram.

[0042] Figure 4 yes Camera coordinate system at time A schematic diagram of the pose when the plane is not parallel to the sea level. Detailed Implementation

[0043] The specific embodiments of this application will be further described below with reference to the accompanying drawings.

[0044] This application discloses an image processing-based method for detecting sea surface area. Please refer to [link / reference]. Figure 1 The flowchart shown illustrates that the sea-line detection method includes the following steps:

[0045] Step 110, use the camera to acquire Images of the sea and sky scene at any given moment, and images obtained using a motion sensing unit. Measured motion data at time points, integer parameters .

[0046] The sea-line detection method of this application utilizes an observation system for data acquisition. The observation system includes a camera and a motion sensing unit, with the motion sensing unit rigidly connected to the camera. The observation system is fixed on a marine vessel such as a ship or a floating platform. The camera used in this application is a high frame rate camera, and the image acquisition frequency of the camera is comparable to the data acquisition frequency of the motion sensing unit. Furthermore, this step synchronizes the original image frames acquired by the camera and the original data acquired by the motion sensing unit. For the time synchronization of the sea and sky scene image frames and the measured motion data at any given time, please refer to existing methods, which will not be elaborated here.

[0047] Define a set of coordinate systems to accurately describe the spatial relationships between the components, including the camera coordinate system, world coordinate system, motion sensing unit coordinate system, image coordinate system, and pixel coordinate system:

[0048] (1) Camera coordinate system With the camera's optical center as the origin , The optical axis aligns with the axial direction and points in the imaging direction. The axis is parallel to the horizontal direction of the image plane. The axes are parallel to the perpendicular direction of the image plane and form a right-handed coordinate system. Initially... Camera coordinate system at time The plane is parallel to the sea level.

[0049] (2) World coordinate system : with the initial The camera coordinate system at any given time serves as the global spatial reference base, i.e., the world coordinate system. and The camera coordinate system coincides with the world coordinate system at any given moment. The origin Located at the beginning At the center of the camera's light at that moment, , , The shafts are respectively with Camera coordinate system at time , , Coincident axes, world coordinate system The plane is parallel to the sea level.

[0050] (3) Motion sensing unit coordinate system The origin is the intersection of the sensor measurement centers of the motion sensing unit. Its three-axis sensitive directions are respectively , , axis.

[0051] (4) Image coordinate system: The point of intersection between the camera optical axis and the image plane, i.e., the principal point of the image. With the origin as the coordinate axis, the two coordinate axes of the image coordinate system are parallel to the two vertical sides of the image plane.

[0052] (5) Pixel coordinate system: with the top left corner of the image as the origin, the u-axis moves horizontally to the right to correspond to the column number, and the v-axis moves vertically downward to correspond to the row number.

[0053] The camera and motion sensing unit perform synchronous data acquisition. The camera, acting as the visual sensing unit, is responsible for acquiring sequential images. The motion sensing unit integrates a three-axis accelerometer and a three-axis gyroscope, used to measure the linear acceleration and angular velocity of the carrier in three-dimensional space, respectively. Therefore, the motion sensing unit acquires... The measured motion data at each moment includes Measured acceleration at time and measured angular velocity In one instance, the motion sensing unit is implemented using an IMU (Inertial Measurement Unit).

[0054] After assembling the observation system, camera parameter calibration was performed to obtain camera intrinsic parameters. Lens distortion is corrected to ensure the accuracy of the imaging geometry. Finally, an assembly error model between the camera and the motion sensing unit is established, that is, a rotational transformation model from the camera coordinate system to the motion sensing unit coordinate system is determined. This provides an accurate spatiotemporal alignment basis for subsequent data fusion. Due to the rigid connection between the camera and the motion sensing unit, this rotational transformation model, once assembled, provides a precise basis for data fusion. It remains unchanged.

[0055] In addition, during actual operation, data preprocessing is required for both the seascape image frames and the measured motion data to ensure data quality. This includes image enhancement processing for the seascape image frames, including denoising, contrast enhancement, and deblurring. Zero-bias compensation and time synchronization are performed on the measured motion data; existing methods can be referenced for this part, and will not be elaborated here.

[0056] Step 120, according to Estimation of measured motion data at time points The relative attitude relationship between the camera coordinate system and the sea level at any given time and Camera pose transformation matrix at time step The details are as follows:

[0057] (1) Once the observation system is fixed on the sea vessel, it will move continuously with the waves, causing the camera coordinate system to... The plane is often no longer parallel to the sea level, therefore, firstly based on... Estimation of measured motion data at time points The relative attitude relationship between the camera coordinate system and the sea level at any given time.

[0058] Please refer to Figure 2 In an ideal situation, the theoretical acceleration in the world coordinate system pass through in sequence The rotation transformation models from the world coordinate system to the camera coordinate system at any given time, and the rotation transformation models from the camera coordinate system to the motion sensing unit coordinate system, can be transformed into... The ideal acceleration of the three axes in the coordinate system of the motion sensing unit at all times for:

[0059] (1)

[0060] in, It is a rotational transformation model from the camera coordinate system to the motion sensing unit coordinate system, which can be determined after assembly. Is the world coordinate system to A rotational transformation model of the camera coordinate system at any given time.

[0061] When the camera coordinate system When the plane is parallel to the sea level, around the camera coordinate system Rotation of the axis does not affect the position of the sea-line in the image, therefore the world coordinate system can be adjusted to... Rotation transformation model of camera coordinate system at time point Decomposed into Euler angles about the other two axes, therefore:

[0062] (2)

[0063] in, It is about the world coordinate system Axis rotation The rotation matrix, about the world coordinate system Rotation Obtain the intermediate transition coordinate system. It continues to revolve around the intermediate transition coordinate system Axis rotation The rotation matrix.

[0064] With motion sensing unit in Measured acceleration at time Compared with ideal acceleration Using the minimum difference as a constraint, an objective equation based on the Levenberg-Marquardt nonlinear optimization algorithm is constructed. Solving the objective equation yields Rotational transformation model at time And it can be solved. Euler angles of time and Therefore, it can be determined The relative attitude relationship between the camera coordinate system and the sea level at any given time.

[0065] (2) According to Estimation of measured motion data at time points Camera pose transformation matrix at time step .

[0066] In one embodiment, estimation is performed using the quaternion method. Camera pose transformation matrix at time step This includes the following steps:

[0067] Read the measurements taken by the motion sensing unit Measured angular velocity at time t and build Measured angular velocity at time t The corresponding pure quaternion The quaternion The real part is 0, and the imaginary part is the measured angular velocity of the three axes. .

[0068] Estimation using the fourth-order Runge-Kutta method Quaternion of time for:

[0069] (3)

[0070] in,

[0071] (4)

[0072] in, yes Quaternion at time, initial Quaternion of time It is initialized during the calibration phase. To represent quaternion multiplication, It is the time step.

[0073] The quaternion calculated according to formula (3) Normalization is performed:

[0074] (5)

[0075] In equation (5), It is a quaternion The modulus length. Then determine the normalized quaternion. real part and the virtual part Thus, quaternions Transform into a rotation matrix :

[0076] (6)

[0077] The rotation matrix is ​​calculated according to equation (6). That is, the motion sensing unit in Attention transformation matrix at time step Then, combine this with the rotation transformation model from the camera coordinate system to the motion sensing unit coordinate system. Then it can be calculated Camera pose transformation matrix at time step :

[0078] (7)

[0079] Step 130, based on The relative attitude relationship between the camera coordinate system and the sea level at any given time is determined. The direction of the horizon line in the seascape image frame at any given moment.

[0080] In the initial Camera coordinate system at time When the plane is parallel to the sea level, any point in the world coordinate system and its projection in the pixel coordinate system There is a corresponding relationship as shown in equation (8):

[0081] (8)

[0082] in, This is the intrinsic parameter matrix of the camera. , For camera intrinsic parameters, image principal point It is the intersection of the camera's optical axis and the image frame of the sea and sky scene. It is the camera coordinate system The extrinsic parameter matrix when the plane is parallel to the sea level. It is a non-zero scale factor for homogeneous coordinates, used to convert homogeneous coordinates into non-homogeneous pixel coordinates.

[0083] exist Camera coordinate system at time When a plane is not parallel to the sea level, points in the world coordinate system and its projection in the pixel coordinate system There is also a corresponding relationship between them, as shown in equation (9):

[0084] (9)

[0085] in, It is the camera coordinate system The extrinsic parameter matrix when the plane is not parallel to the sea level. It is a non-zero scale factor for homogeneous coordinates, used to convert homogeneous coordinates into non-homogeneous pixel coordinates.

[0086] As mentioned above, the world coordinate system to Rotation transformation model of camera coordinate system at time point It can be decomposed into Euler angles about the X and Z axes, therefore:

[0087] (10)

[0088] By establishing a joint camera coordinate system The relationship between the plane being parallel and not parallel to the sea level, and the derivation based on the above conditions, is as shown in equation (11):

[0089] (11)

[0090] in, yes inverse matrix and . Expressing the request The reverse. . A point in the world coordinate system Projection in pixel coordinate system homogeneous pixel coordinates, A point in the world coordinate system Projection in pixel coordinate system The homogeneous pixel coordinates.

[0091] Therefore, for the intermediate parameter matrix There are:

[0092] (12)

[0093] Camera coordinate system When the plane is parallel to the sea level, the sea horizon appears as a horizontal line in the camera's image plane. If we take two points on the sea horizon in this case, we define... The direction of the horizon in the seascape image frame at any given moment The angle between the pixel coordinate system and the u-axis is ,Pick For any two points on the horizon of the sea and sky scene in a frame at a given time, then... , and Two points taken from the horizon respectively and The coordinate difference between the u-axis and v-axis in the pixel coordinate system. An expression can be derived from these two sets of corresponding points:

[0094] (13)

[0095] (14)

[0096] Simplifying the above formulas, we can obtain:

[0097] (15)

[0098] Then, in the solution, we get The camera coordinate system at any given moment revolves around the world coordinate system. Euler angles of rotation of axis Then, substituting into equation (15) will determine the result. The direction of the horizon in the seascape image frame at any given moment The angle between the pixel coordinate system and the u-axis This determined the direction of the sea-line. .

[0099] Step 140, according to the direction of the horizon. The long-line extraction of the sea and sky scene image frame at a given time is obtained by directional long-line extraction. Sea-line observation results at time .

[0100] Having already estimated and determined the direction of the horizon line in the seascape image frame through step 130, this step can then perform detection in that direction, thereby improving detection efficiency and accuracy. The steps include the following:

[0101] 1. Calculation The directional curvature spectrum of the sea-sky scene image frame at time t is given by the direction of the horizon. Any pixel in a seascape image frame at any given moment :

[0102] extract The current pixel in the sea and sky scene image frame at any given moment. The image within a local window centered on a pixel. Then, based on the Facet model, according to the image Calculate pixels Second derivative at , , .

[0103] Then, the pixel points are calculated according to the following formula. Along the direction of the sea-line at the current local window scale Reference curvature :

[0104] (16)

[0105] One approach is to calculate the reference curvature according to formula (16). Directly used as this pixel point In the direction of the horizon directional curvature on The orientation curvature spectrum is obtained by summing the orientation curvature of all pixels.

[0106] To better adapt to the changes in antenna width under different resolution cameras, in another embodiment, after obtaining each pixel... Along the direction of the sea-line at the current local window scale Reference curvature Then, change the local window scale and repeat the above process to calculate the pixel points respectively. In the direction of the sea-line at multiple different local window scales Reference curvature on For example, calculations are typically performed at three different local window scales. The local window scale used is set based on engineering experience, such as the common 7*7, 9*9, and 11*11. Then the current pixel is... Reference curvature at all local window scales The maximum value is taken as the pixel point. In the direction of the horizon directional curvature on The orientation curvature spectrum is obtained by summing the orientation curvature of all pixels.

[0107] 2. After binarizing the curvature spectrum of this orientation, the extracted length reaches the length threshold. The directional line segments are taken as a set of directional line segments. The directional line segments extracted from the set of directional line segments are along the direction of the sea-line and have a length that reaches a length threshold. When binarizing the curvature spectrum of this orientation direction, the curvature of the orientation direction is... Reaching the curvature threshold Set the pixel value of the pixel to 255, and adjust the curvature of the orientation direction. Curvature threshold not reached The pixel value of each pixel is set to 0, thus completing the binarization of the directional curvature spectrum.

[0108] When removing short lines to extract oriented line segments, the length threshold A value can be determined based on engineering experience. Alternatively, in another embodiment, a length threshold can be used. , This is a proportionality coefficient, with a typical value of 0.13. It is the width of the sea and sky scene image frame. It is the height of the sea and sky scene image frame. This indicates taking the minimum value. In this embodiment, the length threshold... Adaptive adjustment based on the image size of the seascape image frame helps improve the accuracy of short line removal, thereby enhancing the effectiveness of subsequent analysis.

[0109] 3. Perform a merge operation on each oriented line segment in the oriented line segment set to obtain... Sea-line observation results at time During merging, oriented line segments whose shortest distance between segments is less than a distance threshold and whose included angle is less than an angle threshold are merged. The average of the midpoints of each oriented line segment before merging is used as the midpoint of the merged oriented line segment, and the average of the direction vectors of each oriented line segment before merging is used as the direction vector of the merged oriented line segment. In the seascape image frame at a given time, the line segment that follows the direction vector of the merged directional line segment and passes through the midpoint of the merged directional line segment is taken as the merged directional line segment. By performing a merge operation, separated directional line segments and parallel directional line segments can be merged.

[0110] In practice, the merging operation is performed using a traversal approach. Specifically:

[0111] (1) Iterate through each directional line segment in the set of directional line segments in turn, and use the currently traversed directional line segment as the reference line segment.

[0112] (2) Calculate the shortest distance between each oriented line segment in the oriented line segment set and the reference line segment, and select oriented line segments whose shortest distance to the reference line segment is less than the distance threshold to form a candidate merging set for the reference line segment. If there is no corresponding candidate merging set for the current reference line segment, continue to traverse the next oriented line segment in the oriented line segment set as the reference line segment until the oriented line segment set is traversed completely.

[0113] (3) Calculate the angle between each oriented line segment and the reference line segment in the candidate merging set of the reference line segment, and select oriented line segments in the candidate merging set whose angle between the line segment and the reference line segment is less than the angle threshold. Merge the selected oriented line segments with the reference line segment, determine the direction vector of the merged oriented line segment, and repeat (2) and (3) with the merged oriented line segment as the new reference line segment. If there is no oriented line segment in the candidate merging set of the current reference line segment whose angle between the line segment and the reference line segment is less than the angle threshold, continue to traverse the next oriented line segment in the oriented line segment set as the reference line segment until the oriented line segment set is traversed.

[0114] For example, in one instance, such as Figure 3 As shown, Images of the sea and sky scene at any moment, frame by frame. Figure 3 As shown in (a), calculate The directional curvature spectrum of the sea-sky scene image frame at a given time along the direction of the horizon is as follows: Figure 3 As shown in (b) above, the curvature spectrum of this orientation direction is binarized as follows: Figure 3 As shown in (c), after further short line removal, the remaining length reaches the length threshold. Oriented line segments such as Figure 3 As shown in (d), performing a merge operation on each oriented line segment in the oriented line segment set yields... Sea-line observation results at time like Figure 3 As shown in (e) in the diagram.

[0115] Step 150, according to Camera pose transformation matrix at time step and The position of the horizon at any given time predict Predicted position of the sea-line at a given time .

[0116] in, Time is The moment before the moment, The time is the initial time. The homography matrix can be used to predict... Predicted position of the horizon in the seascape image frame at any given time As shown in equation (17):

[0117] (17)

[0118] in, yes The scale factor corresponding to the time. yes The scale factor corresponding to the time.

[0119] In prediction Predicting the position of the sea-line at a given time requires the use of the initial... The position of the horizon at any given time Therefore, in the initial After the installation and calibration of the observation system are completed, it is also necessary to determine The position of the horizon at any given time It includes the following:

[0120] Please refer to Figure 4 ,because Camera coordinate system at time If the plane is parallel to the sea level, then the normal vector of the sea level can be determined as follows: Assuming the camera coordinate system is... The distance between the plane and the sea level is Therefore, the sea level in the camera coordinate system can be represented by the parametric equation as follows: Any point on the sea level can be represented in the camera coordinate system as: The second pixel coordinates of the projection point in the pixel coordinate system of the image plane can be determined based on the camera projection model. for:

[0121] (18)

[0122] By rearranging equation (18), we can obtain the projection point of any point on the sea level in the pixel coordinate system. have:

[0123] (19)

[0124] Considering that the sea-line is located at a very far distance, the coordinates of a point on the sea-line in the camera coordinate system are... Then, within the error range, we can take... Simplifying equation (19) yields:

[0125] (20)

[0126] From equation (20), it can be seen that in At what moment, when the camera coordinate system When the plane is parallel to the sea level, the projection of the sea-sky line in the world coordinate system onto the pixel coordinate system passes through the principal point of the sea-sky scene image frame. Furthermore, its direction is parallel to the u-axis of the pixel coordinate system. Since the camera principal point is usually located in the center region of the image, the sea-line will appear as a horizontal straight line passing through the center of the image. Based on this, when using the camera to acquire... After capturing the sea and sky scene image frames at a given moment, extract directly... The image frame passing through the principal point of the sea and sky scene at a certain moment. And the pixels along the straight line in the horizontal direction of the image plane are obtained The position of the horizon at any given time .

[0127] Step 160, based on the observation results from the sea-sky horizon. Predicted location of the sea-line The error of the camera pose transformation matrix Perform iterative optimization until the iteration termination condition is met to obtain The position of the horizon at any given time .

[0128] The observation results of the sea-sky line obtained in step 140 and the predicted position of the sea-line obtained in step 150 Joint optimization is performed, with optimization variables including sea-line parameters and attitude error corrections at various time points. Sea-line observation results are then calculated. Predicted location of the sea-line The deviation between them on the image plane is used as Single-frame observation error at time 1 Specifically, the results of sea-line observations can be calculated. Pixel to sea-line prediction position The sum of squared distances.

[0129] Then the overall objective function is calculated as follows: .in, Represents the single-frame observation error across all frames. the sum of This represents the attitude error correction amount for all frames. The sum of squares, It is the regularization coefficient. This indicates taking the minimum value.

[0130] The overall objective function is determined by solving the nonlinear least squares method. Attitude error correction at time step The attitude error correction amount Superimposed on the camera pose transformation matrix Up, and utilize the updated camera pose transformation matrix Continue iterating until the iteration termination condition is met, then obtain the final predicted position of the sea-line. As The position of the horizon at any given time Among them, when the maximum number of iterations or the attitude error correction amount is reached... The iteration termination condition is determined when the change is less than the error threshold. The maximum number of iterations can be set by the user.

[0131] The above descriptions are merely preferred embodiments of this application, and this application is not limited to the above embodiments. It is understood that other improvements and variations that can be directly derived or conceived by those skilled in the art without departing from the spirit and concept of this application should be considered to be included within the protection scope of this application.

Claims

1. A method for detecting sea-line elevation based on image processing, characterized in that, The sea level detection method includes: Using a camera to acquire Images of the sea and sky scene at any given moment, and images obtained using a motion sensing unit. Real-time motion data, rigid connection between the motion sensing unit and the camera, integer parameters. ; according to Estimation of measured motion data at time points The relative attitude relationship between the camera coordinate system and the sea level at any given time and Camera pose transformation matrix at time step Among them, the world coordinate system and The camera coordinate system coincides with the world coordinate system at any given moment. The plane is parallel to the sea level; based on The relative attitude relationship between the camera coordinate system and the sea level at any given time is determined. The direction of the horizon line in the sea-sky scene image frame at any given time, and the direction of the horizon line is used to determine the... The long-line extraction of the sea and sky scene image frame at a given time is obtained by directional long-line extraction. Sea-line observation results at time ; according to Camera pose transformation matrix at time step and The position of the horizon at any given time predict Predicted position of the sea-line at a given time And based on the observation results of the sea-sky line Predicted location of the sea-line The error of the camera pose transformation matrix Perform iterative optimization until the iteration termination condition is met to obtain The position of the horizon at any given time ;in, Time is The moment before the moment, The camera coordinate system at any given moment coincides with the world coordinate system.

2. The method for detecting sea level according to claim 1, characterized in that, Based on the direction of the horizon The long-line extraction of the sea and sky scene image frame at a given time is obtained by directional long-line extraction. Sea-line observation results at time include: calculate The directional curvature spectrum of the sea and sky scene image frame at any given time in the direction of the sea-sky line; After binarizing the curvature spectrum of the orientation direction, the orientation line segments whose length reaches the length threshold are extracted as the orientation line segment set; Perform a merge operation on each oriented line segment in the oriented line segment set to obtain Sea-line observation results at time .

3. The method for detecting sea level according to claim 2, characterized in that, Performing a merge operation on each oriented line segment in the oriented line segment set includes: For oriented line segments whose shortest distance between segments is less than a distance threshold and whose included angle between segments is less than an angle threshold, merge them. Calculate the average of the midpoints of each oriented line segment before merging as the midpoint of the merged oriented line segment. Calculate the average of the direction vectors of each oriented line segment before merging as the direction vector of the merged oriented line segment. The line segment in the sea and sky scene image frame at a given time that follows the direction vector of the merged directional line segment and passes through the midpoint of the merged directional line segment is taken as the merged directional line segment.

4. The method for detecting sea level according to claim 2, characterized in that, calculate The orientation curvature spectrum of the sea-sky scene image frame at a given time along the direction of the horizon includes: for Any pixel in a seascape image frame at any given moment ,extract The current pixel in the sea and sky scene image frame at any given moment. The image within a local window centered on a pixel. Based on the Facet model, according to the image Calculate pixels Second derivative at , , And calculate the pixel points Along the direction of the sea-line at the current local window scale Reference curvature ;in, yes The direction of the horizon in the seascape image frame at any given moment The angle with the horizontal direction of the image; Calculate the pixel points separately In the direction of the sea-line at multiple different local window scales Reference curvature on , to pixels Reference curvature at all local window scales The maximum value is used as the pixel point In the direction of the horizon directional curvature on And obtain the directional curvature spectrum.

5. The method for detecting sea level according to claim 1, characterized in that, Camera coordinate system With the camera's optical center as the origin , The optical axis aligns with the axial direction and points in the imaging direction. The axis is parallel to the horizontal direction of the image plane. The axes are parallel to the perpendicular direction of the image plane and form a right-handed coordinate system; the world coordinate system. of , , The shafts are respectively with Camera coordinate system at time , , Axis coincidence; determine the obtained The relative attitude relationship between the camera coordinate system and the sea level at any given time includes... The camera coordinate system at any given moment revolves around the world coordinate system. Euler angles of rotation of axis ;based on The relative attitude relationship between the camera coordinate system and the sea level at any given time is determined. The direction of the horizon in the sea-sky scene image frame at any given time includes: Sure The direction of the horizon in the seascape image frame at any given moment The angle between the pixel coordinate system and the u-axis satisfy , For camera intrinsic parameters, the pixel coordinate system takes the top left corner of the sea and sky scene image frame as the origin, with the u-axis pointing horizontally to the right and the v-axis pointing vertically downwards.

6. The method for detecting sea level according to claim 1, characterized in that, according to Estimation of measured motion data at time points The relative attitude relationship between the camera coordinate system and the sea level at any given time includes: Theoretical acceleration in world coordinate system Perform coordinate transformation to obtain Ideal acceleration in the coordinate system of the motion sensing unit at all times ;in, It is a rotation transformation model from the camera coordinate system to the motion sensing unit coordinate system. Is the world coordinate system to The rotation transformation model of the camera coordinate system at time t and , It is about the world coordinate system Axis rotation Obtain the rotation matrix of the intermediate transition coordinate system. It is around the intermediate transition coordinate system Axis rotation rotation matrix; Solve the objective equation of the LM nonlinear optimization algorithm get Euler angles of time and To determine The relative attitude relationship between the camera coordinate system and the sea level at any given time, where, It is the motion sensing unit in The measured acceleration at any given moment.

7. The method for detecting sea level according to claim 1, characterized in that, according to Estimation of measured motion data at time points Camera pose transformation matrix at time step include: Build Measured angular velocity at time t The corresponding pure quaternion And the fourth-order Runge-Kutta method was used to estimate Quaternion of time , , , , ;in, To represent quaternion multiplication, It is the time step. yes Quaternion at time; For the estimated quaternions Normalization processing Then, determine the quaternion. real part and the virtual part And calculate the rotation matrix. As a motion sensing unit Attention transformation matrix at time step ; Combining the rotation transformation model from the camera coordinate system to the motion sensing unit coordinate system ,get Camera pose transformation matrix at time step .

8. The method for detecting sea level according to claim 1, characterized in that, according to Camera pose transformation matrix at time step and The position of the horizon at any given time predict Predicted position of the sea-line at a given time include: Prediction using homography matrix Predicted position of the horizon in the seascape image frame at any given time ;in, This is the intrinsic parameter matrix of the camera. , , For camera intrinsic parameters, image principal point It is the intersection of the camera's optical axis and the image frame of the sea and sky scene; yes The scale factor corresponding to the time. yes The scale factor corresponding to the time.

9. The method for detecting sea level according to claim 1, characterized in that, The sea level detection method also includes: Using a camera to acquire Extracting images of the sea and sky scene at any given moment. The image frame passing through the principal point of the sea and sky scene at a certain moment. And the pixels along the straight line in the horizontal direction of the image plane are obtained The position of the horizon at any given time Image principal point It is the intersection of the camera's optical axis and the image frame of the sea and sky scene.

10. The method for detecting sea level according to claim 1, characterized in that, Based on the observation results of the sea-sky line Predicted location of the sea-line The error of the camera pose transformation matrix Iterative optimization until the iteration termination condition is met includes: Calculate the results of sea-sky observations Predicted location of the sea-line The deviation between them is used as Single-frame observation error at time ; The overall objective function is calculated as follows: ,in, Represents the single-frame observation error across all frames. The sum of This represents the attitude error correction amount for all frames. The sum of squares, It is the regularization coefficient. This indicates taking the minimum value; The overall objective function is determined by solving the nonlinear least squares method. Attitude error correction at time step The attitude error correction amount Superimposed on the camera pose transformation matrix Up, and utilize the updated camera pose transformation matrix Continue iterating until the iteration termination condition is met.

Images