Ship camera attitude correction method based on multi-camera cooperation

Abstract

The invention discloses a ship camera attitude correction method based on multi-camera cooperation, and relates to the technical field of ship navigation perception, and the method comprises the steps: carrying a plurality of cameras facing a bow, a stern, a port and a starboard in the front, rear, left and right directions of a compass deck; acquiring environment image sequences in front of the ship, behind the ship, in the left side direction and in the right side direction of the ship by using each carried camera; using a sea-sky-line identification method to obtain sea-sky-line characteristics of each frame of environment image; acquiring an inertial navigation data sequence by using a carried inertial measurement unit, and fusing the inertial navigation data sequence with the attitude correction sequence of the camera to obtain a camera attitude sequence at the current moment; and adjusting the camera attitude according to the camera attitude sequence. According to the method, dynamic, high-precision and omnidirectional camera attitude correction is realized based on a multi-camera cooperation mechanism, and the robustness of a ship sensing system is effectively improved.

Description

Technical Field

[0001] This application relates to the field of ship navigation perception technology, and in particular to a ship camera attitude correction method based on multi-camera collaboration. Background Technology

[0002] With increasing economic globalization and growing international trade, the demand for cargo transportation is rising, and the importance of the shipping industry is constantly increasing. However, the development of the shipping industry has also led to a series of problems, such as rising labor costs and reduced safety during ship navigation. To address these issues, the concept of intelligent ships has emerged. The "Intelligent Ship Standard 2025" clearly defines the technical means, functions, purposes, and significance of intelligent ships. Intelligent ships utilize modern technologies such as communication and the internet to comprehensively perceive their own and their surrounding environment's information and data, achieving autonomous or even intelligent navigation to achieve safer, more economical, and more environmentally friendly goals. This is also an inevitable trend in the future development of ships.

[0003] During intelligent ship navigation, shipborne cameras, as core devices for environmental perception, target recognition, and safety monitoring, directly impact the reliability of the perceived data due to their attitude stability. Attitude deviations can lead to target misidentification and sea-line positioning errors, consequently causing errors in the intelligent ship's navigation decision-making and threatening navigational safety. Therefore, intelligent correction of ship camera attitude is of great significance to the development of intelligent ships.

[0004] Traditional ship camera attitude correction methods primarily rely on a single sensor or manual adjustment. However, these methods have significant limitations: First, they depend on a single sensor. For example, when estimating camera attitude solely through an inertial measurement unit (IMU), the integration error accumulates significantly over long periods, resulting in large errors in long-haul ocean scenarios and failing to meet high-precision sensing requirements. Second, manual calibration and single-camera vision methods are poorly adaptable. Manual calibration requires system shutdown and cannot cope with dynamic attitude changes caused by wave turbulence and wind disturbances. Furthermore, existing monocular or binocular camera sea-line detection methods, due to their limited field of view (typically ≤120°), cannot cover the ship's omnidirectional environment, leading to attitude correction failure. Summary of the Invention

[0005] To address the aforementioned problems and technical requirements, this application proposes a method for correcting the attitude of a ship's camera based on multi-camera collaboration. The technical solution of this application is as follows: A method for correcting the attitude of ship cameras based on multi-camera collaboration is characterized in that several cameras are mounted on the compass deck of the ship in the forward, aft, port, and starboard directions, respectively, facing the bow, stern, port side, and starboard side, and the viewing angles of the multiple cameras cover the omnidirectional environment around the ship. The method for correcting the attitude of ship cameras includes: The system uses its onboard cameras to acquire environmental image sequences in front of the ship, behind the ship, to the left of the ship, and to the right of the ship. Each environmental image sequence includes n environmental images within a predetermined time period T before the current moment. The n environmental images of any two environmental image sequences correspond one-to-one. Each environmental image contains the sea surface and the sky. The sea-sky line recognition method is used to obtain the sea-sky line features of each environmental image. For any camera, the attitude correction sequence of the camera is determined based on the sea-line features of each frame of the environmental image sequence of each camera whose orientation is 90 degrees different from that of the camera, and in combination with the sea-line features of each frame of the environmental image sequence of the camera. The attitude correction sequence includes n attitude corrections, and each attitude correction corresponds to each frame of the environmental image sequence of the camera. The inertial measurement unit is used to acquire inertial navigation data sequence, which includes m ship attitude parameters within the current time and the previous predetermined time T. The inertial navigation data sequence is fused with the camera attitude correction sequence to obtain the camera attitude sequence at the current time, which includes n camera attitude parameters within the current time and the previous predetermined time T, where m and n are integer parameters and n≤m. Adjust the camera's attitude according to the current camera attitude sequence for the current moment and the subsequent predetermined duration T.

[0006] The further technical solution involves determining the camera's attitude correction sequence, including: For any frame of the environment image sequence from the camera According to environmental images The sea-line characteristics determine the camera's baseline attitude value. Based on environmental images The sea-line features of the environmental images of each camera, which are 90 degrees different from the camera's orientation, are used to determine the camera attitude offset value. ; Set the camera pose base value With camera attitude offset value By fusion, an environmental image is obtained. Corresponding camera attitude correction amount ; The camera pose correction values ​​corresponding to each frame of the environment image are arranged in the order of each frame of the environment image sequence to obtain the camera pose correction value sequence.

[0007] The further technical solution is that the camera attitude parameters include roll angle and pitch angle, and the sea-line characteristics include sea-line slope; Based on environmental images The sea-line features of the environmental images of each camera, which are 90 degrees different from the camera's orientation, are used to determine the camera attitude offset value. ,in, It is an environmental image. Corresponding to the l The horizon slope of the environmental image of each camera, where P is the total number of cameras whose orientation differs from the camera's orientation by 90 degrees. It means to reverse the direction.

[0008] A further technical solution involves setting the camera's basic pose values... With camera attitude offset value By fusion, an environmental image is obtained. Corresponding camera attitude correction amount , These are weighting parameters, and the farther the camera is from the average distance of all cameras to its left and right, the higher the weighting parameter. The larger.

[0009] A further technical solution involves fusing the inertial navigation data sequence with the camera attitude correction sequence to obtain the camera attitude sequence at the current moment, including: Determine the error function Z between the inertial navigation data sequence and the attitude correction sequence. Under the constraint that each attitude correction in the attitude correction sequence has a unique match in the inertial navigation data sequence, use an optimization algorithm to determine the matching matrix that minimizes the error function Z. The matching matrix indicates the matching relationship between the attitude correction and the inertial navigation data. Based on the matching matrix, each attitude correction value in the attitude correction value sequence is fused with its matched inertial navigation data to obtain the camera attitude sequence at the current moment.

[0010] Its further technical solution is an error function. ,in, It is the i-th attitude correction in the attitude correction sequence. With the j-th inertial navigation data in the inertial navigation data sequence absolute error and , It is the attitude correction amount in the matching matrix. With inertial navigation data The matching value, and and Matching , and If not matched .

[0011] A further technical solution involves obtaining the camera pose sequence at the current moment, including: For any i-th attitude correction in the attitude correction sequence Determine the attitude correction amount in the inertial navigation data sequence. Matched inertial navigation data ; Adjust the attitude amount With inertial navigation data By fusing, the i-th camera pose parameter in the camera pose sequence is obtained. , It is a fusion weight; The camera attitude parameters obtained by fusing the various attitude corrections in the attitude correction sequence are arranged in the order of the attitude corrections in the attitude correction sequence to obtain the camera attitude sequence at the current time.

[0012] Further technical solutions include: the ship camera attitude correction method also includes: When the deviation between N consecutive camera attitude parameters and their corresponding inertial navigation data in the camera attitude sequence exceeds a predetermined threshold, the camera's detection frequency is increased by a predetermined amount.

[0013] A further technical solution involves using a sea-line recognition method to acquire any frame of the environmental image. The sea-line features include: Environmental images The input is fed into a pre-trained CNN model to segment the sky and sea areas, resulting in a binarized segmented image. An edge detection algorithm is used to extract several candidate edges from the binarized segmented image, and the gray-level statistical features of each candidate edge are extracted. The gray-level statistical features indicate the gray-level distribution of the candidate edges. The sea-line feature is obtained by fitting all candidate edges whose gray-level statistical features meet the gray-level distribution requirements using the straight-line fitting method.

[0014] A further technical solution involves mounting the ship camera attitude correction method on a host computer that deploys a ROS2 system. The ship camera attitude correction method also includes: Each camera on board the ship receives GPS time data from the ship's intranet and publishes a unified time synchronization topic using ROS2 topics, defining a unified timestamp for the environmental images acquired by each camera.

[0015] The beneficial technical effects of this application are: This application discloses a method for ship camera attitude correction based on multi-camera collaboration. A panoramic camera system is constructed by mounting cameras at the bow, aft, port, and starboard directions on the ship's compass deck. A panoramic camera collaboration architecture is established, and environmental images from multiple cameras are used to correct the attitude of each camera from a visual perspective. Furthermore, the accuracy of camera attitude correction is further improved by fusing inertial navigation data with the visually corrected camera attitude. This method, based on a panoramic camera collaboration architecture and multi-source data fusion, achieves high-precision, omnidirectional, and dynamic correction of ship camera attitude. It avoids the long-term cumulative error problem of traditional single-sensor methods, effectively improving the robustness of camera attitude correction. Simultaneously, it enhances the reliability of perception under complex sea conditions, making it particularly suitable for the camera attitude stability requirements during long-duration voyages.

[0016] Based on the panoramic camera collaborative architecture, a master-slave camera collaborative mechanism is designed to ensure the global uniformity of camera attitude parameters, reduce data synchronization errors among cameras, and effectively solve the problem of conflicting correction directions among cameras, thereby improving the overall consistency of omnidirectional environmental perception. Furthermore, based on the difference in perspective between the slave and master cameras, the attitude parameters of the master camera are corrected using the sea-line features in the environmental image from the slave camera. This allows for a more accurate determination of the master camera's attitude parameter correction. Furthermore, fusing inertial navigation data further enhances the accuracy and reliability of the camera attitude parameters.

[0017] Furthermore, since the data frequency of the IMU is often higher than that of the camera, traditional interpolation methods for data alignment can lead to distortion of the inertial navigation data, resulting in errors compared to the actual situation. To address this issue, the method in this application matches the inertial navigation data with the visually corrected camera pose, achieving more accurate data alignment and laying an important foundation for subsequent data fusion to obtain accurate camera pose. Attached Figure Description

[0018] Figure 1 This is a schematic diagram showing the installation location of the ship's camera.

[0019] Figure 2 This is a flowchart of a method for correcting the attitude of a ship's camera.

[0020] Figure 3 This is an example of a master-slave camera collaboration mechanism architecture for a panoramic camera.

[0021] Figure 4 This is a schematic diagram illustrating the calculation of attitude parameters between the master and slave cameras. Detailed Implementation

[0022] The specific embodiments of this application will be further described below with reference to the accompanying drawings.

[0023] The method for correcting the attitude of a ship camera based on multi-camera collaboration disclosed in this application is applied to a ship equipped with multiple cameras. The ship has several cameras mounted on the compass deck in the forward, aft, left, and right directions, facing the bow, stern, port side, and starboard side, respectively. The field of view of the multiple cameras covers the omnidirectional environment around the ship, and all the cameras constitute a panoramic camera.

[0024] The compass deck of a ship is a dedicated deck located atop the superstructure. Its core function is to provide a wide field of view and a stable mounting platform for navigation equipment, ensuring that instruments such as the compass are not obstructed by the hull structure and can accurately determine the ship's course. Cameras can be positioned on the compass deck along the four sides of the railings of the equipment room platform, or on the four sides below the compass deck, ideally at the center of each side. However, if other structures exist on the sides, they can be positioned slightly off-center, as long as they do not deviate too far from the transverse or longitudinal axes of the compass deck, to ensure that environmental images around the ship can be captured. To achieve omnidirectional coverage, at least one camera is installed in each direction. Each camera can be composed of two fisheye cameras (field of view 150°-180°). The camera at the front captures environmental images ahead of the ship, the camera at the rear captures environmental images behind the ship, the camera on the left captures environmental images on the left side, and the camera on the right captures environmental images on the right side. A schematic diagram of the camera arrangement is shown in the example below. Figure 1 As shown, Figure 1 (a) is a side view. Figure 1 (b) is a top view.

[0025] In addition, the ship is also equipped with an inertial measurement unit (IMU). Furthermore, the ships addressed in this application also include conventional components of traditional ships, which will not be described in detail in this embodiment.

[0026] Based on the panoramic camera architecture composed of the aforementioned multiple cameras, please refer to the ship camera attitude correction method in this application. Figure 2 The flowchart shown illustrates the specific steps of this method as follows: Step 1: Use the onboard cameras to acquire environmental image sequences in front of the ship, behind the ship, to the left of the ship, and to the right of the ship. Each environmental image sequence includes n environmental images within a predetermined time period T before the current moment. The n environmental images of any two environmental image sequences correspond one-to-one. Each environmental image contains the sea surface and the sky.

[0027] To ensure that each frame of the environmental image sequence acquired by each camera corresponds one-to-one, data synchronization between the cameras is required, including time synchronization and frequency synchronization. The method in this application is implemented on a host computer with a ROS2 system. In one embodiment, the specific method for time synchronization is as follows: each camera on the ship receives its own GPS time data via the ship's intranet and publishes a unified time synchronization topic using ROS2 topics, defining a unified timestamp for each environmental image acquired by the camera. Similarly, the communication protocol is unified to the ROS2 topic format to ensure consistent frequencies across the cameras. After data synchronization, each frame of any two environmental image sequences corresponds one-to-one, meaning that the timestamps of each frame are the same. For details on the ROS2 topic mechanism, please refer to existing technologies; this application will not elaborate further.

[0028] Step 2: Use the sea-line recognition method to obtain the sea-line features of each frame of environmental image.

[0029] Due to the complex and ever-changing navigation environment of ships, traditional edge detection methods have low robustness in detecting sea-line terrain under complex environments and are prone to failure under complex weather conditions such as strong light, fog, and waves. To improve the robustness of sea-line detection, this application employs a hybrid algorithm combining a CNN model and a traditional edge detection algorithm. Specifically, it uses a sea-line recognition method to acquire any frame of the environmental image. The sea-line features include: (1) Environmental images The input is fed into a pre-trained CNN model to segment the sky and sea areas, followed by image binarization to obtain a binarized segmented image. The CNN model employs a conventional CNN architecture such as FCN (Fully Convolutional Network) and is trained offline. For specific model training methods, please refer to existing techniques. The training dataset is constructed based on historical environmental image data of ship navigation.

[0030] (2) Use the edge detection algorithm to extract several candidate edges in the binarized segmented image, and extract the gray-level statistical features of each candidate edge. The gray-level statistical features indicate the gray-level distribution of the candidate edge.

[0031] Since the sea-line line appears as a straight line in environmental images, edge detection can be performed on the segmented binary image. However, due to complex weather conditions, environmental images often contain false edges caused by noise interference. Edge detection algorithms extract several candidate edges from the binary segmented image, which include both true sea-line line edges and false edges. Therefore, it is necessary to further filter out false edges caused by fog, reflections, etc.

[0032] Considering that the gray-level gradient of a real sea-line horizon is continuous and consistent in direction, with a moderate gray-level variance, while the gray-level gradient of a false edge caused by fog is uniform but has a small variation, resulting in a small overall gray-level variance, and the gray-level gradient of a false edge caused by reflection exhibits a sudden increase in gray-level value and a chaotic gray-level gradient direction, this application further distinguishes false edges from real sea-line horizons by combining the gray-level statistical characteristics of each candidate edge. By statistically analyzing the gray-level values ​​of each candidate edge, gray-level statistical characteristics are obtained. When the gray-level statistical characteristics of a candidate edge meet the gray-level distribution requirements, the candidate edge is determined to be a real sea-line horizon. The specific method for statistically analyzing the gray-level values ​​of candidate edges can employ existing techniques, including calculating the mean, variance, and gradient of the gray-level values ​​of the candidate edge. The gray-level distribution requirements can be customized based on experience gained from historical data. For example, when 80% of the pixels on a candidate edge have a gray-level gradient direction along the horizontal direction, and the gray-level variance is within the range of 50 to 200, the gray-level distribution requirements of the sea-line horizon are met.

[0033] (3) Finally, the linear fitting method is used to fit all candidate edges whose gray-level statistical features meet the gray-level distribution requirements to obtain the sea-line feature, which includes the position and slope of the sea-line. Any existing linear fitting method can be used, and this application does not impose any restrictions.

[0034] Step 3: During ship navigation, the attitude parameters of each camera must be monitored in real time and continuously corrected to ensure the stable operation of each camera, thereby ensuring the accuracy and effectiveness of the acquired environmental images. The attitude correction of each camera can be performed using the method of this application. For any given camera, based on the sea-line characteristics of each frame of the environmental image sequence from cameras whose orientation differs from that camera by 90 degrees, and in combination with the sea-line characteristics of each frame of the environmental image sequence from that camera, the attitude correction sequence of that camera is determined. The attitude correction sequence includes n attitude correction values, and each attitude correction value corresponds to each frame of the environmental image sequence from that camera.

[0035] Based on a panoramic camera architecture, a master-slave camera collaboration mechanism is constructed. Any camera in the panoramic camera system can act as the master camera, and all cameras whose orientation differs from the master camera by 90 degrees are slave cameras. The master-slave collaboration architecture is as follows: Figure 3As shown, the sequence of environmental images acquired by the camera facing the bow constitutes the bow video stream, the sequence of environmental images acquired by the camera facing the stern constitutes the stern video stream, the sequence of environmental images acquired by the camera facing the port side constitutes the port side video stream, and the sequence of environmental images acquired by the camera facing the starboard side constitutes the starboard side video stream. When correcting the attitude of the camera facing the bow, the bow video stream is the primary stream, while the port and starboard side video streams are secondary streams. The master camera is responsible for receiving data from all slave cameras, including environmental images acquired by the slave cameras and their extrinsic parameters (installation position, installation height, installation angle, etc.). The slave cameras acquire environmental images in real time and upload the sea-line features to the master camera, thereby achieving camera attitude correction of the master camera at the visual level using a master-slave collaborative mechanism.

[0036] In one embodiment, determining the camera's pose correction sequence includes: For any frame of the environmental image sequence of the camera According to environmental images The sea-line characteristics determine the camera's baseline attitude value. Based on environmental images The sea-line features of the environmental images of each camera whose orientation differs from the camera's by 90 degrees are used to determine the camera attitude offset value. ; Set the camera pose base value With camera attitude offset value By fusion, an environmental image is obtained. Corresponding camera attitude correction amount .

[0037] Because the shooting perspectives of the main camera and the slave cameras differ, the calculation errors for the attitude parameters of different cameras vary. For example, when the main camera is facing the bow, it captures images of the ship's forward direction. The ship's left and right tilt can be seen from the main camera's viewpoint; that is, the sea-line feature of the main camera indicates the roll angle. The roll angle calculated using the environmental image from the main camera is relatively accurate. When calculating the pitch angle using the environmental image from the main camera, the vertical pixel difference between the sea-line position under normal heading conditions and the sea-line position after attitude change needs to be used for angle calculation. For example, a 10-pixel upward shift of the sea-line might mean a 10-pixel shift at the midpoint of the sea-line, with unequal shifts on the left and right sides, making it impossible to set a reference point. Furthermore, sea-line recognition itself has a certain error; the cumulative error from two sea-line recognitions from the same perspective, plus pixel estimation and the percentage of pixel change, results in a high error when calculating the pitch angle using the environmental image from the main camera. However, the sea-line features of the environmental images from slave cameras facing the port and starboard sides of the ship can indicate the pitch angle, so the pitch angle of the main camera can be corrected using the viewpoint of the slave cameras.

[0038] Camera attitude parameters include roll angle and pitch angle, and sea-line characteristics include sea-line slope, which can be used to calculate roll angle and pitch angle.

[0039] Specifically, based on environmental images The camera attitude offset value of the camera is determined by analyzing the sea-line features of the environmental images from each camera (i.e., slave cameras) whose orientation differs from the main camera's by 90 degrees. ,in, It is an environmental image. Corresponding to the l The horizon slope of the environmental image of a camera, where P is the total number of cameras whose orientation differs from that camera by 90 degrees. It means to reverse the direction.

[0040] Further adjust the camera pose baseline values With camera attitude offset value By fusion, an environmental image is obtained. Corresponding camera attitude correction amount , These are weighting parameters, and the farther the camera is from the average distance of all cameras to its left and right, the higher the weighting parameter. The larger.

[0041] The required roll and pitch angle corrections vary depending on the camera. Taking a single camera installed in each orientation as an example, there are two main scenarios: (1) When the cameras facing the bow and the stern are used as the main cameras, i.e., when the objects to be corrected are the cameras facing the bow and the stern, the roll angle is... The slope k of the sea-line in the environmental image of the main camera can be calculated. .

[0042] The basic pitch angle value is calculated based on the vertical distance from the center of the environmental image of the main camera to the horizon. , h It is the vertical distance from the center of the current environmental image to the sea-line. It is the vertical distance from the center of the environmental image to the horizon when the ship is horizontal. f This is the main camera's focal length. Please refer to it. Figure 4 Image 1 shows the environment when the ship is level (roll angle is 0), and image 2 shows the environment when the ship's roll angle is not 0. If only one camera is installed for each orientation, then there are two slave cameras used to correct the main camera, resulting in a pitch angle offset value. .

[0043] Because the pitch angle error calculated from the environmental image of the main camera is relatively large at this time, it is necessary to use the slope of the sea-line from the camera to correct it and obtain the pitch angle correction amount. Specifically, the basic pitch angle value... With pitch angle offset value By fusion, an environmental image is obtained. Corresponding pitch angle correction .

[0044] (2) When the cameras facing the port side and the starboard side of the ship are used as the main cameras, i.e., when the objects to be corrected are the cameras facing the port side and the starboard side, the pitch angle is... It can be calculated based on the vertical distance from the center of the environmental image of the main camera to the sea-line. .

[0045] The basic roll angle value is calculated based on the sea-line slope k of the camera environment image from the main camera. If only one camera is installed in each orientation, then there are two slave cameras used to correct the main camera, and the roll angle offset value... .

[0046] Because the roll angle calculated from the environmental image of the main camera has a large error at this time, it is necessary to use the horizon slope of the camera to correct for the roll angle correction. Specifically, the basic roll elevation angle value is... With roll angle offset value By fusion, an environmental image is obtained. Corresponding roll angle correction .

[0047] Among them, weight parameters The settings need to be adjusted based on the distance and installation angle between the slave and master cameras. The greater the distance and angle difference between the slave and master cameras, the greater the difference in the overlapping scene images. This indicates a larger difference in the field of view and pixel width between the slave and master cameras. In this case, the reference value of the slave camera to the master camera decreases, so the weight of the camera pose offset value corresponding to the slave camera should be reduced accordingly. The weight parameters... The specific value can be set based on accumulated experimental experience and the actual ship's own equipment parameters.

[0048] Based on the timestamps of the camera's environmental image sequence, each frame of the main camera's environmental image corresponds one-to-one with each frame of the secondary camera's environmental image. The camera attitude correction is calculated based on the sea-line characteristics of the environmental images from both the main and secondary cameras, as well as their camera parameters. Therefore, each frame's timestamp corresponds to a camera attitude correction. The camera attitude corrections corresponding to each frame are arranged in the order of the environmental images in the camera's environmental image sequence to obtain the camera's attitude correction sequence. This ensures that the timestamps of the attitude correction sequence are consistent with the environmental image sequence, and that the order of the attitude corrections in the sequence is accurate.

[0049] Step 4: Use the onboard inertial measurement unit (IMU) to acquire the inertial navigation data sequence. The inertial navigation data sequence includes m ship attitude parameters within the current time and the previous predetermined time period T. Then, fuse the inertial navigation data sequence with the camera attitude correction sequence to obtain the camera attitude sequence at the current time. The camera attitude sequence includes n camera attitude parameters within the current time and the previous predetermined time period T, where m and n are integer parameters and n≤m.

[0050] Traditional methods use a single IMU to acquire ship attitude. Since the ship's compass deck is a rigid body, the ship's attitude measured by the IMU can be directly used as the camera attitude. However, the IMU's attitude estimation is affected by integration errors, which accumulate over time and cause significant deviations. Therefore, this application analyzes environmental images captured by the camera to obtain attitude correction values ​​from a visual perspective and fuses them with inertial navigation data to improve the accuracy of camera attitude correction. However, since the frequency of IMU data acquisition is often higher than the video stream frame rate of the environmental images captured by the camera, direct frequency alignment can lead to errors. In one example, the IMU acquisition frequency is 50Hz, and the video stream frame rate is 15Hz. Unifying the frequencies through a topic published in ROS2 results in some IMU data distortion. For example, if one frame of environmental image corresponds to three sets of inertial navigation data, the program might use the first set of data when unifying the frequency, leading to errors compared to the actual situation. Therefore, it is necessary to match and fuse the attitude correction value sequence and the inertial navigation data sequence.

[0051] In one embodiment, fusing the inertial navigation data sequence with the camera's attitude correction sequence to obtain the camera attitude sequence at the current moment includes: (1) Determine the error function Z between the inertial navigation data sequence and the attitude correction sequence. Under the constraint that each attitude correction in the attitude correction sequence has a unique match in the inertial navigation data sequence, use an optimization algorithm to determine the matching matrix that minimizes the error function Z. The matching matrix indicates the matching relationship between the attitude correction and the inertial navigation data.

[0052] At the same time, the camera pose obtained from visual analysis and the camera pose estimated by the IMU should be consistent. Based on this principle, it is necessary to ensure that the data closest to the attitude correction is matched in the inertial navigation data sequence. Moreover, to obtain the most accurate matching data, the core constraint during matching is to ensure that each attitude correction corresponds one-to-one with an inertial navigation data point, that is, each attitude correction has a unique matching inertial navigation data point in the sequence. Then, an optimization algorithm is used to solve the error function Z under the constraints to obtain the matching matrix D. Each element of the matching matrix D corresponds to the matching relationship between two data points. For example, if the attitude correction sequence is used as the row and the inertial navigation data sequence is used as the column, the size of the calculated matching matrix D is n×m. Any element of the matching matrix... This represents the matching relationship between the i-th attitude correction data in the attitude correction sequence and the j-th inertial navigation data in the inertial navigation data sequence.

[0053] In one embodiment, an error function is designed based on an optimization objective of matching the data in the inertial navigation data sequence that is closest to the attitude correction. ,in, It is the i-th attitude correction in the attitude correction sequence. With the j-th inertial navigation data in the inertial navigation data sequence absolute error and , It is the attitude correction amount in the matching matrix. With inertial navigation data The matching value, and and Matching , and If not matched , , When roll angle correction is required, the attitude correction amount... Corresponding roll angle correction, inertial navigation data It is the roll angle measured by the IMU; when pitch correction is needed, the attitude correction amount is... Corresponding pitch angle correction, inertial navigation data The pitch angle is measured by the IMU. For the specific calculation of the roll and pitch angle corrections, please refer to the example in step 3.

[0054] The error function Z is the sum of errors for all pairings. Minimizing Z achieves the goal of a one-to-one match where the overall size is closest. The optimization algorithm can employ existing techniques, such as the least squares method or the Hungarian algorithm.

[0055] (2) Based on the matching matrix, the attitude correction values ​​in the attitude correction value sequence are fused with their matching inertial navigation data to obtain the camera attitude sequence at the current time.

[0056] Specifically, obtaining the camera pose sequence at the current moment includes: For any i-th attitude correction in the attitude correction sequence Determine the attitude correction amount in the inertial navigation data sequence. Matched inertial navigation data ; Adjust the attitude amount With inertial navigation data By fusing, the i-th camera pose parameter in the camera pose sequence is obtained. , It is a fusion weight. The specific values ​​are set based on experimental experience; the camera attitude parameters obtained by fusing the various attitude correction values ​​in the attitude correction value sequence are arranged in the order of the attitude correction values ​​in the attitude correction value sequence to obtain the current camera attitude sequence, so as to ensure that the timestamp of the camera attitude sequence is consistent with the attitude correction value sequence and the order of the camera attitude parameters in the camera attitude sequence is accurate.

[0057] Step 5: Adjust the camera attitude for the current time and the subsequent predetermined duration T according to the camera attitude sequence at the current time.

[0058] It should be noted that because the camera attitude sequence corresponds to a predetermined duration T, the camera attitude can be adjusted according to the camera attitude parameters at each timestamp in the sequence, both at the current moment and for the subsequent predetermined duration T. Since the timestamps are corresponding, the camera attitude parameters are accurate, thus ensuring the continuity of camera attitude adjustment along the timeline. The specific value of the predetermined duration T is determined through experimental analysis based on actual application conditions to ensure the accuracy of the camera attitude parameters.

[0059] Camera attitude adjustment methods can be implemented by driving the gimbal motor to adjust the camera's roll and pitch angles in real time; alternatively, software methods can be used to adjust the camera's attitude based on the environmental images captured by the camera, such as driving a host computer to perform real-time cropping of the environmental image to correct image shifts caused by changes in the camera's roll and pitch angles. For specific adjustment processes, please refer to existing technologies; this application will not elaborate further.

[0060] Furthermore, while adjusting the camera attitude in real time, it is also necessary to analyze the attitude correction status of each camera in real time. Based on the changes in camera attitude before and after correction, it is determined whether the camera detection frequency needs to be adjusted to adapt to the camera attitude correction requirements.

[0061] In one embodiment, the ship camera attitude correction method further includes: when the deviation between N consecutive camera attitude parameters and their corresponding inertial navigation data in the camera attitude sequence exceeds a predetermined threshold, increasing the detection frequency of the camera by a predetermined amount.

[0062] When the corrected camera attitude parameters differ significantly from the attitude parameters in the inertial navigation data before correction, it indicates that the current detection frequency of the camera cannot meet the requirements for high-precision attitude correction. Increasing the detection frequency can increase the image sampling density of the camera, enabling more timely and accurate capture of dynamic attitude changes, thereby optimizing the correction effect, reducing correction deviation, and ensuring the reliability of the camera's perception data and the consistency of omnidirectional environmental perception. The specific values ​​of the consecutive number N, the deviation threshold, and the frequency increase are set based on experimental experience. In this application, N=3, the deviation threshold is 1°, and the frequency increase is 10Hz.

[0063] This application achieves high-precision correction of camera attitude through a closed-loop mechanism of "multi-camera collaborative perception - multi-feature fusion detection - multi-source data calibration - dynamic optimization and adjustment", laying an important foundation for subsequent target recognition and other scenarios in real ship applications.

[0064] The above descriptions are merely preferred embodiments of this application, and this application is not limited to the above embodiments. It is understood that other improvements and variations that can be directly derived or conceived by those skilled in the art without departing from the spirit and concept of this application should be considered to be included within the protection scope of this application.

Claims

1. A method for correcting the attitude of a ship's camera based on multi-camera collaboration, characterized in that, The ship's compass deck is equipped with several cameras facing the bow, stern, port, and starboard directions, respectively, with the multiple cameras providing omnidirectional coverage of the ship's surroundings. The ship's camera attitude correction method includes: The system uses its onboard cameras to acquire environmental image sequences in front of the ship, behind the ship, to the left of the ship, and to the right of the ship. Each environmental image sequence includes n environmental images within a predetermined time period T before the current moment. The n environmental images of any two environmental image sequences correspond one-to-one. Each environmental image contains the sea surface and the sky. The sea-sky line recognition method is used to obtain the sea-sky line features of each environmental image. For any camera, the attitude correction sequence of the camera is determined based on the sea-line features of each frame of the environmental image sequence of each camera whose orientation differs from that of the camera by 90 degrees, and in combination with the sea-line features of each frame of the environmental image sequence of the camera. The attitude correction sequence includes n attitude correction values, and each attitude correction value corresponds to each frame of the environmental image sequence of the camera. The inertial measurement unit is used to acquire an inertial navigation data sequence, which includes m ship attitude parameters within the current time and the previous predetermined time period T. The inertial navigation data sequence is fused with the attitude correction sequence of the camera to obtain the camera attitude sequence at the current time. The camera attitude sequence includes n camera attitude parameters within the current time and the previous predetermined time period T, where m and n are integer parameters and n≤m. Adjust the camera's attitude according to the current camera attitude sequence for the current moment and the subsequent predetermined duration T.

2. The ship camera attitude correction method according to claim 1, characterized in that, Determining the camera's attitude correction sequence includes: For any frame of the environmental image sequence of the camera According to the environmental image The sea-line features determine the camera's base attitude value. According to the environmental image The sea-line features of the environmental images of each camera, whose orientation differs from the camera's by 90 degrees, are used to determine the camera attitude offset value of the camera. ; Set the camera pose base value With camera attitude offset value The environmental image is obtained by fusion. Corresponding camera attitude correction amount ; The camera pose correction values ​​corresponding to each frame of the environment image are arranged in the order of each frame of the environment image sequence to obtain the camera pose correction value sequence.

3. The ship camera attitude correction method according to claim 2, characterized in that, The camera attitude parameters include roll angle and pitch angle, and the sea-line feature includes sea-line slope; According to the environmental image The sea-line features of the environmental images of each camera, whose orientation differs from the camera's by 90 degrees, are used to determine the camera attitude offset value of the camera. ,in, The environmental image Corresponding to the l The horizon slope of the environmental image of each camera, where P is the total number of cameras whose orientation differs from the orientation of the stated camera by 90 degrees. It means to reverse the direction.

4. The ship camera attitude correction method according to claim 2, characterized in that, Camera pose baseline values With camera attitude offset value The environmental image is obtained by fusion. Corresponding camera attitude correction amount , These are weighting parameters, and the farther the camera is from the average distance of all cameras to its left and right, the higher the weighting parameter. The larger.

5. The method for correcting the attitude of a ship's camera according to claim 1, characterized in that, The inertial navigation data sequence is fused with the camera attitude correction sequence to obtain the camera attitude sequence at the current moment, which includes: Determine the error function Z between the inertial navigation data sequence and the attitude correction sequence. Under the constraint that each attitude correction in the attitude correction sequence has a unique match in the inertial navigation data sequence, use an optimization algorithm to determine the matching matrix that minimizes the error function Z. The matching matrix indicates the matching relationship between the attitude correction and the inertial navigation data. Based on the matching matrix, each attitude correction value in the attitude correction value sequence is fused with its matched inertial navigation data to obtain the camera attitude sequence at the current moment.

6. The ship camera attitude correction method according to claim 5, characterized in that, The error function ,in, It is the i-th attitude correction in the attitude correction sequence. With the j-th inertial navigation data in the inertial navigation data sequence absolute error and , It is the attitude correction amount in the matching matrix. With inertial navigation data The matching value, and and Matching , and If not matched .

7. The method for correcting the attitude of a ship's camera according to claim 5, characterized in that, The current camera pose sequence includes: For any i-th attitude correction in the attitude correction sequence Determine the attitude correction amount in the inertial navigation data sequence. Matched inertial navigation data ; Adjust the attitude amount With inertial navigation data By fusing, the i-th camera pose parameter in the camera pose sequence is obtained. , It is a fusion weight; The camera attitude parameters obtained by fusing the various attitude corrections in the attitude correction sequence are arranged in the order of the attitude corrections in the attitude correction sequence to obtain the camera attitude sequence at the current time.

8. The method for correcting the attitude of a ship's camera according to claim 1, characterized in that, The ship camera attitude correction method also includes: When the deviation between N consecutive camera attitude parameters and their corresponding inertial navigation data in the camera attitude sequence exceeds a predetermined threshold, the detection frequency of the camera is increased by a predetermined amount.

9. The method for correcting the attitude of a ship's camera according to claim 1, characterized in that, Obtain any frame of environmental image using the sea-line recognition method The sea-line features include: Environmental images The data is input into a pre-trained CNN model to segment the sky and sea areas, resulting in a binarized segmented image. An edge detection algorithm is used to extract several candidate edges from the binarized segmented image, and the gray-level statistical features of each candidate edge are extracted. The gray-level statistical features indicate the gray-level distribution of the candidate edge. The sea-line feature is obtained by fitting all candidate edges whose gray-level statistical features meet the gray-level distribution requirements using the straight-line fitting method.

10. The method for correcting the attitude of a ship's camera according to claim 1, characterized in that, The ship camera attitude correction method is mounted on a host computer with a ROS2 system deployed on it, and the ship camera attitude correction method also includes: Each camera on the ship receives GPS time data from the ship's intranet and publishes a unified time synchronization topic using ROS2 topics, defining a unified timestamp for the environmental images acquired by each camera.

Images