Panoramic image splicing method and system based on real-time attitude of ship body

Abstract

The invention relates to a panoramic image splicing method and system based on the real-time attitude of a ship body, and the method comprises the steps: synchronously collecting an original image, a rolling angle data and a pitch angle data through arranging a plurality of fisheye cameras in the circumferential direction of the ship body in combination with inertial navigation equipment; hardware-level clock synchronization is realized by using a second pulse signal of a global positioning system, and alignment of an image and attitude data is completed through a timestamp matching algorithm; the method comprises the following steps: acquiring internal and external parameters of a camera by adopting a Zhang's calibration method, performing de-preprocessing on an image, constructing a rotation matrix based on a real-time attitude, projecting pixel points to a stable world coordinate system, and generating a projection image; for an overlapping region between adjacent projection images, dynamically calculating a fusion weight according to an attitude angle change rate, and realizing adaptive weighted splicing; and finally, a seamless panoramic image is generated and coded and output to a display terminal and a storage device, panoramic image splicing in a ship body shaking scene is realized, and real-time performance, robustness and environmental adaptability are considered.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent ship environment perception, and particularly relates to a panoramic image stitching method and system based on real-time ship body posture. BACKGROUND

[0002] In the process of intelligent ship navigation and berthing, 360° panoramic vision coverage is the core requirement for realizing environment perception, route planning and safety monitoring, and is the core technical means for ensuring navigation safety, supporting autonomous navigation, assisting obstacle avoidance decision-making and improving operation efficiency.

[0003] Generating a panoramic view by image stitching through multiple cameras arranged around the ship body is the current mainstream solution. However, the traditional shipborne panoramic camera stitching method faces many technical bottlenecks in actual application, especially in dynamic sea conditions, which seriously affects the reliability and practicality of the system, and there are the following core problems: 1) posture perception deficiency leading to projection distortion: the ship body is affected by waves to produce roll (roll angle) and pitch (pitch angle), the traditional fixed projection model cannot adapt to the change of camera posture in real time, and the real-time posture change of the ship body is not taken into account in the projection transformation. When the ship body shakes, the actual posture of the camera has changed, and the fixed parameter projection model cannot adaptively adjust it, resulting in serious horizontal misplacement, image stretching or compression and other geometric distortions in the generated panoramic bird's eye view, which seriously affects the accuracy of visual perception. 2) poor data synchronization leading to inaccurate correction: the existing scheme does not realize high-precision time synchronization of the camera and the attitude sensor (IMU), and the two often work independently, with a time synchronization error of not less than five milliseconds, which leads to mismatch between attitude correction and image frame, and the attitude data lags behind the image data, resulting in "correction delay" phenomenon, which leads to decreased stitching accuracy. 3) contradiction between environmental adaptability and real-time performance: the existing scheme relies on image feature matching (such as SIFT, ORB) or deep learning model for stitching, which has high computational complexity, and the end-to-end processing delay is generally more than one hundred milliseconds, which cannot meet the millisecond-level response requirement of the ship in complex water areas for real-time obstacle avoidance or berthing operation, and is significantly affected by light, sea fog, etc. In complex environments such as sea fog and strong light, the features fail, the reliability is poor and the calculation time is long, which cannot meet the millisecond-level real-time requirement of ship navigation.

[0004] In summary, the existing shipborne panoramic vision system has systematic defects in posture perception deficiency, data synchronization lag, insufficient real-time performance and robustness, and cannot meet the demand for high-precision, low-latency and strong environmental adaptability of the panoramic vision perception of intelligent ships in complex sea conditions. Therefore, a new panoramic vision scheme that can perceive the ship body posture in real time, accurately synchronize multi-source data, and realize fast stitching without distortion is needed to overcome the above technical bottlenecks and improve the overall performance and reliability of the intelligent ship environment perception system. SUMMARY

[0005] In order to solve the problems of the current intelligent ship in the process of sailing and berthing, such as the distortion of panoramic image caused by the disturbance of ship body posture, the obvious splicing seam of multi-camera, the failure of traditional method relying on image feature matching in sea fog and strong light, and the insufficient real-time performance caused by large system response delay, the present application provides a panoramic image splicing method based on real-time posture of ship body, which can effectively eliminate the geometric distortion caused by ship body sway, realize high-precision time synchronization of image and posture data, and realize non-distortion panoramic image splicing in ship body sway scene, taking into account real-time performance and environmental robustness. The present application also relates to a panoramic image splicing system based on real-time posture of ship body.

[0006] The technical scheme of the present application is as follows:

[0007] A panoramic image splicing method based on real-time posture of ship body, characterized by comprising the following steps:

[0008] Original image and posture angle acquisition step: a plurality of fisheye cameras and inertial navigation equipment are installed on the deck of the ship body, the fisheye cameras are arranged circumferentially around the ship body, there is an overlapping area between the fields of view of adjacent fisheye cameras to realize 360° visual coverage of the environment around the ship body, the original images of the four sides of the ship body are collected in real time by each fisheye camera, and the posture angle data of the ship body are collected in real time by the inertial navigation equipment, the posture angle data including roll angle and pitch angle;

[0009] Image and posture angle data synchronization processing step: Zhang's calibration method is used to calibrate each fisheye camera to obtain the intrinsic matrix and extrinsic matrix of each fisheye camera; then the second pulse signal provided by the global positioning system is used to synchronize the hardware level clock of the fisheye camera and the inertial navigation equipment to obtain a unified time reference; based on the unified time reference, the timestamp matching algorithm is used to synchronize the data level time of the original image and the posture angle data, so that each frame of original image is associated with the posture angle data at the corresponding time;

[0010] Image preprocessing step: the original images collected by each fisheye camera are preprocessed to obtain preprocessed images; the preprocessing includes fisheye distortion correction based on the intrinsic matrix and noise filtering processing;

[0011] Posture correction and projection conversion step: based on the intrinsic matrix, the pixel coordinates in the preprocessed image are converted into three-dimensional directional vectors in the camera coordinate system; based on the roll angle and pitch angle at the corresponding time associated with the current frame of preprocessed image, a posture rotation matrix is constructed; according to the posture rotation matrix, the three-dimensional directional vectors in the camera coordinate system are converted to the stable world coordinate system with the ship body as the reference, and the three-dimensional directional vectors in the stable world coordinate system are projected to the two-dimensional panoramic image plane through the cylindrical surface projection model to generate the corrected projection image;

[0012] The panoramic splicing and fusion step: based on the external parameter matrix, an overlapping area between adjacent fisheye camera corresponding projection images is obtained, and based on the roll angle and the pitch angle at the current moment and the moment before the current moment, the roll angle change rate and the pitch angle change rate at the current moment are calculated respectively; according to the roll angle change rate and the pitch angle change rate, the pixel fusion weight of the overlapping area of each pair of adjacent projection images is dynamically calculated; then based on the dynamically calculated pixel fusion weight, each pixel value in the overlapping area is weighted and fused, and the pixel value of the corresponding projection image of the non-overlapping area other than the overlapping area is retained, and then a seamless 360° panoramic image is generated;

[0013] The result output step: the generated 360° panoramic image is encoded into a video stream, and the video stream is transmitted in real time to a display terminal of a shipborne bridge and a local storage device, and the splicing and video display of the panoramic image are completed.

[0014] Preferably, in the image preprocessing step, the preprocessing of the original image specifically includes:

[0015] First, the original image is subjected to fisheye distortion correction, which includes calculating the corrected coordinates (x', y') of the original image pixel coordinates (x, y) in the original image by using the fisheye distortion coefficient f in the intrinsic parameter matrix and through a correction formula, and then obtaining a corrected image; wherein the correction formula is as follows:

[0016]

[0017]

[0018] Then, the corrected image is subjected to noise filtering processing by using a Gaussian filtering algorithm to suppress image noise introduced by factors such as sea surface reflection and water mist, and finally a preprocessed image is output.

[0019] Preferably, in the attitude correction and projection conversion step, the cylindrical projection model is as follows:

[0020]

[0021]

[0022] wherein, is a three-dimensional direction vector in the stable world coordinate system; is the horizontal and vertical center coordinates of the two-dimensional panoramic image plane.

[0023] ​​​​​Preferably, in the image and attitude angle data synchronization processing step, the synchronization error of the data level time synchronization is less than or equal to 1 millisecond.

[0024] Preferably, in the result output step, the generated 360° panoramic image is encoded into a video stream, and is transmitted in real time to a display terminal in the shipboard cabin at a frame rate of thirty frames per second through the HDMI protocol; at the same time, the generated 360° panoramic image is encoded into a video file in the MPEG-4 format, and is stored in a local storage device at a code rate of eight megabits per second.

[0025] A panoramic image stitching system based on real-time attitude of a ship body, characterized by comprising, in sequence, an original image and attitude angle acquisition module, an image and attitude angle data synchronization processing module, an image preprocessing module, an attitude correction and projection conversion module, a panoramic stitching and fusion module, and a result output module,

[0026] The original image and attitude angle acquisition module is provided with a plurality of fisheye cameras and an inertial navigation device installed on the deck of the ship body, the fisheye cameras are arranged circumferentially around the ship body, there is an overlapping area between the fields of view of adjacent fisheye cameras to realize 360° visual coverage of the environment around the ship body, the original images of the surroundings of the ship body are acquired in real time by the fisheye cameras, and the attitude angle data of the ship body are acquired in real time by the inertial navigation device, the attitude angle data including a roll angle and a pitch angle;

[0027] The image and attitude angle data synchronization processing module calibrates each fisheye camera by Zhang's calibration method to obtain the intrinsic matrix and the extrinsic matrix of each fisheye camera, performs hardware level clock synchronization of the fisheye cameras and the inertial navigation device by a second pulse signal provided by a global positioning system to obtain a unified time reference, and performs data level time synchronization of the original images and the attitude angle data based on the unified time reference and by a timestamp matching algorithm, so that each frame of original image is associated with the attitude angle data at the corresponding time;

[0028] The image preprocessing module pre-processes the original images acquired by the fisheye cameras to obtain pre-processed images, the pre-processing including fisheye distortion correction based on the intrinsic matrix and noise filtering processing;

[0029] The attitude correction and projection conversion module converts the pixel coordinates in the pre-processed images into three-dimensional directional vectors in the camera coordinate system based on the intrinsic matrix, constructs an attitude rotation matrix based on the roll angle and the pitch angle at the corresponding time associated with the current frame of pre-processed image, converts the three-dimensional directional vectors in the camera coordinate system to a stable world coordinate system with the ship body as the reference according to the attitude rotation matrix, projects the three-dimensional directional vectors in the stable world coordinate system to a two-dimensional panoramic image plane through a cylindrical surface projection model, and generates a corrected projection image;

[0030] The panoramic stitching and fusion module obtains the overlapping area between the corresponding projected images of adjacent fisheye cameras based on the extrinsic parameter matrix, and calculates the roll angle change rate and pitch angle change rate at the current moment based on the roll angle and pitch angle at the previous moment. Based on the roll angle change rate and pitch angle change rate, the pixel fusion weight of the overlapping area of ​​each pair of adjacent projected images is dynamically calculated. Then, based on the dynamically calculated pixel fusion weight, each pixel value in the overlapping area is weighted and fused, and the pixel values ​​of the corresponding projected images of the non-overlapping areas are retained, thereby generating a seamless 360° panoramic image.

[0031] The output module encodes the generated 360° panoramic image into a video stream and transmits the video stream in real time to the display terminal in the ship's bridge and the local storage device to complete the stitching of the panoramic image and the display of the video.

[0032] Preferably, the image preprocessing module specifically includes the following steps for preprocessing the original image:

[0033] First, fisheye distortion correction is performed on the original image. Fisheye distortion correction includes correcting the pixel coordinates in the original image ( , Using the fisheye distortion coefficients in the intrinsic parameter matrix The corrected coordinates are obtained by calculating using the correction formula. , This process is repeated to obtain the corrected image; the correction formula is as follows:

[0034]

[0035]

[0036] The Gaussian filtering algorithm is then used to filter noise in the corrected image to suppress image noise introduced by factors such as sea surface reflection and water mist, and finally outputs the preprocessed image.

[0037] Preferably, in the attitude correction and projection conversion module, the cylindrical projection model is as follows:

[0038]

[0039]

[0040] in, To stabilize the three-dimensional direction vector in the world coordinate system; These are the horizontal and vertical center coordinates of the two-dimensional panoramic image plane.

[0041] Preferably, in the image and attitude angle data synchronization processing module, the synchronization error of the clock synchronization is less than or equal to 1 millisecond.

[0042] Preferably, in the result output module, the generated 360° panoramic image is encoded into a video stream and transmitted in real time to the display terminal in the ship's bridge via the HDMI protocol at a frame rate of thirty frames per second; at the same time, the generated 360° panoramic image is encoded into an MPEG-4 format video file and stored in a local storage device at a bit rate of eight megabits per second.

[0043] The technical effects of this invention are as follows:

[0044] This invention provides a panoramic image stitching method based on real-time ship attitude, also known as a shipborne panoramic camera stitching method based on real-time ship attitude. First, multiple fisheye cameras are arranged circumferentially on the ship's deck, ensuring appropriate overlap in the fields of view of adjacent cameras. This achieves complete 360° visual coverage of the surrounding environment, eliminating the blind spots of traditional single-camera systems and providing necessary spatial intersection for subsequent image stitching. It ensures that there are matching or fusionable common areas between adjacent images, improving stitching continuity and robustness. Simultaneously, an inertial navigation unit (IMU) collects real-time roll and pitch angle data of the ship to perceive its dynamic attitude changes under wave action. This provides crucial input for subsequent attitude-based image correction, avoiding image distortion caused by ship swaying. Based on this, Zhang's calibration method was used to accurately calibrate each fisheye camera, obtaining its intrinsic parameter matrix (including focal length, principal point coordinates, and distortion coefficients) and extrinsic parameter matrix (describing the spatial position and orientation of the camera in the ship's coordinate system). This provided a high-precision geometric basis for subsequent image distortion correction, projection transformation, and multi-view alignment, significantly improving the accuracy and stability of the stitching. Furthermore, to further address the time misalignment problem among multiple sensors, the pulse-of-seconds (PPS) signal provided by the Global Positioning System (GPS) was introduced as a unified time reference. This enabled hardware-level clock synchronization between the fisheye cameras and the inertial navigation equipment, ensuring that the timestamps of all devices were aligned with UTC standard time, eliminating clock drift between multiple data sources. Based on this, a timestamp matching algorithm was used to perform data-level time synchronization between the original images and attitude angle data, ensuring that each frame of the image was precisely associated with the attitude information corresponding to its acquisition time. The time synchronization error was controlled within 1 millisecond, fundamentally avoiding projection correction deviations caused by attitude lag. Subsequently, the original images were preprocessed, including fisheye distortion correction and noise filtering based on the intrinsic parameter matrix: distortion correction effectively restored the geometric authenticity of the image and eliminated the edge curvature caused by the inherent barrel distortion of the fisheye lens; noise filtering suppressed high-frequency noise interference under complex sea conditions such as sea fog, rain, snow, and strong light reflection, improved the image signal-to-noise ratio, and provided high-quality input data for subsequent processing.Next, based on the intrinsic parameter matrix, the pixel coordinates in the preprocessed image are converted into three-dimensional direction vectors in the camera coordinate system, and an attitude rotation matrix is ​​constructed by combining the roll angle and pitch angle synchronized with the current frame. Then, the three-dimensional direction vectors are transformed into a stable world coordinate system with the ship as the reference, dynamically compensating for the camera attitude offset caused by the ship's swaying, so that the virtual projection reference always remains "horizontally stable", avoiding tilting or stretching of the panoramic image and eliminating geometric distortion of the image caused by the ship's swaying. This achieves a "static in motion" visual effect - even if the ship sways, the final output bird's-eye view still presents a top-down view in a "stable ship" state, greatly improving the comfort of human observation and navigation usability. Then, a cylindrical projection model is used to map it onto a two-dimensional panoramic image plane to generate a geometrically stable, distortion-free, and tilt-free local bird's-eye projection image. This process achieves "dynamic projection correction", so even if the ship sways violently under the action of waves, the output bird's-eye view remains horizontally stable, which is significantly better than the traditional fixed projection model. After obtaining the four-way attitude-corrected bird's-eye view projection images, the spatial overlap area between adjacent images is determined based on the pre-calibrated extrinsic parameter matrix. By calculating the rate of change of roll and pitch angles (i.e., angular velocity) between the current and previous moments, the severity of the ship's attitude change is perceived in real time, serving as the basis for dynamically adjusting the fusion strategy and achieving rapid response to motion. Then, the pixel fusion weight of each pair of adjacent images in the overlapping area is dynamically calculated based on the angular velocity. When the attitude of a camera on one side is unstable due to the tilt of the ship, its fusion weight is automatically reduced, while the camera with a relatively stable attitude receives a higher weight, thereby suppressing visual discontinuities such as jitter and flicker. A weighted fusion operation is performed on the overlapping area, while the original pixel values ​​of the corresponding projection image are directly retained in the non-overlapping area, ultimately generating a seamless, continuous, and distortion-free 360° panoramic image. Finally, the generated 360° panoramic image is encoded into a video stream and transmitted in real time to the display terminal in the ship's bridge and the local storage device. This completes the stitching of the ship's real-time attitude with the panoramic video, achieving distortion-free panoramic image stitching under ship swaying scenarios. It balances real-time performance and environmental robustness, possesses long-term data value, and fully supports the safe operation of intelligent ships.

[0045] This invention constructs a high-precision, real-time, and environmentally adaptable 360° panoramic image generation solution, which is particularly suitable for pilotage and berthing operations of ships in complex sea conditions. Its core effects include the following aspects:

[0046] 1) Unique "Attitude-Projection" Linkage Mechanism: By deeply coupling the ship's roll and pitch angles, which are collected in real time by the inertial navigation device (IMU), with the cylindrical projection model, and by constructing a dynamic rotation matrix to adjust the projection coordinates in real time, unlike existing fixed projection schemes, this mechanism can fundamentally eliminate severe geometric distortions such as horizontal misalignment and image stretching caused by ship rolling (roll and pitch), generating a panoramic view with excellent geometric consistency, providing an accurate and reliable visual foundation for subsequent navigation and monitoring. 2) High-Precision Time Synchronization and Dynamic Weight Optimization: By introducing GPS second pulse signals, a time synchronization between the fisheye camera and the inertial navigation device is achieved at a value no greater than one millisecond (…). High-precision clock synchronization fundamentally avoids attitude correction lag (the "correction delay" caused by data lag), ensuring the timeliness and accuracy of attitude correction. Based on this, a dynamic weight optimization strategy based on the attitude change rate is proposed for overlapping area stitching. This strategy allows the fusion weights to adaptively adjust with the severity of ship swaying, significantly reducing the visibility of stitching seams (up to 90% or more), effectively solving the "stitching mark" problem caused by traditional fixed-weight fusion, and achieving a seamless and smooth visual transition. 3) High real-time performance and strong environmental robustness: To meet the millisecond-level real-time requirements of ship berthing and obstacle avoidance scenarios, a heterogeneous computing architecture with FPGA and GPU working together is adopted, that is, image preprocessing, attitude correction, and panoramic fusion are executed in parallel, ensuring that the end-to-end processing latency does not exceed fifty milliseconds. This invention achieves high-performance real-time processing, meeting the real-time operational needs of ships such as berthing and obstacle avoidance. 4) Strong environmental adaptability: This invention does not rely on image feature matching; it achieves stitching solely through IMU attitude and projection correction. Even under harsh imaging conditions such as level 5 sea fog and direct sunlight, the stitching accuracy remains above 95%, representing a 46% performance improvement compared to traditional feature-dependent vision solutions (accuracy approximately 65%), demonstrating extremely strong adaptability to complex marine environments. In summary, this invention successfully applies panoramic vision technology to highly dynamic and noisy ship swaying scenarios, solving a long-standing technical bottleneck in this field. It not only effectively improves the perception capabilities and operational safety of ships in various complex marine environments but also provides strong technical support for the development of intelligent ships, offering stable, reliable, real-time, and industrial-grade robust key technical guarantees for the perception systems of intelligent ships.

[0047] Furthermore, the synchronization error of the clock synchronization is less than or equal to 1 millisecond. By controlling the projection deviation caused by attitude lag within a negligible range (<0.01°), it ensures that each frame of the image is corrected using the attitude angle at its actual acquisition time, fundamentally avoiding image distortion or stitching misalignment caused by "correction delay", significantly improving the geometric accuracy of single-channel bird's-eye view, and laying the foundation for subsequent seamless stitching.

[0048] Furthermore, the generated 360° panoramic images are encoded into a video stream that can be transmitted via an HDMI interface. This stream is then transmitted in real-time to the display terminal in the ship's bridge via the HDMI protocol at a frame rate of 30 frames per second. This enables high-definition, low-latency, smooth, and stable real-time visual feedback, ensuring that crew members receive continuous, uninterrupted panoramic environmental information during critical operations such as berthing, pilotage, and obstacle avoidance. This significantly improves human-machine interaction efficiency and navigation safety. Simultaneously, the generated 360° panoramic images are encoded into MPEG-4 format video files and stored on local storage devices at a bitrate of 8Mbps. This achieves high-fidelity recording of the entire navigation process, meeting the requirements for long-term continuous storage (e.g., over 24 hours). Furthermore, it effectively balances storage space and image quality while ensuring clear and discernible image details, providing distortion-free panoramic visual data support for intelligent ship navigation monitoring, berthing assistance, and accident retrospective analysis.

[0049] This invention also relates to a panoramic image stitching system based on real-time ship attitude. This system corresponds to the aforementioned panoramic image stitching method based on real-time ship attitude and can be understood as a system that implements the aforementioned panoramic image stitching method based on real-time ship attitude. It includes a raw image and attitude angle acquisition module, an image and attitude angle data synchronization processing module, an image preprocessing module, an attitude correction and projection conversion module, a panoramic stitching and fusion module, and a result output module, all connected sequentially. These modules work collaboratively, deeply fusing the raw images acquired by the fisheye camera with the ship's roll and pitch angle data acquired in real-time by the inertial navigation system. Combined with a high-precision time synchronization mechanism and a dynamic projection correction model, stable and distortion-free 360° panoramic image generation is achieved under complex sea conditions. This solution fundamentally overcomes the problems of image misalignment, stretching, and stitching distortion caused by changes in ship attitude in traditional panoramic systems, significantly improving the environmental perception capabilities of ships in high-risk operational scenarios such as pilotage and berthing. The system achieves microsecond-level time synchronization between image and attitude data via GPS second pulse signals, effectively avoiding correction delays caused by sensor asynchrony. Fisheye distortion correction and coordinate projection transformation are performed based on calibration parameters to ensure the accuracy of geometric modeling. Furthermore, a dynamic weight fusion mechanism driven by the attitude change rate is introduced to achieve adaptive smooth stitching in overlapping areas of adjacent bird's-eye views, suppressing visual jumps and flickering. The entire system does not rely on image feature matching, overcoming the limitation of feature point failure under harsh conditions such as strong light, sea fog, rain, and snow, and possesses strong environmental robustness. The final generated panoramic image has advantages such as high resolution, low latency (end-to-end latency <50ms), and seamless continuity. It can be pushed to the bridge display terminal in real time and stored locally, fully supporting crew members in environmental monitoring, path planning, and emergency obstacle avoidance decisions. This solution achieves a balance of high real-time performance, high reliability, and strong adaptability, providing key technical support for the autonomous navigation and safe berthing of intelligent ships. Attached Figure Description

[0050] Figure 1 This is a flowchart of the panoramic image stitching method based on the real-time attitude of the ship hull according to the present invention. Detailed Implementation

[0051] The present invention will now be described with reference to the accompanying drawings.

[0052] This invention relates to a panoramic image stitching method based on the real-time attitude of a ship's hull. The flowchart of this method is as follows: Figure 1 As shown, the steps are as follows:

[0053] I. Raw Image and Attitude Angle Acquisition Steps: Multiple fisheye cameras and inertial navigation devices are installed on the ship's deck. The fisheye cameras are arranged around the hull, and the fields of view of adjacent fisheye cameras overlap to achieve 360° visual coverage of the surrounding environment. Raw images of the ship's surroundings are acquired in real time by each fisheye camera, and the attitude angle data of the ship is acquired in real time by the inertial navigation device. The attitude angle data includes roll angle and pitch angle.

[0054] Specifically, four fisheye cameras (each with a field of view of 120° or more) are first deployed on the ship's deck. These four fisheye cameras are arranged around the hull (e.g., one fisheye camera with a field of view of 120° or more is deployed at the bow, stern, port, and starboard sides). There is a 15%-20% overlap in the field of view between adjacent fisheye cameras to achieve 360° visual coverage of the surrounding environment. The four fisheye cameras acquire raw images of the ship's surroundings in real time at a rate of 30 frames per second (30fps). The output original image resolution is 2560×1440. A high-precision MEMS (Micro-Electro-Mechanical Systems) IMU (Inertial Measurement Unit, sampling rate 100Hz) is installed near the ship's center of gravity. The IMU collects the ship's attitude angle data (roll angle) in real time at a frequency of 100Hz. and pitch angle (and record the timestamp corresponding to each data point). .

[0055] II. Image and Attitude Angle Data Synchronization Processing Steps: Zhang's calibration method is used to calibrate each fisheye camera to obtain the intrinsic and extrinsic parameter matrices of each fisheye camera; then, the fisheye cameras and inertial navigation equipment are clock-synchronized using the second pulse signal provided by the Global Positioning System to obtain a unified time reference; based on the unified time reference, a timestamp matching algorithm is used to perform data-level time synchronization between the original image and the attitude angle data, so that each frame of the original image is associated with the attitude angle data at the corresponding time.

[0056] Specifically, the Zhang calibration method is first used to calibrate each fisheye camera, obtaining the intrinsic parameter matrix K (including focal length) of each fisheye camera. Principal point coordinates ), extrinsic parameter matrix (rotation matrix relative to the world coordinate system) Translation vector ) and distortion coefficient ( The intrinsic parameter matrix is ​​shown in the following equation:

[0057]

[0058] in, Focal length The coordinates of the main point.

[0059] Then, the inertial navigation system (IMU) is activated and zero-bias calibration is performed: the IMU is required to remain stationary for 30 seconds upon startup, and the output data of the IMU during these 30 seconds is collected. The average value is calculated as the zero-bias compensation value, which is then subtracted from all subsequent data to eliminate initial drift. Next, the fisheye camera and the IMU are synchronized at the hardware level using the PPS signal provided by the Global Positioning System (GPS) to obtain a unified time reference (ensuring that both share the same high-precision time reference, i.e., the time references of the fisheye camera and the IMU are aligned). Subsequently, based on the unified time reference, a timestamp matching algorithm is used to perform data-level time synchronization of the original images and attitude angle data, i.e., each frame of image... Associated with the IMU attitude data closest in time to satisfy the synchronization error of data-level time synchronization. ,Right now This ensures that each frame of the original image is associated with the attitude angle data at a corresponding moment, guaranteeing synchronization between subsequent images and the attitude angle data. The synchronization formula is shown below:

[0060]

[0061] In the above formula, Timestamp for each fisheye camera image frame; Timestamps for each pre-calibrated camera image frame With IMU data timestamps The time difference (synchronization error) between them.

[0062] III. Image Preprocessing Steps: The original images acquired by each fisheye camera are preprocessed to obtain preprocessed images; the preprocessing includes fisheye distortion correction and noise filtering based on the intrinsic parameter matrix.

[0063] Specifically, this step is mainly executed by the FPGA (Field-Programmable Gate Array) hardware module to meet the processing requirements of low latency and high throughput. First, fisheye distortion correction is performed on the pixel coordinates of the original images captured by each fisheye camera using pre-calibrated distortion coefficients. Fisheye distortion correction includes correcting the pixel coordinates in the original image (…). , Using the intrinsic parameter matrix fisheye distortion coefficient The corrected distortion-free coordinates are obtained by calculating using the correction formula. , This process eliminates the inherent distortion of the fisheye, thereby obtaining a corrected image. The correction formula is shown below:

[0064]

[0065]

[0066] Then, a Gaussian filtering algorithm is used to filter noise in the corrected image, specifically a 3×3 Gaussian filter (standard deviation). This process aims to suppress image noise introduced by factors such as sea surface reflection and water mist, ultimately outputting a preprocessed image. .

[0067] IV. Attitude Correction and Projection Transformation Steps: Based on the intrinsic parameter matrix, the pixel coordinates in the preprocessed image are converted into 3D orientation vectors in the camera coordinate system; and based on the roll and pitch angles associated with the current frame of the preprocessed image at the corresponding moment, an attitude rotation matrix is ​​constructed; according to the attitude rotation matrix, the 3D orientation vectors in the camera coordinate system are transformed to a stable world coordinate system based on the ship hull, and then projected onto the 2D panoramic image plane using a cylindrical projection model to generate the corrected projected image. This step corresponds to... Figure 1 The single image shown uses IMU attitude data to correct projection distortion, or in other words, a single image uses IMU attitude data to correct bird's-eye view projection.

[0068] Specifically, this step involves dynamic projection correction based on IMU pose, a process executed in parallel on the GPU (Graphics Processing Unit) for efficient computation. First, through the intrinsic parameter matrix... Inverse transformation (inverse matrix) The pixel coordinates in the preprocessed image ( Convert to a 3D direction vector in the camera coordinate system As shown in the following formula:

[0069]

[0070] in, This is the transpose of the matrix.

[0071] Subsequently, IMU attitude data synchronized with the current frame preprocessed image is acquired, namely, the roll angle at the current moment based on the real-time acquisition of the IMU and associated with the current frame preprocessed image. (around the X-axis) and pitch angle Construct the attitude rotation matrix (around the Y-axis). The attitude rotation matrix By rotation matrix about the X-axis and rotation matrix around the Y-axis Multiplying the vectors yields a vector used to transform the 3D orientation vectors in the camera coordinate system to a "ship-stable" world coordinate system. This transformation essentially "counters" the actual tilt of the ship, ensuring that the image points maintain a horizontal orientation in virtual space.

[0072] The rotation matrix around the X-axis (correcting for roll) is shown below:

[0073]

[0074] The rotation matrix around the Y-axis (correcting pitch) is shown below:

[0075]

[0076] Based on the attitude rotation matrix The three-dimensional direction vector in the camera coordinate system Transform to a stable world coordinate system based on the ship's hull, as shown in the following equation:

[0077]

[0078] Finally, the three-dimensional direction vectors in the stable world coordinate system are projected onto the two-dimensional panoramic image plane (cylindrical panoramic image plane) using the cylindrical projection model, generating a corrected, distortion-free projection image. The process involves projecting a three-dimensional direction vector from a stable world coordinate system onto a virtual "vertical cylindrical surface," which is then unfolded to form a stable, distortion-free bird's-eye view unaffected by the ship's swaying. The cylindrical surface projection model (projection formula) is shown below:

[0079]

[0080]

[0081] in, For the projection focal length ( =1200), The coordinates of the center of the two-dimensional panoramic image plane in the horizontal and vertical directions are (for example, when the width of the two-dimensional panoramic image plane is 4096, then...). When the height of the two-dimensional panoramic image plane is 1024, then This process is used to align the projection results to the center area of ​​the image, ensuring the visual centering and integrity of the final image. Through this series of transformations, the horizontal misalignment and stretching distortion of the image caused by the ship's rolling are eliminated. There is no need to rely on time-consuming image feature matching. The image that was originally distorted by the ship's rolling is "straightened" and a stable, distortion-free single-path bird's-eye view image is generated, thus solving the projection distortion caused by the ship's rolling.

[0082] V. Panoramic Stitching and Fusion Steps: Based on the extrinsic parameter matrix, the overlapping area between the corresponding projected images of adjacent fisheye cameras is obtained. Based on the roll and pitch angles at the current moment and the moment before, the roll angle change rate and pitch angle change rate at the current moment are calculated respectively. According to the roll angle change rate and pitch angle change rate, the pixel fusion weight of the overlapping area of ​​each pair of adjacent projected images is dynamically calculated. Then, based on the dynamically calculated pixel fusion weight, each pixel value in the overlapping area is weighted and fused. The pixel values ​​of the corresponding projected images are retained for the non-overlapping areas, thereby generating a seamless 360° panoramic image. This step corresponds to... Figure 1 The multi-path corrected projection images shown are stitched and fused into a 360° panoramic image, or multiple bird's-eye views are stitched and fused into a 360° surround view.

[0083] After the above attitude correction and projection transformation, four corrected projection images are obtained. (Bird's-eye view) images are stitched and merged to generate a complete 360° panoramic view. Specifically, the overlapping area between the corresponding projected images of adjacent fisheye cameras is first obtained based on the extrinsic parameter matrix (e.g., the overlapping area between fisheye cameras 1 and 2). The overlapping area of ​​fisheye cameras 2 and 3 For these overlapping regions, instead of employing the traditional fixed-weight fusion strategy, a dynamic weight fusion mechanism is introduced to dynamically adjust the fusion weights, addressing the visual discontinuity issue caused by drastic changes in ship attitude. Specifically, based on the roll and pitch angles at the current moment and the moment before, the rate of change of the roll angle at the current moment is calculated. and pitch angle change rate As shown in the following formula:

[0084]

[0085]

[0086] Then, based on the roll angle change rate and pitch angle change rate, the pixel fusion weight of the overlapping region of each pair of adjacent projected images (such as adjacent projected images i and j) is dynamically calculated. For example, when the ship rapidly tilts to the right, the attitude of the left-hand camera is relatively more stable, and its fusion weight in the overlapping region will be automatically increased. When the ship sways violently, the weight distribution tends to be more even to avoid ghosting or blurring at the fusion point due to severe distortion of a single image. The pixel fusion weight of projected image i in the overlapping region is calculated according to the following formula:

[0087]

[0088] in, , This is the attitude influence coefficient. As initial weights, and with the projected image Adjacent projected images Pixel fusion weights .

[0089] Finally, based on dynamically calculated pixel fusion weights, each pixel value within the overlapping region is weighted and fused, and for each pair of adjacent projected images, the non-overlapping regions (excluding the overlapping areas) directly retain their corresponding projected images. The original pixel values ​​are then used to generate a seamless 360° panoramic image. (resolution) This effectively eliminates seams in overlapping areas, especially when the ship is rocking, resulting in a smoother and more natural stitching and resolving the issue of noticeable seams in overlapping areas. The weighted pixel values ​​are shown in the following formula:

[0090]

[0091] in, and These are the pixel values ​​of adjacent projected images.

[0092] VI. Result Output Steps: Corresponding Figure 1 The 360° panoramic image encoding and output are shown. The generated 360° panoramic image... The video stream is encoded into a video stream that can be transmitted via an HDMI interface and transmitted in real time to the display terminal in the ship's bridge via the HDMI protocol at a frame rate of 30 frames per second (30fps). At the same time, the generated 360° panoramic image is encoded into an MPEG-4 format video file and stored in a local storage device (local hard drive) at a bit rate of 8 megabits per second (8Mbps). This completes the stitching of the panoramic image and the video display, providing distortion-free panoramic visual data support for intelligent ship navigation monitoring, berthing assistance, and accident retrospective analysis.

[0093] This invention also relates to a panoramic image stitching system based on real-time ship attitude. This system corresponds to the aforementioned panoramic image stitching method based on real-time ship attitude and can be understood as a system implementing the aforementioned method. The system includes, in sequence, an original image and attitude angle acquisition module, an image and attitude angle data synchronization processing module, an image preprocessing module, an attitude correction and projection conversion module, a panoramic stitching and fusion module, and a result output module. Specifically,

[0094] The original image and attitude angle acquisition module is equipped with multiple fisheye cameras and inertial navigation devices on the ship's deck. The fisheye cameras are arranged around the hull, and the fields of view of adjacent fisheye cameras overlap to achieve 360° visual coverage of the environment around the ship. The original images around the ship are acquired in real time by each fisheye camera, and the attitude angle data of the ship, including roll angle and pitch angle, are acquired in real time by the inertial navigation device.

[0095] The image and attitude angle data synchronization processing module uses Zhang's calibration method to calibrate each fisheye camera to obtain the intrinsic and extrinsic parameter matrices of each fisheye camera; then, it uses the second pulse signal provided by the Global Positioning System to perform hardware-level clock synchronization between the fisheye cameras and the inertial navigation equipment to obtain a unified time reference; based on the unified time reference, it uses a timestamp matching algorithm to perform data-level time synchronization between the original image and the attitude angle data, so that each frame of the original image is associated with the attitude angle data at the corresponding time.

[0096] The image preprocessing module preprocesses the raw images captured by each fisheye camera to obtain a preprocessed image; the preprocessing includes fisheye distortion correction and noise filtering based on the intrinsic parameter matrix.

[0097] The attitude correction and projection conversion module converts the pixel coordinates in the preprocessed image into a three-dimensional direction vector in the camera coordinate system based on the intrinsic parameter matrix; and constructs an attitude rotation matrix based on the roll angle and pitch angle at the corresponding moment associated with the current frame of the preprocessed image; according to the attitude rotation matrix, the three-dimensional direction vector in the camera coordinate system is converted to a stable world coordinate system based on the ship hull, and the three-dimensional direction vector in the stable world coordinate system is projected onto the two-dimensional panoramic image plane through a cylindrical projection model to generate a corrected projection image;

[0098] The panoramic stitching and fusion module obtains the overlapping area between the corresponding projected images of adjacent fisheye cameras based on the extrinsic parameter matrix, and calculates the roll angle change rate and pitch angle change rate at the current moment based on the roll angle and pitch angle at the previous moment. Based on the roll angle change rate and pitch angle change rate, the pixel fusion weight of the overlapping area of ​​each pair of adjacent projected images is dynamically calculated. Then, based on the dynamically calculated pixel fusion weight, each pixel value in the overlapping area is weighted and fused, and the pixel values ​​of the corresponding projected images of the non-overlapping areas are retained, thereby generating a seamless 360° panoramic image.

[0099] The output module encodes the generated 360° panoramic image into a video stream and transmits the video stream in real time to the display terminal in the ship's bridge and the local storage device to complete the stitching of the panoramic image and the display of the video.

[0100] Preferably, the image preprocessing module specifically includes the following preprocessing steps for the original image:

[0101] First, fisheye distortion correction is performed on the original image. Fisheye distortion correction includes correcting the pixel coordinates in the original image ( , Using the fisheye distortion coefficients in the intrinsic parameter matrix The corrected coordinates are obtained by calculating using the correction formula. , This process is repeated to obtain the corrected image; the correction formula is as follows:

[0102]

[0103]

[0104] The Gaussian filtering algorithm is then used to filter noise in the corrected image to suppress image noise introduced by factors such as sea surface reflection and water mist, and finally outputs the preprocessed image.

[0105] Preferably, in the attitude correction and projection conversion module, the cylindrical projection model is as follows:

[0106]

[0107]

[0108] in, To stabilize the three-dimensional direction vector in the world coordinate system; These are the horizontal and vertical center coordinates of the two-dimensional panoramic image plane.

[0109] Preferably, in the image and attitude angle data synchronization processing module, the synchronization error of the clock synchronization is less than or equal to 1 millisecond.

[0110] Preferably, in the result output module, the generated 360° panoramic image is encoded into a video stream and transmitted in real time to the display terminal in the ship's bridge via the HDMI protocol at a frame rate of thirty frames per second; at the same time, the generated 360° panoramic image is encoded into an MPEG-4 format video file and stored in a local storage device at a bit rate of eight megabits per second.

[0111] This invention provides an objective and scientific panoramic image stitching method and system based on real-time ship attitude. It fundamentally overcomes the problems of image misalignment, stretching, and stitching distortion caused by changes in ship attitude in traditional panoramic systems, significantly improving the environmental perception capabilities of ships in high-risk operational scenarios such as pilotage and berthing. The entire solution does not rely on image feature matching, overcoming the limitations of feature point failure under harsh conditions such as strong light, sea fog, rain, and snow, and possesses strong environmental robustness. It achieves a balance between high real-time performance, high reliability, and strong adaptability, providing key technical support for the autonomous navigation and safe berthing of intelligent ships.

[0112] It should be noted that the specific embodiments described above enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail with reference to the accompanying drawings and embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention. In short, all technical solutions and improvements that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention patent.

Claims

1. A panoramic image stitching method based on real-time ship attitude, characterized in that, Includes the following steps: Raw image and attitude angle acquisition steps: Multiple fisheye cameras and inertial navigation devices are installed on the ship's deck. The fisheye cameras are arranged around the hull circumference, and the fields of view of adjacent fisheye cameras overlap to achieve 360° visual coverage of the environment around the ship. Raw images of the ship's surroundings are acquired in real time by each fisheye camera, and the attitude angle data of the ship is acquired in real time by the inertial navigation device. The attitude angle data includes roll angle and pitch angle. Image and attitude angle data synchronization processing steps: Zhang's calibration method is used to calibrate each fisheye camera to obtain the intrinsic and extrinsic parameter matrices of each fisheye camera; then, the fisheye cameras and inertial navigation equipment are synchronized at the hardware level using the second pulse signal provided by the Global Positioning System to obtain a unified time reference; based on the unified time reference, a timestamp matching algorithm is used to perform data-level time synchronization between the original image and attitude angle data, so that each frame of the original image is associated with the attitude angle data at the corresponding time. Image preprocessing steps: The original images captured by each fisheye camera are preprocessed to obtain preprocessed images; the preprocessing includes fisheye distortion correction and noise filtering based on the intrinsic parameter matrix; Attitude correction and projection transformation steps: Based on the intrinsic parameter matrix, the pixel coordinates in the preprocessed image are converted into three-dimensional orientation vectors in the camera coordinate system; and based on the roll angle and pitch angle at the corresponding moment associated with the current frame of the preprocessed image, the attitude rotation matrix is ​​constructed. The three-dimensional direction vector in the camera coordinate system is transformed to the stable world coordinate system based on the ship hull according to the attitude rotation matrix, and the three-dimensional direction vector in the stable world coordinate system is projected onto the two-dimensional panoramic image plane through the cylindrical projection model to generate the corrected projection image. Panoramic stitching and fusion steps: Based on the extrinsic parameter matrix, the overlapping area between the corresponding projected images of adjacent fisheye cameras is obtained. Based on the roll angle and pitch angle at the current time and the time before the current time, the roll angle change rate and pitch angle change rate at the current time are calculated respectively. According to the roll angle change rate and pitch angle change rate, the pixel fusion weight of the overlapping area of ​​each pair of adjacent projected images is dynamically calculated. Then, based on the dynamically calculated pixel fusion weight, each pixel value in the overlapping area is weighted and fused. The pixel values ​​of the corresponding projected images of the non-overlapping areas other than the overlapping areas are retained, thereby generating a seamless 360° panoramic image. Output steps: Encode the generated 360° panoramic image into a video stream, and transmit the video stream in real time to the display terminal in the ship's bridge and the local storage device to complete the stitching of the panoramic image and the video display.

2. The panoramic image stitching method based on real-time ship attitude according to claim 1, characterized in that, The image preprocessing step specifically includes the following: First, fisheye distortion correction is performed on the original image. Fisheye distortion correction includes correcting the pixel coordinates in the original image ( , Using the fisheye distortion coefficients in the intrinsic parameter matrix The corrected coordinates are obtained by calculating using the correction formula. , This process is repeated to obtain the corrected image; the correction formula is as follows: ， ， The Gaussian filtering algorithm is then used to filter noise in the corrected image to suppress image noise introduced by factors such as sea surface reflection and water mist, and finally outputs the preprocessed image.

3. The panoramic image stitching method based on real-time ship attitude according to claim 1, characterized in that, In the attitude correction and projection transformation steps, the cylindrical projection model is shown below: ， ， in, To stabilize the three-dimensional direction vector in the world coordinate system; These are the horizontal and vertical center coordinates of the two-dimensional panoramic image plane.

4. The panoramic image stitching method based on real-time ship attitude according to claim 3, characterized in that, In the image and attitude angle data synchronization processing step, the synchronization error of the data-level time synchronization is less than or equal to 1 millisecond.

5. The panoramic image stitching method based on real-time ship attitude according to claim 4, characterized in that, In the result output step, the generated 360° panoramic image is encoded into a video stream and transmitted in real time to the display terminal in the ship's bridge via the HDMI protocol at a frame rate of thirty frames per second; at the same time, the generated 360° panoramic image is encoded into an MPEG-4 format video file and stored in the local storage device at a bit rate of eight megabits per second.

6. A panoramic image stitching system based on real-time ship attitude, characterized in that, It includes, in sequence, a raw image and attitude angle acquisition module, an image and attitude angle data synchronization processing module, an image preprocessing module, an attitude correction and projection conversion module, a panoramic stitching and fusion module, and a result output module. The original image and attitude angle acquisition module is equipped with multiple fisheye cameras and inertial navigation devices on the ship's deck. The fisheye cameras are arranged around the hull, and the fields of view of adjacent fisheye cameras overlap to achieve 360° visual coverage of the environment around the ship. The original images around the ship are acquired in real time by each fisheye camera, and the attitude angle data of the ship, including roll angle and pitch angle, are acquired in real time by the inertial navigation device. The image and attitude angle data synchronization processing module uses Zhang's calibration method to calibrate each fisheye camera to obtain the intrinsic and extrinsic parameter matrices of each fisheye camera; then, it uses the second pulse signal provided by the Global Positioning System to perform hardware-level clock synchronization between the fisheye cameras and the inertial navigation equipment to obtain a unified time reference; based on the unified time reference, it uses a timestamp matching algorithm to perform data-level time synchronization between the original image and the attitude angle data, so that each frame of the original image is associated with the attitude angle data at the corresponding time. The image preprocessing module preprocesses the raw images captured by each fisheye camera to obtain a preprocessed image; the preprocessing includes fisheye distortion correction and noise filtering based on the intrinsic parameter matrix. The attitude correction and projection conversion module converts the pixel coordinates in the preprocessed image into a three-dimensional direction vector in the camera coordinate system based on the intrinsic parameter matrix; and constructs an attitude rotation matrix based on the roll angle and pitch angle at the corresponding moment associated with the current frame preprocessed image. The three-dimensional direction vector in the camera coordinate system is transformed to the stable world coordinate system based on the ship hull according to the attitude rotation matrix, and the three-dimensional direction vector in the stable world coordinate system is projected onto the two-dimensional panoramic image plane through the cylindrical projection model to generate the corrected projection image. The panoramic stitching and fusion module obtains the overlapping area between the corresponding projected images of adjacent fisheye cameras based on the extrinsic parameter matrix, and calculates the roll angle change rate and pitch angle change rate at the current moment based on the roll angle and pitch angle at the previous moment. Based on the roll angle change rate and pitch angle change rate, the pixel fusion weight of the overlapping area of ​​each pair of adjacent projected images is dynamically calculated. Then, based on the dynamically calculated pixel fusion weight, each pixel value in the overlapping area is weighted and fused, and the pixel values ​​of the corresponding projected images of the non-overlapping areas are retained, thereby generating a seamless 360° panoramic image. The output module encodes the generated 360° panoramic image into a video stream and transmits the video stream in real time to the display terminal in the ship's bridge and the local storage device to complete the stitching of the panoramic image and the display of the video.

7. The panoramic image stitching system based on real-time ship attitude according to claim 6, characterized in that, The image preprocessing module specifically includes the following preprocessing steps for the original image: First, fisheye distortion correction is performed on the original image. Fisheye distortion correction includes correcting the pixel coordinates in the original image ( , Using the fisheye distortion coefficients in the intrinsic parameter matrix The corrected coordinates are obtained by calculating using the correction formula. , This process is repeated to obtain the corrected image; the correction formula is as follows: ， ， The Gaussian filtering algorithm is then used to filter noise in the corrected image to suppress image noise introduced by factors such as sea surface reflection and water mist, and finally outputs the preprocessed image.

8. The panoramic image stitching system based on real-time ship attitude according to claim 6, characterized in that, In the attitude correction and projection conversion module, the cylindrical projection model is shown below: ， ， in, To stabilize the three-dimensional direction vector in the world coordinate system; These are the horizontal and vertical center coordinates of the two-dimensional panoramic image plane.

9. The panoramic image stitching system based on real-time ship attitude according to claim 8, characterized in that, In the image and attitude angle data synchronization processing module, the synchronization error of the clock synchronization is less than or equal to 1 millisecond.

10. The panoramic image stitching system based on real-time ship attitude according to claim 9, characterized in that, In the result output module, the generated 360° panoramic image is encoded into a video stream and transmitted in real time to the display terminal in the ship's bridge via the HDMI protocol at a frame rate of thirty frames per second; at the same time, the generated 360° panoramic image is encoded into an MPEG-4 format video file and stored in the local storage device at a bit rate of eight megabits per second.

Images