Offset prediction network training method, fisheye image stitching method, system and ship

By constructing a deformable mesh geometric loss function and a cross-camera consistency loss function, the problem of error accumulation in traditional fisheye camera stitching is solved, achieving high-quality panoramic stitching of fisheye images, ensuring geometric consistency and temporal smoothness in the stitching process, and improving the clarity of ship monitoring images.

CN123118356APending Publication Date: 2026-11-17CATL ELECTRIC BOAT TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611557933.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-09
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In traditional fisheye camera stitching methods, errors accumulate with each iteration, leading to a decline in image stitching quality and misalignment. This is particularly problematic in ship surround view systems, where achieving high-quality panoramic stitching is difficult.

Method used

The offset prediction network training method is adopted. By constructing a deformable mesh geometric loss function and a cross-camera consistency loss function, the deformation parameters of multiple fisheye cameras are jointly constrained to achieve one-time global stitching, thereby reducing error accumulation and misalignment probability.

Benefits of technology

It improves the quality and accuracy of panoramic fisheye image stitching, ensures geometric consistency and temporal smoothness in the stitching process, reduces ghosting and stitching gaps, and provides clear ship monitoring images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118356A_ABST
    Figure CN123118356A_ABST
Patent Text Reader

Abstract

This application relates to the field of fisheye image stitching technology, and discloses a method for training an offset prediction network, a fisheye image stitching method, a system, and a ship. The training method includes: determining a first homography matrix based on bird's-eye view samples from each fisheye camera; determining a first projected image based on the bird's-eye view samples and the first homography matrix; determining a first deformable mesh based on the bird's-eye view samples; predicting a first local deformable offset for each fisheye camera based on the bird's-eye view samples, the first projected image, and the prediction network; the prediction network has corresponding initial parameters; constructing a geometric loss function based on each first local deformable offset, the first deformable mesh, the first homography matrix, and the overlapping area of ​​adjacent cameras; and updating the network's initial parameters based on the loss function. Thus, by using the geometric loss function to jointly constrain the deformable parameters of each camera, the deformable meshes of each camera are geometrically consistent, and a one-time global joint stitching of all cameras is achieved, improving the image stitching quality and accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of fisheye image stitching technology, specifically to an offset prediction network training method, a fisheye image stitching method, a system, and a ship. Background Technology

[0002] Currently, the common surround view solution for ships is to install a fisheye camera around each of the four sides of the hull, forming a four-camera surround view system. Each fisheye camera covers an area of ​​about 180 degrees, so that the four-camera surround view system can provide a 360-degree surround view.

[0003] Traditionally, when stitching multiple fisheye cameras, a pair-by-pair cascading stitching method is used. Specifically, the stitching of N cameras is decomposed into N-1 rounds of dual-camera stitching, with each round processed independently. This method leads to the accumulation of errors round by round and can cause stitching misalignment, reducing the quality of image stitching.

[0004] In conclusion, improving the quality of image stitching is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, this application provides an offset prediction network training method, a fisheye image stitching method, a system, and a ship to improve image stitching quality.

[0006] In a first aspect, this application provides a method for training an offset prediction network, comprising: acquiring bird's-eye view samples corresponding to multiple fisheye cameras on a ship; determining a first homography matrix corresponding to each fisheye camera based on each bird's-eye view sample; determining a first projected image based on each bird's-eye view sample and the corresponding first homography matrix; determining a first deformable mesh based on the bird's-eye view samples; predicting a first local deformation offset between the first deformable mesh and each fisheye camera based on each bird's-eye view sample, each first projected image, and the deformable mesh offset prediction network; the deformable mesh offset prediction network having initial parameters; constructing a deformable mesh geometric loss function based on the first local deformation offset corresponding to each fisheye camera, the first deformable mesh, each first homography matrix, and the overlapping area of ​​adjacent fisheye cameras; and updating the initial parameters of the deformable mesh offset prediction network based on the deformable mesh geometric loss function.

[0007] The scheme disclosed in some embodiments of this application uses a deformable mesh offset prediction network to predict the first local deformation offset corresponding to each fisheye camera during the training phase. Based on the first local deformation offset corresponding to each fisheye camera, the first deformable mesh, each first homography matrix, and the overlapping area of ​​adjacent fisheye cameras, a deformable mesh geometric loss function is constructed. The deformable mesh offset prediction network is updated through the deformable mesh geometric loss function, so as to achieve joint constraint on the deformation parameters corresponding to all fisheye cameras during the training phase, making the deformable mesh corresponding to each fisheye camera geometrically consistent, and enabling the trained deformable mesh offset prediction network to output the local deformation offset corresponding to each fisheye camera in parallel, thereby achieving a one-time global joint stitching of all fisheye cameras. This solves the error accumulation problem of pairwise cascade stitching, reduces the probability of image stitching misalignment, and improves the quality and accuracy of fisheye image panoramic stitching.

[0008] In some embodiments, constructing a deformable mesh geometric loss function based on the first local deformation offset corresponding to each fisheye camera, the first deformable mesh, each first homography matrix, and the overlapping region of adjacent fisheye cameras includes: determining a rigid offset corresponding to each fisheye camera based on the first deformable mesh and each first homography matrix; determining a rigid constraint loss function based on the rigid offset corresponding to each fisheye camera and the first local deformation offset corresponding to each fisheye camera; determining a second deformable mesh corresponding to each fisheye camera based on the first deformable mesh and the first local deformation offset corresponding to each fisheye camera; determining a cross-camera consistency loss function based on the second deformable mesh corresponding to adjacent fisheye cameras and the overlapping region of adjacent fisheye cameras; and constructing the deformable mesh geometric loss function based on the rigid constraint loss function and the cross-camera consistency loss function.

[0009] The above method can utilize rigid constraint loss function to achieve rigid constraints and cross-camera consistency loss function to achieve cross-camera consistency constraints. The two work together to ensure global consistency, reduce stitching errors, ghosting, and stitching gaps, and improve the stitching quality of fisheye images.

[0010] In some embodiments, the cross-camera consistency loss function includes an overlapping region mapping consistency loss function. The cross-camera consistency loss function is determined based on the second deformable mesh corresponding to adjacent fisheye cameras and the overlapping regions of adjacent fisheye cameras. This includes: using the second deformable mesh corresponding to adjacent fisheye cameras, mapping each overlapping pixel in the overlapping region of adjacent fisheye cameras to a panoramic coordinate system to obtain a first panoramic pixel and a second panoramic pixel corresponding to each overlapping pixel; and determining the overlapping region mapping consistency loss function based on the first panoramic pixel and the second panoramic pixel corresponding to each overlapping pixel.

[0011] The above method can constrain the mapping position within the overlapping area, making the mapping position of the same object within the overlapping area consistent, thereby maintaining local consistency in the overlapping area and reducing ghosting.

[0012] In some embodiments, the cross-camera consistency loss function further includes a boundary gradient consistency loss function. The cross-camera consistency loss function is determined based on the second deformable mesh corresponding to the adjacent fisheye camera and the overlapping region of the adjacent fisheye camera, including: determining the gradient of the second deformable mesh corresponding to the adjacent fisheye camera at the boundary of the corresponding overlapping region; and determining the boundary gradient consistency loss function based on the gradient of the second deformable mesh corresponding to the adjacent fisheye camera at the boundary of the corresponding overlapping region.

[0013] The above method can constrain the gradient continuity of the second deformed mesh at the boundary of the overlapping region, so as to make the boundary of the overlapping region transition smoothly and reduce the splicing gap.

[0014] In some embodiments, the method further includes: determining a corresponding sea surface area image based on each of the bird's-eye image samples; determining a sea surface plane constraint loss function based on the sea surface area image, the intrinsic parameter matrix of each of the fisheye cameras, the sea level, the first deformable mesh, and the first local deformation offset corresponding to each of the fisheye cameras; and updating the initial parameters of the deformable mesh offset prediction network based on the deformable mesh geometric loss function, including updating the initial parameters of the deformable mesh offset prediction network based on the deformable mesh geometric loss function and the sea surface plane constraint loss function.

[0015] The above method allows the deformable mesh geometric loss function and the sea surface plane constraint loss function to work together in the training of the deformable mesh offset prediction network. This enables the deformable mesh offset prediction network to accept both image-level and geometric-level constraints in a single training session. As a result, the deformable mesh offset prediction network can learn a deformation law that is consistent between the fisheye camera mappings in the overlapping area and conforms to the physical geometry in the sea surface area, thereby improving the image stitching quality.

[0016] In some embodiments, determining a sea surface plane constraint loss function based on the sea surface area image, the intrinsic parameter matrix of each fisheye camera, the sea level, the first deformable mesh, and the first local deformation offset corresponding to each fisheye camera includes: mapping each sea surface pixel in the sea surface area image to a panoramic coordinate system based on the intrinsic parameter matrix of each fisheye camera and the sea level to obtain an ideal panoramic pixel corresponding to each sea surface pixel; determining a second deformable mesh corresponding to each fisheye camera based on the first deformable mesh and the first local deformation offset corresponding to each fisheye camera; mapping each sea surface pixel in the sea surface area image to the panoramic coordinate system using the second deformable mesh corresponding to each fisheye camera to obtain a panoramic sea surface pixel corresponding to each sea surface pixel; and determining the sea surface plane constraint loss function based on the ideal panoramic pixel and the panoramic sea surface pixel.

[0017] The above method can be used to make the deformed mesh output by the deformed mesh offset prediction network in the sea area close to the ideal position determined by geometric prior, so that the mapping result of the second deformed mesh synthesized by the first local deformation offset in the sea area matches the ideal panoramic pixel points, thereby improving the image stitching quality.

[0018] In some embodiments, the method further includes: determining a second deformable mesh corresponding to each fisheye camera based on the first deformable mesh and the first local deformable offset corresponding to each fisheye camera; determining a mesh temporal smoothing loss function based on the second deformable mesh corresponding to each fisheye camera and the second historical deformable mesh corresponding to each fisheye camera in a historical frame; and updating the initial parameters of the deformable mesh offset prediction network based on the deformable mesh geometric loss function, including updating the initial parameters of the deformable mesh offset prediction network based on the deformable mesh geometric loss function and the mesh temporal smoothing loss function.

[0019] The above method can incorporate constraints based on the time dimension determined by the deformable mesh into the update of the deformable mesh offset prediction network, thereby constraining the amount of change of the deformable mesh in multiple frames, reducing high-frequency jitter at the mesh level, reducing the probability of inter-frame jumps in the deformable mesh, making the ship monitoring screen smooth and stable, so that the driver can clearly observe the dock and obstacles, and reduce visual fatigue or misjudgment caused by screen jitter.

[0020] In some embodiments, determining the first homography matrix corresponding to each fisheye camera based on each of the bird's-eye image samples includes: determining the motion parameters of the ship based on each of the bird's-eye image samples; and determining the first homography matrix corresponding to each fisheye camera based on the motion parameters of the ship and the relative poses between the fisheye cameras.

[0021] By jointly estimating the ship's motion parameters from the features of all fisheye cameras and deriving the homography matrix of each camera using calibration parameters and rigid constraints, globally consistent coarse alignment can be achieved. This suppresses the problems of error accumulation and misalignment in traditional pairwise cascaded stitching, effectively improving the quality of image stitching.

[0022] In some embodiments, the method further includes: determining a motion smoothing loss function based on the motion parameters of the ship and the historical motion parameters of the ship in historical frames; and updating the initial parameters of the deformable mesh offset prediction network based on the deformable mesh geometric loss function, including: updating the initial parameters of the deformable mesh offset prediction network based on the deformable mesh geometric loss function and the motion smoothing loss function.

[0023] The above method allows for the inclusion of time-dimensional constraints based on motion parameters in the updating of the deformable mesh offset prediction network. This enables the deformable mesh offset prediction network to learn the ability to output smooth and continuous motion parameters, preventing abrupt changes in the ship's roll angle between frames. This reduces the probability of sudden motion changes and drastic shifts in motion trends. Furthermore, the smoothing of motion parameters can drive frame-by-frame smoothing of the first homography matrix and the coarse alignment results, thereby establishing temporal stability at the source of the network input.

[0024] In some embodiments, determining a motion smoothing loss function based on the ship's motion parameters and the ship's historical motion parameters in a historical frame includes: determining a velocity loss function and an acceleration loss function based on the ship's motion parameters and the ship's historical motion parameters in a historical frame; and determining the motion smoothing loss function based on the velocity loss function and the acceleration loss function.

[0025] The above method allows the velocity loss function and the acceleration loss function to participate in the motion smoothing loss function, thereby reducing the abrupt changes in yaw angle between frames and making the monitoring image smoother and more stable.

[0026] In some embodiments, before determining the first homography matrix corresponding to each fisheye camera based on each of the bird's-eye image samples, the method further includes: extracting multi-scale features of the bird's-eye image samples using a feature extraction network to obtain feature map samples of the bird's-eye image samples; determining the first homography matrix corresponding to each of the fisheye cameras based on each of the bird's-eye image samples; and determining the first projected image based on each of the bird's-eye image samples and the corresponding first homography matrix, including: determining the first homography matrix corresponding to each of the fisheye cameras based on the feature map samples of each of the bird's-eye image samples; and determining the first projected feature image based on the feature map samples of each of the bird's-eye image samples and the corresponding first homography matrix.

[0027] The above method can gather the structural context around the weak texture region of the sea surface, so that the solution of the first homography matrix is ​​still relatively accurate in weak texture scenes with few feature points, thereby improving the alignment accuracy and thus improving the accuracy of the local deformation offset and the deformation mesh offset prediction network.

[0028] In some embodiments, before determining the first homography matrix corresponding to each fisheye camera based on the feature map samples of each bird's-eye image sample, the method further includes: determining weak texture region samples based on the feature map samples of the bird's-eye image sample; performing cross-camera attention interaction on the weak texture region samples in the overlapping regions of adjacent fisheye cameras to obtain fused and enhanced weak texture region samples; and obtaining enhanced feature map samples based on the feature map of the bird's-eye image sample and the fused and enhanced weak texture region samples.

[0029] The above method can be used to enhance the features of weak texture regions across cameras, so that each fisheye camera can see the overlapping features of adjacent fisheye cameras. This allows weak texture regions to obtain supplementary information from adjacent cameras, achieving cross-camera information enhancement, while the features of non-weak texture regions can remain unchanged, thus reducing the amount of computation.

[0030] In some embodiments, performing cross-camera attention interaction on weak texture region samples in the overlapping region of adjacent fisheye cameras to obtain fused and enhanced weak texture region samples includes: performing cross-camera attention interaction on weak texture region samples in the overlapping region of adjacent fisheye cameras in a first direction to obtain fused and enhanced first weak texture region samples; performing cross-camera attention interaction on weak texture region samples in the overlapping region of adjacent fisheye cameras in a second direction to obtain fused and enhanced second weak texture region samples; wherein the first direction and the second direction are opposite in direction.

[0031] The above method allows cross-camera attention to propagate bidirectionally along the phase chain, enabling each camera to ultimately fuse information from all cameras and utilize indirect cross-camera information, thereby facilitating improved image stitching quality.

[0032] Secondly, this application provides a fisheye image stitching method, comprising: acquiring fisheye images captured by multiple fisheye cameras on a ship; obtaining a corresponding bird's-eye view based on each fisheye image; determining a second homography matrix corresponding to each fisheye camera based on each bird's-eye view; determining a second projection map based on each bird's-eye view and the corresponding second homography matrix; determining a third deformable mesh based on the bird's-eye view; predicting a second local deformation offset between the third deformable mesh and each fisheye camera based on each bird's-eye view, each second projection map, and a deformable mesh offset prediction network trained using the offset prediction network training method described above; determining a fourth deformable mesh corresponding to each fisheye camera based on the third deformable mesh and the second local deformation offset; and determining a panoramic image based on the fourth deformable mesh corresponding to each fisheye camera and each bird's-eye view.

[0033] By using the offset prediction network training method to train a deformable mesh offset prediction network to determine the deformable mesh of the fisheye camera, the global consistency, temporal smoothness and geometric accuracy established in the training phase are continued without loss in the inference phase. The entire stitching process is a one-time parallel process of coarse alignment, local fine deformation of the mesh and unified canvas fusion. This can solve the problem of error accumulation caused by independent processing of each stitching round in the traditional pairwise cascaded stitching method, thereby improving the image stitching quality.

[0034] In some embodiments, determining a panoramic image based on the fourth deformable grid corresponding to each fisheye camera and each bird's-eye view includes: performing image transformation on each bird's-eye view based on the fourth deformable grid corresponding to each fisheye camera to obtain deformed images of each bird's-eye view; determining the coordinate range of a unified canvas based on the coordinate range of the deformed images; and mapping each deformed image to the unified canvas to obtain a panoramic image.

[0035] The above method can achieve the acquisition of complete panoramic images with consistent geometric scales with extremely low computational overhead, thereby improving the image stitching quality.

[0036] In some embodiments, mapping each of the deformed images to the unified canvas to obtain a panoramic image includes: if a single deformed image is mapped to a pixel position on the unified canvas, determining the pixel data of the pixel position based on the pixel data of the deformed image at the pixel position; if multiple deformed images are mapped to pixel positions on the unified canvas, determining the weight of each deformed image at the pixel position based on the position of the pixel position in each deformed image, and performing weighted fusion of the pixel data of each deformed image at the pixel position based on the weight of each deformed image at the pixel position to obtain the pixel data of the pixel position.

[0037] The above method allows all pixel positions on a unified canvas to be processed separately according to their coverage. Non-overlapping areas are directly valued to reduce computational overhead, while overlapping areas are weighted and fused to achieve a smooth transition, thereby improving image stitching quality.

[0038] In some embodiments, determining the weight of each deformed image at the pixel position based on the position of the pixel position in each deformed image includes: determining the weight of each deformed image at the pixel position based on the distance from the pixel position to the effective region boundary of each deformed image on the uniform canvas; wherein the distance is positively correlated with the weight.

[0039] The above method allows pixel data at all pixel positions on a unified canvas to transition in a position-dependent and continuous manner, achieving a smooth transition, reducing stitching gaps, and improving image stitching quality.

[0040] In some embodiments, determining the second homography matrix corresponding to each fisheye camera based on each of the bird's-eye views includes: determining the motion parameters of the ship based on each of the bird's-eye views; and determining the second homography matrix corresponding to each fisheye camera based on the motion parameters of the ship and the relative poses between the fisheye cameras.

[0041] In some embodiments, determining the motion parameters of the ship based on each of the bird's-eye views includes: performing global average pooling on each of the bird's-eye views to obtain an average pooling result; concatenating the average pooling results to obtain a global feature vector; and determining the motion parameters of the ship based on the global feature vector.

[0042] In some embodiments, determining the second homography matrix corresponding to each fisheye camera based on the motion parameters of the ship and the relative poses between the fisheye cameras includes: determining the rotation matrix of the reference camera at the current moment based on the rotation matrix of the reference camera in the previous moment and the motion parameters of the ship; determining the rotation matrix of the fisheye cameras other than the reference camera at the current moment based on the rotation matrix of the fisheye cameras other than the reference camera relative to the reference camera; and determining the second homography matrix of each fisheye camera based on the rotation matrix of each fisheye camera at the current moment, the intrinsic parameter matrix of each fisheye camera, the translation vector of the fisheye camera relative to the reference camera, the normal vector of the reference plane, and the distance from the reference plane to the reference camera.

[0043] In some embodiments, before determining the second homography matrix corresponding to each fisheye camera based on each bird's-eye view, the method further includes: extracting multi-scale features of the bird's-eye view using a feature extraction network to obtain a feature map of the bird's-eye view; determining the second homography matrix corresponding to each fisheye camera based on each bird's-eye view; and determining a second projection map based on each bird's-eye view and the corresponding second homography matrix, including: determining the second homography matrix corresponding to each fisheye camera based on the feature map of each bird's-eye view; and determining a second projection feature map based on the feature map of each bird's-eye view and the corresponding second homography matrix.

[0044] In some embodiments, before determining the second homography matrix corresponding to each fisheye camera based on the feature maps of each bird's-eye view, the method further includes: determining a weak texture region based on the feature maps of the bird's-eye view; performing cross-camera attention interaction on the weak texture regions in the overlapping regions of adjacent fisheye cameras to obtain a fused and enhanced weak texture region; and obtaining an enhanced feature map based on the feature maps and the fused and enhanced weak texture regions.

[0045] In some embodiments, performing cross-camera attention interaction on weak texture regions in the overlapping regions of adjacent fisheye cameras to obtain fused and enhanced weak texture regions includes: performing cross-camera attention interaction on weak texture regions in the overlapping regions of adjacent fisheye cameras in a first direction to obtain a fused and enhanced first weak texture region; and performing cross-camera attention interaction on weak texture regions in the overlapping regions of adjacent fisheye cameras in a second direction to obtain a fused and enhanced second weak texture region; wherein the first direction and the second direction are opposite in direction.

[0046] Thirdly, this application provides a fisheye image stitching system, comprising: fisheye cameras installed at different locations on a ship for acquiring fisheye images; and a processor communicatively connected to each of the fisheye cameras for implementing the steps of the fisheye image stitching method as described in any of the above claims.

[0047] Fourthly, this application provides a ship including the fisheye image stitching system described above.

[0048] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0049] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiments below. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart of a method for training an offset prediction network according to some embodiments of this application; Figure 2 This is a layout diagram of a ship's lateral multi-fisheye camera according to some embodiments of this application; Figure 3 This is a flowchart illustrating the construction of deformable mesh geometric loss functions in some embodiments of this application; Figure 4 This is a flowchart illustrating the determination of the cross-camera consistency loss function in some embodiments of this application; Figure 5 A flowchart illustrating the determination of the sea surface level constraint loss function for some embodiments of this application; Figure 6 A flowchart for determining the mesh temporal smoothing loss function provided in some embodiments of this application; Figure 7 A flowchart for determining the first homography matrix for some embodiments of this application; Figure 8 This is a flowchart of a fisheye image stitching method according to some embodiments of this application; Figure 9 This is a flowchart illustrating the process of obtaining panoramic images in some embodiments of this application; Figure 10 This is a diagram of the cross-camera feature interaction network structure for some embodiments of this application. Detailed Implementation

[0050] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0052] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0053] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0054] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0055] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0056] In the description of the embodiments of this application, the technical terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this application and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0057] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0058] Rigid constraints refer to multiple cameras fixedly mounted on the same rigid body, with the relative poses of the cameras remaining constant, and all cameras sharing the same rigid body motion (6 degrees of freedom).

[0059] Cross-camera attention is an attention mechanism that allows features from one camera to interact with features from another camera to obtain complementary information across different viewpoints.

[0060] Global rigid motion: The hull moves as a rigid body with 6 degrees of freedom, including 3-axis rotation (roll, pitch, yaw) and 3-axis translation (dx, dy, dz).

[0061] Rigidly constrained deformation: Local deformation under the guidance of global rigid motion prior. The deformed mesh is constrained by rigid constraint loss and is not allowed to deviate too far from the global motion prediction.

[0062] Cross-camera consistency: The deformation mapping of adjacent cameras in the overlapping area must be consistent, that is, the pixel mapping relationship of the same 3D point in the views of two cameras must be compatible.

[0063] Sea surface prior: The sea surface can be approximated as a horizontal plane (Z=0). This geometric prior is used to constrain the deformation of the sea surface region.

[0064] Overlapping area mask: A mask pre-calculated by calibration parameters that identifies the area in each camera view that overlaps with the adjacent camera.

[0065] With the development of the shipping industry and the trend towards larger ships, the safe and efficient operation of ships, as the core carriers of the shipping industry, is of great significance to ensuring the stability of the global supply chain. However, as the trend towards larger and more specialized ships becomes increasingly apparent, the difficulty and risks of offshore berthing operations are also increasing. Especially in narrow waters, strong winds, strong currents, at night, or in severe weather conditions, it is difficult for operators to accurately judge the distance and relative position of the ship to the dock, other ships, and obstacles, which can easily lead to collisions, causing casualties, property damage, and environmental pollution.

[0066] Currently, a common surround-view system for ships involves installing a fisheye camera on each of the four sides of the hull, forming a four-camera surround-view system: a bow camera (C_front), a stern camera (C_rear), a port camera (C_left), and a starboard camera (C_right). Each of the four cameras covers approximately 180 degrees of field of view.

[0067] Traditional multi-fisheye camera stitching involves fusing two images from each pair of adjacent cameras (e.g., port side + bow). While simple, this method has blind spots. A more advanced approach uses a cascaded stitching method, breaking down the stitching of X cameras into X-1 rounds of two-camera stitching (e.g., C0+C1, ​​C1+C2, C2+C3). Each round uses a deep learning network independently for spatial alignment, temporal stabilization, and smoothing optimization. Examples include two-stage alignment using a correlation layer and cost volume, TPS (Thin Plate Spline) transformation, and 3D convolutional temporal smoothing. Alternatively, SuperPoint + LightGlue can be used for feature extraction and matching, generating a fusion mask through an unsupervised network, supporting only two-image stitching.

[0068] Because each stitching round is processed independently, errors accumulate with each round, leading to stitching misalignment and reduced image stitching quality. Specifically, in pairwise cascading stitching, errors from each round accumulate in subsequent rounds, causing increasingly larger alignment offsets at the remote cameras. For example, in stitching four cameras C0-C3, the alignment error of C0+C1 will propagate to C1+C2, then to C2+C3, and ultimately the alignment deviation of C3 may far exceed acceptable limits.

[0069] To address this, the applicant proposes a migration prediction network training method, a fisheye image stitching method, a system, and a vessel. During the training phase, a deformable mesh migration prediction network is used to predict the first local deformation offset corresponding to each fisheye camera. Based on the first local deformation offset corresponding to each fisheye camera, the first deformable mesh, each first homography matrix, and the overlapping area of ​​adjacent fisheye cameras, a deformable mesh geometric loss function is constructed. This function updates the deformable mesh migration prediction network, enabling joint constraints on the deformation parameters corresponding to all fisheye cameras during the training phase. This ensures that the deformable meshes corresponding to each fisheye camera are geometrically consistent, and allows the trained deformable mesh migration prediction network to output the local deformation offset corresponding to each fisheye camera in parallel. This enables a one-time global joint stitching of all fisheye cameras, thus solving the error accumulation problem of pairwise cascaded stitching, reducing the probability of image stitching misalignment, and improving the quality and accuracy of panoramic fisheye image stitching.

[0070] See Figure 1 This is a flowchart of a bias prediction network training method according to some embodiments of this application, which may include the following steps: S11: Obtain bird's-eye view samples corresponding to multiple fisheye cameras on the ship.

[0071] In some embodiments of this application, multiple fisheye cameras may be installed on the ship. A fisheye camera is an image acquisition device that uses a fisheye lens to achieve ultra-wide field-of-view imaging. Its field of view is typically greater than 180°, enabling it to capture a wide range of environmental images. The fisheye images acquired are distorted; straight lines in the edge regions of the fisheye image are rendered as curves. In some embodiments, one fisheye camera can be installed around each of the ship's four sides, forming a four-camera surround-view system. In some embodiments, considering that traditional four-camera surround-view systems can provide 360-degree surround view but have blind spots in lateral berthing scenarios, the lateral (port and starboard) cameras of the ship can be expanded. For example, a single camera on each side can be expanded into multiple fisheye cameras (typically 4-8) arranged in an arc along the ship's hull, forming a lateral multi-camera array to reduce the degree of lateral blind spots. This allows the operator to clearly observe key facilities such as the dock's edge, mooring bollards, and anti-collision tires, and accurately determine the distance between the ship and the dock.

[0072] In some embodiments, the number and layout of fisheye cameras on a ship can be determined according to the ship type, size, and berthing requirements. Specifically, N fisheye cameras can be arranged around the hull. The camera layout includes, but is not limited to, the following configurations: (1) Six-camera mode: six directions including bow, stern, upper port side, lower port side, upper starboard side, and lower starboard side; (2) Eight-camera mode: two cameras are arranged in each of the four directions of bow, stern, port side, and starboard side to achieve full vertical coverage; (3) More-camera mode: flexible configuration based on field-of-view coverage requirements and ship size. In some embodiments, the number of lateral cameras can be determined according to the ship type and size, and M (M≥3) fisheye cameras can be arranged along the arc of the hull on the port side and / or starboard side, with the field-of-view overlap rate of adjacent fisheye cameras not less than 30%, to improve the reliability of cross-camera feature matching. Specifically, see [link to relevant documentation]. Figure 2 This is a layout diagram of a ship's lateral multi-fisheye camera according to some embodiments of this application. It should be noted that... Figure 2 The following example illustrates the use of four fisheye cameras arranged laterally on a ship. It should be noted that some embodiments of this application support flexible configuration and splicing of any number of fisheye cameras.

[0073] In some embodiments of this application, bird's-eye view samples corresponding to each fisheye camera on the ship can be obtained. These bird's-eye view samples are image samples presenting the environment within the field of view of the fisheye camera from a top-down perspective. Specifically, the bird's-eye view samples can be obtained by performing fisheye distortion correction and perspective projection transformation on the original fisheye images acquired by the fisheye cameras. The original fisheye images are the raw image data directly acquired and output by the fisheye cameras without distortion correction and perspective projection transformation processing. It should be noted that in some embodiments, the original fisheye images and corresponding bird's-eye view samples can be images acquired and processed before the ship's actual voyage, so as to facilitate the pre-training of a deformable mesh offset prediction network. This allows for the direct panoramic stitching of fisheye images based on the trained deformable mesh offset prediction network when the ship is actually put into use. Alternatively, in some embodiments, the original fisheye images and corresponding bird's-eye view samples can be images acquired and processed during the initial stage of the ship's commissioning, so as to facilitate the training of the deformable mesh offset prediction network. Alternatively, in some embodiments, the original fisheye image and the corresponding bird's-eye view sample can be images collected and processed during the actual use of the ship, so as to update the trained deformable mesh offset prediction network accordingly, thereby improving the accuracy and precision of the deformable mesh offset prediction network, and further improving the fisheye image stitching quality.

[0074] Specifically, the intrinsic parameter matrix of each fisheye camera can be pre-calibrated and obtained. : ,in, and Let be the focal length (in pixels) of fisheye camera i (i.e., the i-th fisheye camera) in the x and y directions. and Let the coordinates of the principal point of the i-th fisheye camera be . Specifically, the x-coordinate of the principal point (the intersection of the optical axis and the image plane). Specifically, the y-coordinate of the principal point. Furthermore, the fisheye distortion polynomial model can be pre-calibrated and obtained: ,in, Let be the angle between the incident ray and the optical axis. The angle after distortion. , , , This represents the radial distortion coefficient. Additionally, the relative pose between adjacent fisheye cameras can be pre-calibrated. (4×4 homogeneous transformation matrix): ;in, It is a 3×3 rotation matrix that describes the pose relationship between the (i+1)th fisheye camera and the ith fisheye camera. This is a 3×1 translation vector, describing the position of the origin of the (i+1)th fisheye camera in the coordinate system of the ith fisheye camera. The aforementioned relative pose can be obtained through offline calibration and remains unchanged during operation (rigid constraint).

[0075] Based on the above, the intrinsic parameter matrix can be used. and distortion coefficient (Specifically may include) , , , The original fisheye image undergoes distortion correction. Then, the distortion-corrected image is projected onto a bird's-eye view using perspective projection transformation to obtain projected images from multiple bird's-eye view perspectives (i.e., bird's-eye view samples). The perspective projection transformation matrix corresponds to this transformation. It can be determined by camera extrinsic parameters and target viewpoint.

[0076] That is, the above process can be represented as:

[0077] in, For fisheye distortion correction function, The raw fisheye image captured by fisheye camera i. For perspective projection transformation function, This is a bird's-eye view sample corresponding to fisheye camera i.

[0078] The above process converts the original fisheye images captured by the various fisheye cameras on the ship from different perspectives into bird's-eye view samples from a top-down perspective, providing a data basis for subsequently determining the first homography matrix and predicting the first local deformation offset.

[0079] S12: Determine the first homography matrix corresponding to each fisheye camera based on each bird's-eye image sample, and determine the first projected image based on each bird's-eye image sample and the corresponding first homography matrix.

[0080] Based on acquiring bird's-eye view samples corresponding to multiple fisheye cameras, a first homography matrix corresponding to each fisheye camera can be determined according to each bird's-eye view sample. The first homography matrix is ​​a matrix representing the projection transformation relationship between two planes, used to map pixel coordinates in the bird's-eye view sample to corresponding pixel coordinates in the projected image. For example, the first homography matrix can be determined based on the corresponding point pairs between the bird's-eye view sample corresponding to the fisheye camera and a preset reference plane, to characterize the projection mapping relationship from the bird's-eye view sample to the reference plane. Alternatively, the ship's motion parameters can be determined based on the bird's-eye view samples corresponding to the fisheye cameras, and the first homography matrix corresponding to each fisheye camera can be determined based on the ship's motion parameters and the pre-calibrated relative poses between adjacent fisheye cameras.

[0081] In some embodiments, the first homography matrix can specifically be the homography matrix of each fisheye camera relative to a reference camera. In this case, the first homography matrix is ​​used to describe the mapping relationship between the image plane where the bird's-eye sample corresponding to the fisheye camera is located and the coordinate system of the reference camera. The reference camera can be used to provide a common coordinate system, which can be a two-dimensional coordinate system established based on the image plane where the bird's-eye sample corresponding to the reference camera is located. The reference camera is a pre-determined fisheye camera among multiple fisheye cameras. For example, the first fisheye camera can be designated as the reference camera, or a fisheye camera located at the midships of the ship can be used as the reference camera, or a fisheye camera with good imaging quality or centered field of view can be designated as the reference camera, etc. In some embodiments, the coordinate system of the reference camera can be the same as the panoramic coordinate system, that is, the coordinate system of the reference camera can be used as the panoramic coordinate system, which is the coordinate system corresponding to the panoramic image obtained by stitching the fisheye image with the corresponding bird's-eye image. Alternatively, the coordinate system of the reference camera and the panoramic coordinate system can also have an overall translation, overall rotation, or overall scaling, etc.

[0082] Based on the determined first homography matrix corresponding to each fisheye camera, for each fisheye camera, a first projected image can be determined according to the bird's-eye image sample corresponding to that fisheye camera and the first homography matrix corresponding to that fisheye camera. Specifically, the first projected image can be obtained by projecting the bird's-eye image sample using the first homography matrix, that is, the first projected image is the projected image obtained by mapping the bird's-eye image sample to the corresponding coordinate system according to the corresponding first homography matrix. For example, if the first homography matrix is ​​the homography matrix of each fisheye camera relative to a reference camera, then the bird's-eye image sample can be mapped to the coordinate system of the reference camera through the first homography matrix.

[0083] The above process enables coarse alignment of the bird's-eye view samples from each fisheye camera using the first homography matrix corresponding to each camera, mapping all the bird's-eye view samples to the same coordinate system, allowing direct comparison and overlay of the first projected images from different fisheye cameras. Furthermore, since the bird's-eye view samples from each fisheye camera are mapped and coarsely aligned using the same coordinate system, errors in the coarse alignment stage are not propagated through each iteration.

[0084] S13: Determine the first deformed mesh based on the bird's-eye view samples. Based on each bird's-eye view sample, each first projected image, and the deformed mesh offset prediction network, predict the first local deformation offset of the first deformed mesh and each fisheye camera. The deformed mesh offset prediction network has initial parameters.

[0085] Considering that the first homography matrix provides coarse alignment, objects such as containers and mooring bollards on the dock are roughly aligned, but non-rigid factors such as water waves and dynamic objects (such as small boats and floating objects) cause local misalignment, a deformable mesh offset prediction network can be used for fine adjustment.

[0086] Specifically, the first deformable mesh can be determined based on the bird's-eye view sample. For example, a regular mesh (which is the first deformable mesh) can be defined on the bird's-eye view sample or the corresponding first projected image. That is, the first deformable mesh is a regular mesh defined on the bird's-eye view sample or the corresponding first projected image, composed of uniformly distributed mesh vertices. In some embodiments, the number of first deformable meshes can be one, meaning each fisheye camera can correspond to the same first deformable mesh. For example, the first deformable mesh can be determined based on the bird's-eye view sample corresponding to the reference camera or the corresponding first projected image. For example, the first deformable mesh can be obtained by dividing the bird's-eye view sample or the first projected image according to a preset number of rows and columns. The first deformable mesh contains multiple mesh vertices and mesh edges connecting the mesh vertices, and each mesh vertex is uniformly distributed in the image plane corresponding to the bird's-eye view sample or the first projected image. Through the aforementioned process, sparse mesh vertices can be used as deformation control points to provide the target for the first local deformation offset. The coordinate system of the first deformed mesh can be the same as the coordinate system of the first projected image, both of which can be the coordinate system of the reference camera, and in some embodiments, both can be the panoramic coordinate system.

[0087] Furthermore, based on the bird's-eye view samples corresponding to each fisheye camera, the first projected image corresponding to each fisheye camera, and the deformable mesh offset prediction network, the first local deformation offset of the first deformable mesh relative to each fisheye camera can be predicted. That is, the first local deformation offset of the first deformable mesh corresponding to that fisheye camera is determined based on the bird's-eye view samples and the first projected image. The deformable mesh offset prediction network is a neural network that takes the bird's-eye view samples and the corresponding first projected image as input and outputs the offset of each grid vertex in the first deformable mesh. In other words, the deformable mesh offset prediction network is a neural network model used to predict the spatial offset of each grid vertex, thereby handling local non-rigid factors that cannot be captured by the globally rigid operation. For example, the deformable mesh offset prediction network can use a 5-layer convolutional structure (of course, other layers of convolutional structures are also possible, depending on the number of grid vertices, etc.), outputting 2-channel offsets (x-direction and y-direction). Alternatively, the deformable mesh offset prediction network can use an encoder-decoder structure, etc. The first local deformation offset is the two-dimensional position offset of each grid vertex in the first deformed grid relative to the position of the regular grid, predicted by the deformed grid offset prediction network. In other words, the first local deformation offset represents the local offset of each grid vertex relative to the position of the first deformed grid. The physical meaning of the first local deformation offset is: local position correction caused by local non-rigid factors (such as water waves, dynamic floating objects, etc.) that cannot be captured by global rigid motion.

[0088] The deformable mesh migration prediction network has initial parameters, which are the parameters it possesses before being updated, i.e., the parameters it has at the start of the current iteration. Specifically, in the first iteration, the initial parameters of the deformable mesh migration prediction network can be set through initialization. For example, they can be set using random initialization (e.g., Gaussian distribution, uniform sampling distribution, etc.), Xavier / Glorot initialization, or He / Kaiming initialization. In iterations other than the first, the initial parameters of the deformable mesh migration prediction network are the parameters updated in the previous iteration. That is, for each iteration, the updated parameters obtained by updating the initial parameters of the deformable mesh migration prediction network according to the deformable mesh geometric loss function can be used as the initial parameters of the deformable mesh migration prediction network in the next iteration.

[0089] In some embodiments, the deformable mesh offset prediction network can take a coarsely aligned first projected image and a bird's-eye view sample as input to predict the required offset of each mesh vertex in the first deformable mesh. Exemplarily, the deformable mesh offset prediction network can, based on... Predict the offset of each mesh vertex (i.e., the first local deformation offset), where, Let H be the first local deformation offset corresponding to the i-th fisheye camera, and its size can be H. grid ×W grid ×2, H grid W represents the number of grid vertices in the height direction of the first deformed mesh. grid This indicates the number of grid vertices in the width direction of the first deformed mesh, where 2 indicates one channel each in the x and y directions. The first local deformation offset indicates the offset of each grid vertex relative to the position of the first deformed mesh. For deformable mesh offset prediction network; Indicates a splicing operation; The first projected image corresponding to the i-th fisheye camera; Let be the bird's-eye view sample corresponding to the i-th fisheye camera. After stitching the first projected image and the bird's-eye view sample together, the deformable mesh offset prediction network can jointly process the two images during the feature extraction stage. Starting from the shallow features, it can perceive the misalignment direction and magnitude between the two images at the edges and textures, thereby obtaining the accurate first local deformation offset.

[0090] The position of each grid vertex in the first deformable mesh can be adjusted by the first local deformation offset corresponding to each fisheye camera, thereby causing local deformation of the first projected image and compensating for local nonlinear deformations that the first homography matrix cannot describe. Furthermore, since the first local deformation offset acts on the grid vertices, the output dimension of the deformable mesh offset prediction network is on the order of the grid vertices, which can reduce prediction difficulty and computational overhead, and improve deformation smoothness.

[0091] S14: Construct the deformable mesh geometric loss function based on the first local deformation offset, the first deformable mesh, the first homography matrix, and the overlapping area of ​​adjacent fisheye cameras corresponding to each fisheye camera.

[0092] In some embodiments, the overlapping region of adjacent fisheye cameras can be predetermined. Adjacent fisheye cameras are two fisheye cameras that are adjacent in the arrangement direction of the plurality of fisheye cameras and whose fields of view overlap. The overlapping region of adjacent fisheye cameras refers to the region where the fields of view of two adjacent fisheye cameras overlap; that is, the overlapping region of adjacent fisheye cameras identifies the pixel area commonly visible in the views of the two adjacent fisheye cameras, and can be pre-generated by calibration parameters through projection calculation. In some embodiments, the overlapping region of adjacent fisheye cameras can be represented as an overlapping region mask. The mask is a binary image that identifies the pixel region that is visible in the views of the i-th fisheye camera and the (i+1)-th fisheye camera.

[0093] Based on the above, a deformable mesh geometric loss function can be constructed according to the first local deformation offset, the first deformable mesh, the first homography matrix, and the overlapping area of ​​adjacent fisheye cameras for each fisheye camera. The deformable mesh geometric loss function is a function constructed based on the first local deformation offset, the first deformable mesh, the first homography matrix, and the overlapping area of ​​adjacent fisheye cameras, used to measure the geometric consistency within the overlapping area of ​​adjacent fisheye cameras.

[0094] For example, the rigid offset corresponding to each fisheye camera can be determined based on the first deformed mesh and the first homography matrix corresponding to each fisheye camera. A rigid constraint loss function is then determined based on the rigid offset corresponding to each fisheye camera and the first local deformed offset. Furthermore, a second deformed mesh corresponding to each fisheye camera can be determined based on the first deformed mesh and the first local deformed offset corresponding to each fisheye camera. A cross-camera consistency loss function is then determined based on the second deformed mesh corresponding to each fisheye camera and the overlapping area of ​​adjacent fisheye cameras. Finally, a deformed mesh geometric loss function is determined based on the rigid constraint loss function and the cross-camera consistency loss function.

[0095] The above method enables joint constraints on the first local deformation offsets corresponding to each fisheye camera when constructing the deformable mesh geometric loss function. This ensures that the first local deformation offset includes the loss terms corresponding to each adjacent fisheye camera, allowing the value of the first local deformation offset for any fisheye camera to be constrained by the other fisheye cameras. This prevents errors from propagating sequentially along the stitching link, thereby establishing global consistency during the training phase, reducing stitching misalignment and accumulated errors, and improving image stitching quality. Specifically, some embodiments of this application perform rigid constraint joint optimization. By utilizing the physical prior of multiple fisheye cameras fixed on the same rigid body, the stitching problem of multiple fisheye cameras is jointly optimized into a globally consistent problem.

[0096] S15: Update the initial parameters of the deformable mesh offset prediction network according to the deformable mesh geometric loss function.

[0097] Based on the constructed deformable mesh geometric loss function, the initial parameters of the deformable mesh offset prediction network can be updated according to the deformable mesh geometric loss function to train the deformable mesh offset prediction network and make the output of the deformable mesh gradually tend to satisfy the geometric constraints. That is, the deformable mesh geometric loss function can be used as the optimization objective, and the gradient of the optimization objective with respect to each initial parameter of the deformable mesh offset prediction network can be calculated through the backpropagation algorithm. The initial parameters of the deformable mesh offset prediction network can then be iteratively adjusted according to the gradient, so that the first local deformable offset of the output of the deformable mesh offset prediction network can gradually satisfy the geometric constraints represented by the deformable mesh geometric loss function in subsequent forward propagation. In other words, the deformable mesh geometric loss function acts on the deformable mesh offset prediction network, and the network parameters are jointly optimized through backpropagation with the deformable mesh geometric loss function as a global consistency constraint. For example, the gradient of the deformable mesh geometric loss function with respect to the initial parameters of the deformable mesh offset prediction network can be calculated through the backpropagation algorithm and backpropagated to the deformable mesh offset prediction network. The optimizer can then iteratively update the weights and biases of the deformable mesh offset prediction network according to the gradient to minimize the deformable mesh geometric loss function.

[0098] In this method, since the deformable mesh geometric loss function is composed of the loss terms corresponding to each fisheye camera, the update direction of the mesh parameters is jointly determined by the error conditions of each fisheye camera. Therefore, the first local deformable offsets output by each fisheye camera in the subsequent forward propagation process of the trained deformable mesh offset prediction network are coordinated with each other. This reduces error accumulation and stitching error probability when achieving global joint stitching, thereby improving the fisheye image quality. In other words, compared with the traditional pairwise cascaded stitching method, the deformable mesh geometric loss function in some embodiments of this application simultaneously constrains the geometric consistency of all adjacent fisheye cameras during training, rather than processing them independently round by round. Therefore, it can guarantee the geometric consistency of each fisheye camera in the panoramic image from a global perspective, solving the problems of traditional round-by-round error accumulation and stitching misalignment.

[0099] By updating the initial parameters of the deformable mesh offset prediction network, the network can output more accurate local deformation offsets during subsequent forward propagation, thereby generating more accurate deformable meshes. This improves the quality of fisheye image stitching and solves the problems of error accumulation and stitching misalignment caused by traditional pair-by-pair cascade stitching.

[0100] After training, the network parameters of the deformable mesh offset prediction network have learned to output the corresponding deformable mesh under the given constraints, which can then be used for image stitching. Since all fisheye cameras share the same deformable mesh offset prediction network, and this network has established global consistency constraints during training, the local deformable offsets of each fisheye camera can be coordinated with each other during the image stitching stage, significantly reducing stitching misalignment and cumulative errors.

[0101] See Figure 3 This is a flowchart illustrating the construction of a deformable mesh geometric loss function according to some embodiments of this application. According to some embodiments of this application, constructing a deformable mesh geometric loss function based on the first local deformation offset corresponding to each fisheye camera, the first deformable mesh, each first homography matrix, and the overlapping area of ​​adjacent fisheye cameras may include: determining the rigid offset corresponding to each fisheye camera based on the first deformable mesh and each first homography matrix; determining a rigid constraint loss function based on the rigid offset corresponding to each fisheye camera and the first local deformation offset corresponding to each fisheye camera; determining the second deformable mesh corresponding to each fisheye camera based on the first deformable mesh and the first local deformation offset corresponding to each fisheye camera; determining a cross-camera consistency loss function based on the second deformable mesh corresponding to adjacent fisheye cameras and the overlapping area of ​​adjacent fisheye cameras; and constructing the deformable mesh geometric loss function based on the rigid constraint loss function and the cross-camera consistency loss function.

[0102] In some embodiments of this application, a rigid constraint loss function and a cross-camera consistency loss function can be constructed, and a deformable mesh geometric loss function can be constructed based on the rigid constraint loss function and the cross-camera consistency loss function.

[0103] In this regard, considering that the first local deformation offset should not deviate too far from the global rigid motion, otherwise it will destroy the rigid constraint, the rigid offset can be derived from the global homography, that is, the mapping offset of the first homography matrix to the mesh vertices of the first deformed mesh, and the rigid constraint loss function can be defined as the difference between the first local deformation offset and the rigid offset.

[0104] Specifically, for constructing the rigid constraint loss function, the rigid offset corresponding to each fisheye camera can be determined first based on the first homography matrix and the first deformed mesh corresponding to each fisheye camera. That is, the rigid offset is the positional shift of each mesh vertex in the first deformed mesh under the mapping relationship described by the first homography matrix corresponding to the fisheye camera. In other words, the rigid offset is the offset of the mesh vertex when only considering global rigid operation (excluding local deformation). Its physical meaning is: if the mesh vertex is completely driven by the first homography matrix, where should it move from its position in the first deformed mesh? This movement is the rigid offset. The calculation method is as follows: for each fisheye camera, the first deformed mesh is transformed through the first homography matrix corresponding to that fisheye camera. The difference between the transformed position and the original position is the rigid offset. The specific calculation steps are as follows: Step 101: The vertices of the first deformed mesh Convert to homogeneous coordinates:

[0105] in, This represents the k-th vertex of the first deformed mesh. This represents the coordinates of the k-th grid vertex.

[0106] Step 102: Using the first homography matrix Transform the homogeneous coordinates obtained by the transformation using (the first homography matrix corresponding to the i-th fisheye camera):

[0107] in, and These are the first two components of the homogeneous coordinates obtained after transformation using the first homography matrix. This is the third component (homogeneous component / scaling factor) of the homogeneous coordinates obtained after transformation through the first homography matrix.

[0108] Step 103: Normalize the transformed homogeneous coordinates to obtain the rigidly mapped position. :

[0109] Step 104: Based on the rigid mapping position Calculate the rigid offset:

[0110] in, For the i-th fisheye camera, this is the rigid offset corresponding to the k-th mesh vertex.

[0111] After determining the rigid offset corresponding to each fisheye camera, a rigid constraint loss function can be determined based on the rigid offset and the first local deformation offset corresponding to each fisheye camera. This rigid constraint loss function measures the degree of deviation of the first local deformation offset from the rigid offset. Specifically, the rigid residual corresponding to each mesh vertex can be determined first based on the rigid offset and the first local deformation offset of each fisheye camera. The rigid residuals of each mesh vertex are then summed or averaged to obtain the rigid constraint loss term corresponding to each fisheye camera. Then, the rigid constraint loss terms corresponding to each fisheye camera are summed or averaged to obtain the rigid constraint loss function.

[0112] In some embodiments, for the i-th fisheye camera, the rigid residual corresponding to each mesh vertex can be determined based on the distance between the offset position of each mesh vertex indicated by the first local deformation offset and the mapped position of each mesh vertex indicated by the rigid offset. For example, the rigid residual can be the position corresponding to the first local deformation offset, the Euclidean distance between the positions corresponding to the corresponding rigid offset, or the square of the Euclidean distance. The rigid residuals of each mesh vertex are averaged to obtain the rigid constraint loss term corresponding to the i-th fisheye camera; the rigid constraint loss terms corresponding to each fisheye camera are averaged to obtain the rigid constraint loss function.

[0113] In some embodiments, the rigid constraint loss function can be calculated as follows:

[0114] in, Represents the loss function under rigid constraints. Indicates the number of fisheye cameras. This represents the total number of grid vertices. This indicates the number of grid vertices in the height direction of the first deformed mesh. This indicates the number of grid vertices in the width direction of the first deformed mesh. This represents the first local deformation offset of the k-th mesh vertex predicted by the deformable mesh offset prediction network (representing the local offset of the mesh vertex relative to the position of the first deformable mesh). , (Offsets in the x and y directions), equivalently, ,in, The second deformed mesh is obtained based on the first deformed mesh and the first local deformation offset, where the first local deformation offset is equal to the difference between the vertex positions of the second deformed mesh and the vertex positions of the first deformed mesh. This represents the square of the L2 norm.

[0115] The physical meaning of the rigid constraint loss function is: this loss measures the first local deformation offset predicted by the deformable mesh offset prediction network. With rigid offset The difference between them. When the two are exactly equal, This indicates that the deformable mesh is entirely driven by global rigid motion, with no local deformation; when the first local deformation offset... Deviation from rigid offset hour, This indicates the existence of local non-rigid deformation. The rigid constraint loss function allows for fine-tuning of local deformation within a globally rigid framework: for regions of rigid objects (docks, containers, etc.), Maintain global alignment; non-rigid areas (water waves, dynamic objects, etc.). Allowed deviation However, the degree of deviation is affected Constraints do not compromise global consistency. The rigid constraint loss function anchors local deformations to the global rigid motion prediction. Consistency is a priori guaranteed—local deformations can only be fine-tuned within the rigid framework, fundamentally disallowing deviations from global consistency. That is, after the first homography matrix provides coarse alignment, objects such as containers and mooring bollards on the dock are roughly aligned, but non-rigid factors such as water waves and dynamic objects (such as boats and floating objects) cause local misalignment. Fine-tuning is achieved by predicting local deformation meshes under rigid constraints (predicting the first local deformation offset and determining the second deformation mesh based on the first local deformation offset): rigid objects such as the dock maintain the alignment result of the global motion prediction, while non-rigid regions such as water waves are allowed to have small-scale offsets, but the degree of deviation is limited by the rigid constraint loss and will not compromise global consistency.

[0116] Furthermore, considering that the same object (e.g., the same mooring bollard on the dock in a berthing scene) may appear simultaneously in the images of two adjacent fisheye cameras (overlapping area), if the mapping position of the mooring bollard is inconsistent in the deformed mesh of adjacent fisheye cameras, ghosting or breaks will appear in the stitching result. To address this, cross-camera consistency constraints can be applied to force the mapping position of the same object in the overlapping area to be consistent, greatly reducing ghosting and making the boundary of the overlapping area transition smoothly, thus greatly reducing stitching gaps.

[0117] Specifically, for constructing the cross-camera consistency loss function, the second deformed mesh corresponding to each fisheye camera can be determined first based on the first deformed mesh and the first local deformation offset corresponding to each fisheye camera. Specifically, the second deformed mesh is the deformed mesh obtained by applying the corresponding first local deformation offset to each mesh vertex in the first deformed mesh. More specifically, it can utilize... Determine the second deformable mesh corresponding to each fisheye camera. Indicates the first deformed mesh. This represents the first local deformation offset corresponding to the i-th fisheye camera. This represents the second deformable mesh corresponding to the i-th fisheye camera. The second deformable mesh is used for pixel mapping during actual image stitching. It adaptively adjusts the first deformable mesh using a first local deformable offset to compensate for local parallax and distortion differences between fisheye cameras.

[0118] Then, a cross-camera consistency loss function can be determined based on the second deformable mesh corresponding to adjacent fisheye cameras and the overlapping region of adjacent fisheye cameras. The cross-camera consistency loss function is used to constrain the mapping consistency of adjacent fisheye cameras within the overlapping region, and to measure the difference in mapping coordinates between adjacent fisheye cameras within the overlapping region. For example, based on the second deformable mesh corresponding to adjacent fisheye cameras, the mapping coordinates of each pixel in the overlapping region of adjacent fisheye cameras can be determined. Then, the difference between the coordinates obtained by mapping the same pixel through the second deformable mesh of adjacent fisheye cameras is calculated, and a cross-camera consistency loss function is constructed based on this difference to constrain the local consistency of the overlapping region. Furthermore, a cross-camera consistency loss function can also be constructed based on the gradient difference of the second deformable mesh at the boundary of the overlapping region between adjacent fisheye cameras to constrain the deformation gradient of adjacent fisheye cameras at the boundary of the overlapping region to tend to be consistent, reducing the probability of abrupt changes in the deformable mesh at the stitching boundary, improving the visual continuity of the stitched image, and making the stitched image visually transition naturally.

[0119] Building upon the above, a deformable mesh geometric loss function can be constructed based on the rigid constraint loss function and the cross-camera consistency loss function. This means the rigid constraint loss function and the cross-camera consistency loss function can work collaboratively, embedding both into the optimization objective of the deformable mesh. This ensures the deformable mesh satisfies both rigid and cross-camera consistency constraints during the learning process, eliminating the need for post-processing correction steps in the subsequent inference stage. Specifically, the rigid constraint loss function and the cross-camera consistency loss function can be weighted and summed to obtain the deformable mesh geometric loss function. The weight coefficients corresponding to the rigid constraint loss function and the cross-camera consistency loss function can be determined through validation set parameter tuning. That is: ,in, For deformable mesh geometry loss function, For rigid constraint loss function, These are the weighting coefficients corresponding to the rigid constraint loss function. For cross-camera consistency loss function, These are the weighting coefficients corresponding to the cross-camera consistency loss function.

[0120] The above method can utilize rigid constraint loss function to achieve rigid constraints and cross-camera consistency loss function to achieve cross-camera consistency constraints. The two work together to ensure global consistency, reduce stitching errors, ghosting, and stitching gaps, and improve the stitching quality of fisheye images.

[0121] See Figure 4 This is a flowchart illustrating the determination of a cross-camera consistency loss function according to some embodiments of this application. According to some embodiments of this application, the cross-camera consistency loss function may include an overlapping region mapping consistency loss function. Determining the cross-camera consistency loss function based on the second deformable mesh corresponding to adjacent fisheye cameras and the overlapping regions of adjacent fisheye cameras may include: using the second deformable mesh corresponding to adjacent fisheye cameras, mapping each overlapping pixel in the overlapping region of adjacent fisheye cameras to a panoramic coordinate system to obtain a first panoramic pixel and a second panoramic pixel corresponding to each overlapping pixel; and determining the overlapping region mapping consistency loss function based on the first panoramic pixel and the second panoramic pixel corresponding to each overlapping pixel.

[0122] In some embodiments of this application, considering that the same object may appear simultaneously in the images of two adjacent fisheye cameras (overlapping area), if the mapping position of the mooring bollard by the deformed meshes of adjacent fisheye cameras is inconsistent, ghosting or breaks will occur in the stitching result. Therefore, the cross-camera consistency loss function may specifically include an overlapping area mapping consistency loss function, which requires that the mapping position of the same object in the overlapping area be consistent, thereby reducing ghosting. Specifically, the overlapping area mapping consistency loss function is a constraint term constructed for each overlapping pixel within the overlapping area of ​​adjacent fisheye cameras. It acts on the overlapping area, ensuring that the same position falls into the same position in the panoramic coordinate system after being mapped by the second deformed meshes corresponding to the two adjacent fisheye cameras.

[0123] Specifically, to determine the mapping consistency loss function for overlapping regions, the second deformed mesh corresponding to adjacent fisheye cameras can be used to map each overlapping pixel in the overlapping region of adjacent fisheye cameras to the panoramic coordinate system, thereby obtaining the first and second panoramic pixels corresponding to each overlapping pixel. Here, overlapping pixels are pixels located in the overlapping region of adjacent fisheye cameras, specifically pixels located in the overlapping region of bird's-eye view samples or the first projected image of adjacent fisheye cameras. The panoramic coordinate system is the coordinate system corresponding to the panoramic image obtained by stitching fisheye images; in some embodiments, the panoramic coordinate system can be the same as the coordinate system of the reference camera. The first panoramic pixel is obtained by mapping the overlapping pixel to the panoramic coordinate system via the second deformed mesh corresponding to the first fisheye camera among adjacent fisheye cameras, and the second panoramic pixel is obtained by mapping the overlapping pixel to the panoramic coordinate system via the second deformed mesh corresponding to the second fisheye camera among adjacent fisheye cameras. Specifically, for overlapping pixel p, the pixel coordinates of overlapping pixel p in the bird's-eye view sample corresponding to the i-th fisheye camera can be substituted into the second deformed mesh corresponding to the i-th fisheye camera. The first panoramic pixel is obtained. The pixel coordinates of the overlapping pixel p in the bird's-eye view sample corresponding to the (i+1)th fisheye camera are then substituted into the second deformed mesh corresponding to the (i+1)th fisheye camera. The second panoramic pixel is obtained. .

[0124] The mapping of overlapping pixels to the panoramic coordinate system can be calculated using bilinear interpolation. Specifically, for an overlapping pixel p, the grid cell (e.g., a rectangular cell) into which the pixel coordinates of p fall in the second deformed grid can be determined first. Then, based on the relative positions of the pixel coordinates of p with respect to the four vertices of that grid cell, bilinear interpolation is performed on the coordinates of these four vertices in the panoramic coordinate system to obtain the coordinates of the overlapping pixel p in the panoramic coordinate system.

[0125] Based on the above, an overlapping region mapping consistency loss function can be determined according to the first panoramic pixel and the second panoramic pixel corresponding to each overlapping pixel. This overlapping region mapping consistency loss function is used to measure the mapping consistency of adjacent fisheye cameras within the overlapping region. Specifically, for each overlapping pixel, the mapping consistency residual of the overlapping pixel can be determined according to the difference between the first panoramic pixel and the second panoramic pixel corresponding to that overlapping pixel. The mapping consistency residuals of each overlapping pixel are summed or averaged to obtain the mapping consistency constraint terms corresponding to adjacent fisheye cameras. The mapping consistency constraint terms corresponding to all two adjacent fisheye cameras are summed or averaged to obtain the overlapping region mapping consistency loss function.

[0126] In some embodiments, the consistency loss function for overlapping region mapping can be calculated as follows:

[0127] in, This represents the consistency loss function for overlapping region mapping. This represents the overlapping area between the i-th fisheye camera and the (i+1)-th fisheye camera. This indicates the total number of overlapping pixels within the overlapping area. express and The L2 norm of the difference. The physical meaning of the overlapping region mapping consistency loss function is: for the same pixel point (i.e., scene point) in the overlapping region, the panoramic coordinates obtained should be the same regardless of which fisheye camera's second deformable mesh is used for mapping.

[0128] Based on the above, the rigid constraint loss function and the overlapping region mapping consistency loss function can be weighted and summed to obtain the deformable mesh geometric loss function. The weight coefficients corresponding to the rigid constraint loss function and the overlapping region mapping consistency loss function can be determined through validation set parameter tuning. ,in, These are the weight coefficients corresponding to the consistency loss function for mapping overlapping regions.

[0129] The above method can constrain the mapping position within the overlapping area, making the mapping position of the same object within the overlapping area consistent, thereby maintaining local consistency in the overlapping area and reducing ghosting.

[0130] According to some embodiments of this application, the cross-camera consistency loss function may further include a boundary gradient consistency loss function. The cross-camera consistency loss function is determined based on the second deformed mesh corresponding to the adjacent fisheye camera and the overlapping region of the adjacent fisheye camera. This may include: determining the gradient of the second deformed mesh corresponding to the adjacent fisheye camera at the boundary of the corresponding overlapping region; and determining the boundary gradient consistency loss function based on the gradient of the second deformed mesh corresponding to the adjacent fisheye camera at the boundary of the corresponding overlapping region.

[0131] In some embodiments of this application, considering that constraining the mapping position consistency solely using the overlapping region mapping consistency loss function may lead to discontinuous jumps in the second deformed mesh at the boundary of the overlapping region, the gradient continuity of the second deformed mesh at the boundary of the overlapping region can be further constrained. Specifically, a boundary gradient consistency loss function can be further constructed, that is, the cross-camera consistency loss function can include not only the overlapping region mapping consistency loss function but also the cross-camera consistency loss function, so as to make the boundary of the overlapping region transition smoothly and reduce stitching gaps through the boundary gradient consistency loss function. The boundary gradient consistency loss function is a constraint term constructed for the boundary of the overlapping region of adjacent fisheye cameras, acting on the boundary line of the overlapping region, so that the gradient of the second deformed mesh of the two adjacent fisheye cameras has continuity at the boundary of the overlapping region.

[0132] Specifically, to determine the boundary gradient consistency loss function, we can first determine the gradient of the second deformed mesh corresponding to adjacent fisheye cameras at the corresponding overlapping region boundary. Here, the overlapping region boundary refers to the edge contour of the overlapping region of adjacent fisheye cameras. This overlapping region boundary can be obtained by edge extraction from the overlapping region mask or by projection of the field of view boundary of the adjacent fisheye cameras. Determining the gradient of the second deformed mesh corresponding to adjacent fisheye cameras at the corresponding overlapping region boundary involves calculating the rate of change of the coordinates of the mesh vertices in the spatial direction at the intersections of the two second deformed meshes with the overlapping region boundary. For example, for a boundary mesh vertex in the second deformed mesh, its gradient can be calculated using the finite difference method, that is, by dividing the coordinate difference between the boundary mesh vertex and its adjacent mesh vertices by the mesh spacing to obtain the gradient value at the boundary mesh vertex.

[0133] Based on the above, a boundary gradient consistency loss function can be determined according to the gradients of the second deformed meshes corresponding to adjacent fisheye cameras at the boundaries of the corresponding overlapping regions. This boundary gradient consistency loss function is used to measure the consistency of the deformation rate of adjacent fisheye cameras at the boundaries of the overlapping regions. Specifically, a boundary gradient constraint term can be determined according to the difference between the gradients of the second deformed meshes corresponding to adjacent fisheye cameras at the boundaries of the corresponding overlapping regions; the boundary gradient constraint terms corresponding to all two adjacent fisheye cameras are summed or averaged to obtain the boundary gradient consistency loss function.

[0134] In some embodiments, the boundary gradient consistency loss function can be calculated as follows:

[0135] in, Let be the gradient of the second deformed mesh corresponding to the i-th fisheye camera at the boundary of the corresponding overlapping region. Let be the gradient of the second deformed mesh corresponding to the (i+1)th fisheye camera at the boundary of the corresponding overlapping region. This represents the boundary gradient consistency loss function. The boundary gradient consistency loss function allows the second deformed mesh to smoothly transition at the boundary of the overlapping region between adjacent fisheye cameras.

[0136] The above method can constrain the gradient continuity of the second deformed mesh at the boundary of the overlapping region, so as to make the boundary of the overlapping region transition smoothly and reduce the splicing gap.

[0137] As can be seen from the above embodiments, the overlapping region mapping consistency loss function maintains local consistency in the overlapping region, while the boundary gradient consistency loss function achieves global consistency through smooth propagation of consistency along the camera chain. Both of these functions work in conjunction with the rigid constraint loss function and are embedded in the optimization objective of the deformable mesh migration prediction network. This ensures that the deformable mesh migration prediction network satisfies these constraints during the learning process, eliminating the need for post-processing correction steps during inference. Based on the above, constructing the deformable mesh geometric loss function based on the rigid constraint loss function and the cross-camera consistency loss function yields the deformable mesh geometric loss function. Specifically, the rigid constraint loss function, the overlapping region mapping consistency loss function, and the boundary gradient consistency loss function can be weighted and summed to obtain the deformable mesh geometric loss function. The weight coefficients corresponding to the rigid constraint loss function, the overlapping region mapping consistency loss function, and the boundary gradient consistency loss function can be determined through validation set parameter tuning. That is: ,in, and The weight coefficients are the corresponding weights for the consistency loss function for mapping overlapping regions and the consistency loss function for boundary gradients, respectively.

[0138] According to some embodiments of this application, it may further include: determining the corresponding sea surface area image based on each bird's-eye image sample; determining the sea surface plane constraint loss function based on the sea surface area image, the intrinsic parameter matrix of each fisheye camera, the sea level, the first deformable mesh, and the first local deformation offset corresponding to each fisheye camera; updating the initial parameters of the deformable mesh offset prediction network based on the deformable mesh geometric loss function, which may include: updating the initial parameters of the deformable mesh offset prediction network based on the deformable mesh geometric loss function and the sea surface plane constraint loss function.

[0139] In some embodiments of this application, a sea surface plane constraint loss function can be introduced during the training process of the deformable mesh offset prediction network. This function is a constraint term constructed using the known physical set of the sea surface plane as the supervision signal. The supervision signal comes from the intrinsic parameter matrix of each fisheye camera, the geometric relationship between the camera pose parameters and the sea surface, and calculates the ideal panoramic coordinates of the sea surface pixels through the camera pose parameters. This provides an absolute benchmark for the deformation results of each fisheye camera relative to the physical world. Moreover, the acquisition of this supervision signal does not depend on the texture features in the sea surface area.

[0140] To construct the sea surface plane constraint loss function, we can first determine the corresponding sea surface region image based on each bird's-eye view sample or each first projection image. This sea surface region image is used to represent the sea surface region in the bird's-eye view sample or the first projection image. Each bird's-eye view sample or the first projection image can have its own corresponding sea surface region image. The sea surface region image can be a binary mask image of the bird's-eye view sample or the first projection image, used to indicate whether each pixel in the bird's-eye view sample or the first projection image belongs to the sea surface. Alternatively, the sea surface region image can also be an image in the bird's-eye view sample or the first projection image that retains only the image content of the sea surface region and sets the image content outside the sea surface region to empty. For example, a lightweight segmentation head using 1x1 convolution + sigmoid activation can be used to output a sea surface binary mask from the bird's-eye view sample or the first projection image. : ,in, This indicates that pixel p belongs to the sea surface area. This indicates non-sea surface areas, which allows us to identify sea surface areas in the image.

[0141] Based on the above, the sea surface plane constraint loss function can be determined according to the sea surface area image, the intrinsic parameter matrix of each fisheye camera, the sea level, the first deformed mesh, and the first local deformation offset corresponding to each fisheye camera. The sea level is a horizontal plane with zero height in a preset world coordinate system, i.e., the Z=0 plane. In other words, some embodiments of this application can utilize the geometric prior that the sea surface is approximately horizontal (Z=0). Specifically, the world coordinate system can be established with the sea surface as the reference. For example, with a reference position on the sea surface as the origin, two mutually perpendicular directions within the horizontal plane as the horizontal and vertical axes, and the vertically upward direction as the vertical axis, the sea level is the set of points in the world coordinate system with a vertical axis coordinate of zero. The sea surface plane constraint loss function is a function determined based on the above parameters, used to measure the difference between the mapped position of a sea surface pixel after deformation within the sea surface area and the ideal panoramic coordinates calculated based on the sea surface geometric prior for that sea surface pixel. For example, the sea surface pixels in the sea surface region image can be mapped to the panoramic coordinate system based on the intrinsic parameter matrix of the fisheye camera and the sea level to obtain the corresponding ideal panoramic pixels. Furthermore, a second deformable mesh corresponding to the fisheye camera can be determined based on the first deformable mesh and the first local deformable offset corresponding to the fisheye camera. The sea surface pixels are then mapped using the second deformable mesh to obtain the panoramic sea surface pixels. Finally, a sea surface plane constraint loss function is determined based on the ideal panoramic pixels and the panoramic sea surface pixels.

[0142] Based on the above, when updating the initial parameters of the deformable mesh migration prediction network, the initial parameters can be updated according to the deformable mesh geometric loss function and the sea surface plane constraint loss function. Specifically, the deformable mesh geometric loss function and the sea surface plane constraint loss function can be weighted and summed, and the initial parameters of the deformable mesh migration prediction network can be updated using the total loss function obtained from the weighted summation. That is: ,in, For the total loss function, The loss function is the sea surface level constraint function. These are the weighting coefficients corresponding to the sea surface level constraint loss function, and these weighting coefficients can be determined through the validation set. As can be seen from the above, in some embodiments, In some embodiments, In some embodiments, .

[0143] The above method allows the deformable mesh geometric loss function and the sea surface plane constraint loss function to work together in the training of the deformable mesh offset prediction network. This enables the deformable mesh offset prediction network to accept both image-level and geometric-level constraints in a single training session. As a result, the deformable mesh offset prediction network can learn a deformation law that is consistent between the fisheye camera mappings in the overlapping area and conforms to the physical geometry in the sea surface area, thereby improving the image stitching quality.

[0144] See Figure 5 This is a flowchart illustrating the determination of the sea surface plane constraint loss function in some embodiments of this application. According to some embodiments of this application, determining the sea surface plane constraint loss function based on a sea surface region image, the intrinsic parameter matrices of each fisheye camera, the sea level, a first deformable mesh, and a first local deformable offset corresponding to each fisheye camera may include: mapping each sea surface pixel in the sea surface region image to a panoramic coordinate system based on the intrinsic parameter matrices of each fisheye camera and the sea level to obtain an ideal panoramic pixel corresponding to each sea surface pixel; determining a second deformable mesh corresponding to each fisheye camera based on the first deformable mesh and the first local deformable offset corresponding to each fisheye camera; mapping each sea surface pixel in the sea surface region image to a panoramic coordinate system using the second deformable mesh corresponding to each fisheye camera to obtain a panoramic sea surface pixel corresponding to each sea surface pixel; and determining the sea surface plane constraint loss function based on the ideal panoramic pixel and the panoramic sea surface pixel.

[0145] In some embodiments of this application, considering that in some ship navigation scenarios, the field of view of a fisheye camera is mostly the sea surface, for example, when berthing, about 50%-70% of the lateral field of view is the sea surface, and the sea surface texture is poor (especially at night or on calm water), traditional methods cannot align the sea surface area through feature matching. By utilizing the geometric prior that the sea surface is approximately horizontal (Z=0), the ideal panoramic coordinates of the sea surface pixels are directly calculated through the camera pose parameters, without relying on any texture features, and can be correctly aligned even if the sea surface has no texture at all.

[0146] Specifically, based on the intrinsic parameter matrix of each fisheye camera and the sea level, each sea surface pixel in the corresponding sea surface area image of each fisheye camera can be mapped to the panoramic coordinate system to obtain the ideal panoramic pixel corresponding to each sea surface pixel. The ideal panoramic pixel is the expected pixel position of the sea surface pixel in the panoramic coordinate system, determined based on the intrinsic parameter matrix of the fisheye camera and the sea level. The ideal panoramic pixel represents the position that the sea surface pixel should be mapped to under the geometric prior that the sea surface is approximately horizontal.

[0147] Specifically, for a sea surface pixel belonging to the i-th fisheye camera within the sea surface area indicated by the sea surface image, if the sea surface pixel is located in the first projected image, the pixel corresponding to the sea surface pixel p in the bird's-eye image sample can be determined first based on the known correspondence between the first projected image corresponding to the i-th fisheye camera and the bird's-eye image sample corresponding to the fisheye camera. Then, the original pixel corresponding to the sea surface pixel in the original fisheye image can be determined based on the known pixel correspondence between the bird's-eye image sample and the original fisheye image. The original pixel is back-projected according to the intrinsic parameter matrix of the i-th fisheye camera to obtain the imaging ray of the original pixel in the camera coordinate system. The imaging ray is transformed to the panoramic coordinate system according to the camera pose parameters of the i-th fisheye camera. The intersection point of the imaging ray and the sea surface is solved to obtain the intersection point position of the sea surface pixel on the sea surface. Then, the intersection point position is mapped to the panoramic coordinate system according to the preset mapping relationship to obtain the ideal panoramic pixel corresponding to the sea surface pixel. If the sea surface pixel is located in the bird's-eye view sample, the process for determining the ideal panoramic pixel can be as follows: Based on the known pixel correspondence between the bird's-eye view sample and the original fisheye image, determine the original pixel corresponding to the sea surface pixel in the original fisheye image; back-project the original pixel according to the intrinsic parameter matrix of the i-th fisheye camera to obtain the imaging ray of the original pixel in the camera coordinate system; transform the imaging ray to the panoramic coordinate system according to the camera pose parameters of the i-th fisheye camera; solve for the intersection point of the imaging ray and the sea surface to obtain the intersection point position of the sea surface pixel on the sea surface; then map the intersection point position to the panoramic coordinate system according to a preset mapping relationship to obtain the ideal panoramic pixel corresponding to the sea surface pixel.

[0148] That is, for the pixels in the sea surface area image corresponding to the i-th fisheye camera (Pixel coordinates in the original fisheye image) can be obtained through the intrinsic parameter matrix corresponding to the fisheye camera. Given the Z=0 plane geometric prior, its ideal coordinates in the panoramic coordinate system are directly calculated. (i.e., ideal panoramic pixels), its calculation principle is: pixel p is processed through... The back projection is a ray in the camera coordinate system. Using the camera pose, the ray is transformed to the panoramic coordinate system. Its intersection with the Z=0 plane yields a 3D sea surface point, which is then linearly mapped back to the panoramic coordinate system. This process can be concisely represented as follows: , This indicates the operation of finding the intersection of a ray and the Z=0 plane. This means linearly mapping 3D sea surface points to a panoramic coordinate system (involving only scaling and translation). This calculation process is entirely derived from geometric determinism and requires no feature matching.

[0149] In addition, the second deformable mesh corresponding to each fisheye camera can be determined based on the first deformable mesh and the first local deformable offset corresponding to each fisheye camera. The specific implementation process can be found in the detailed description of the relevant parts in the above embodiments, and will not be repeated here. Based on determining the second deformable mesh corresponding to each fisheye camera, each sea surface pixel in the sea surface area image can be mapped to the panoramic coordinate system using the second deformable mesh corresponding to each fisheye camera, obtaining the corresponding panoramic sea surface pixel. That is, the panoramic sea surface pixel is the pixel position obtained by mapping the sea surface pixel to the panoramic coordinate system using the second deformable mesh corresponding to the fisheye camera. The panoramic sea surface pixel reflects the effect of the first local deformable offset currently output by the deformable mesh offset prediction network on the sea surface pixel. Specifically, for a sea surface pixel in the sea surface area image, the mesh cell in which the pixel coordinates of the sea surface pixel fall within the second deformable mesh can be determined first. Then, based on the relative positions of the sea surface pixel with respect to the four mesh vertices of the mesh cell, the coordinates of the four mesh vertices in the panoramic coordinate system are interpolated using a grid bilinear interpolation method to obtain the panoramic sea surface pixel in the panoramic coordinate system.

[0150] Specifically, if the sea surface pixels are located in the first projected image, for the sea surface area pixels... It passes through the second deformable mesh corresponding to the i-th fisheye camera. The mapped panoramic sea surface pixels are Pixel p falls on the second deformed grid. Within a specific grid cell, this cell is bounded by four grid vertices (top left TL, top right TR, bottom left BL, bottom right BR). The offset of pixel p is calculated using bilinear interpolation based on the offsets of these four vertices. , , , , These are the weights for the four vertices, where the bilinear interpolation weights are determined by the relative position of pixel p within the cell: ,in, The normalized relative position in the x-direction. The normalized relative position in the y-direction, with values ​​ranging from [0, 1]. The panoramic sea surface pixels after pixel p is mapped by the second deformed mesh are: ,in, Let p be the offset of pixel p in the x-direction. Let p be the offset of pixel p in the y-direction. If the sea surface pixel is located in the bird's-eye view sample, the specific process for determining the panoramic sea surface pixel can be as follows: First, determine the grid cell in which the pixel coordinates of the sea surface pixel fall within the second deformed grid. Then, based on the relative positions of the sea surface pixel with respect to the four grid vertices of that grid cell, perform grid bilinear interpolation on the coordinates of the four grid vertices in the panoramic coordinate system. Based on the result of the bilinear interpolation and the corresponding first homography matrix, determine the panoramic sea surface pixel. That is, the panoramic sea surface pixel can be determined using... Obtain panoramic pixels of the sea surface .

[0151] Based on the above, a sea surface plane constraint loss function can be determined according to the ideal panoramic pixels and the sea surface panoramic pixels. This function measures the difference between the deformation result within the sea surface region and the ideal position determined by geometric priors. Specifically, for each sea surface pixel corresponding to a fisheye camera, the sea surface plane residual for that pixel can be determined based on the difference between the ideal panoramic pixels and the corresponding sea surface panoramic pixels. The sea surface plane residuals of each pixel are then summed or averaged to obtain the sea surface plane constraint term for each fisheye camera. Finally, the sea surface plane constraint term for each fisheye camera is summed or averaged to obtain the sea surface plane constraint loss function.

[0152] In some embodiments, the sea surface level constraint loss function can be calculated as follows:

[0153] Among them, the aforementioned sea surface plane constraint loss function takes effect during training, forcing the deformable mesh offset prediction network to output deformable meshes in the sea surface region that are close to the geometrically ideal position, and during inference, the deformable mesh directly outputs deformable meshes that satisfy the constraints.

[0154] The above method enables the deformed mesh output by the deformed mesh offset prediction network in the sea surface region to approach the ideal position determined by geometric priors by utilizing the sea surface plane constraint loss function. In other words, it makes the mapping result of the second deformed mesh synthesized by the first local deformation offset in the sea surface region match the ideal panoramic pixels. Since the sea surface is a weak or even textureless region, the supervision signal based on image observation is difficult to provide effective constraints in the sea surface region. The sea surface plane constraint loss function provides a definite, texture-independent supervision signal in this region, enabling the sea surface region to be correctly aligned.

[0155] See Figure 6This is a flowchart illustrating the determination of the temporal smoothing loss function of the mesh, provided in some embodiments of this application. According to some embodiments of this application, it may further include: determining a second deformed mesh corresponding to each fisheye camera based on a first deformed mesh and a first local deformed offset corresponding to each fisheye camera; determining a mesh temporal smoothing loss function based on the second deformed mesh corresponding to each fisheye camera and a second historical deformed mesh corresponding to each fisheye camera in a historical frame; and updating the initial parameters of the deformed mesh offset prediction network based on the deformed mesh geometric loss function, which may include: updating the initial parameters of the deformed mesh offset prediction network based on the deformed mesh geometric loss function and the mesh temporal smoothing loss function.

[0156] In some embodiments of this application, considering that ships experience continuous roll, pitch, and heave motions during berthing, if each frame is stitched independently, the stitching result will flicker and jump. Therefore, the physical continuity of rigid body motion can be used to constrain temporal stability so that the deformable mesh will not jump between frames, thereby making the berthing monitoring screen smooth and stable. The driver can clearly observe the dock and obstacles, reducing visual fatigue or misjudgment caused by screen jitter.

[0157] Among them, the hull is a rigid body, and its 6 degrees of freedom motion is continuous and smooth in time sequence - this is a direct deduction of the physical law (rigid body motion cannot have instantaneous jumps). Some embodiments of this application use this physical a priori to constrain the temporal stability at the deformable mesh level.

[0158] Specifically, based on the spatial constraints provided by the deformable mesh geometric loss function, a temporal constraint—the mesh temporal smoothing loss function—can be further introduced. The deformable mesh geometric loss function uses the geometric consistency within the overlapping region of adjacent fisheye cameras in the same frame as a monitoring signal, constraining the deformation results of each fisheye camera at the same time to be consistent. The mesh temporal smoothing loss function uses the continuity between the deformable meshes of the same fisheye camera at adjacent time points as a monitoring signal, constraining the deformation results of the same fisheye camera to change smoothly over time. This achieves constraints on the first local deformation offset output by the deformable mesh offset prediction network from both spatial and temporal dimensions.

[0159] Specifically, the determination of the mesh temporal smoothing loss function can be achieved by first determining the second deformed mesh corresponding to each fisheye camera based on the first deformed mesh and the first local deformed offset corresponding to each fisheye camera. The specific implementation process can be found in the detailed description of the relevant parts in the above embodiments, and will not be repeated here. It should be noted that the aforementioned determined second deformed mesh is the second deformed mesh corresponding to the current frame.

[0160] Based on the foregoing, a temporal smoothing loss function can be determined according to the second deformed mesh corresponding to each fisheye camera and the second historical deformed mesh corresponding to each fisheye camera in a historical frame. Here, a historical frame is a frame that is temporally preceding the current frame and has already been processed by the deformed mesh offset prediction network according to the aforementioned steps. In some embodiments, the bird's-eye view samples corresponding to each fisheye camera can come from a video sequence. During training, training samples can be organized according to the video sequence, and the aforementioned steps can be performed on each frame in the video sequence to obtain the second deformed mesh corresponding to each frame. In some embodiments, a historical frame can be the adjacent frame before the current frame, or it can be a frame with a preset number of frames before the current frame, or there can be multiple historical frames. In this case, the temporal smoothing loss function can be determined according to the deformed meshes corresponding to the current frame and multiple historical frames respectively. The second historical deformed mesh is the second deformed mesh corresponding to the historical frame, that is, the deformed mesh corresponding to the historical frame determined according to the first deformed mesh and the first local deformed offset output by the deformed mesh offset prediction network in the historical frame. The temporal smoothing loss function is a loss function used to measure the degree of change of the deformed mesh of the same fisheye camera over time.

[0161] Specifically, for each fisheye camera, the temporal residual of each grid vertex can be determined based on the position of each grid vertex in the second deformed grid corresponding to the fisheye camera and the difference between the position of each grid vertex and the position of the corresponding grid vertex in the second historical deformed grid. The temporal residual of each grid vertex is summed or averaged to obtain the temporal smoothing term of the fisheye camera. The temporal smoothing term of each fisheye camera is summed or averaged to obtain the temporal smoothing loss function of the grid.

[0162] In some embodiments, the grid temporal smoothing loss function can be calculated as follows:

[0163] in, This is the second deformable mesh corresponding to the i-th fisheye camera (i.e., the second deformable mesh corresponding to the current frame). Let be the second historical deformation mesh corresponding to the i-th fisheye camera in the historical frame. This is the grid-based temporal smoothing loss function.

[0164] Based on the above, the initial parameters of the deformable mesh migration prediction network can be updated according to the deformable mesh geometric loss function and the mesh temporal smoothing loss function. Specifically, the deformable mesh geometric loss function and the mesh temporal smoothing loss function can be weighted and summed, and the initial parameters of the deformable mesh migration prediction network can be updated using the total loss function obtained from the weighted summation, that is: ,in, For the total loss function, These are the weighting coefficients corresponding to the grid-based temporal smoothing loss function, which can be determined using the validation set.

[0165] As can be seen from the above, in some embodiments, the initial parameters of the deformable mesh offset prediction network can be updated according to the deformable mesh geometric loss function, the sea surface plane constraint loss function, and the mesh temporal smoothing loss function.

[0166] The above method can incorporate constraints based on the time dimension determined by the deformable mesh into the update of the deformable mesh offset prediction network, thereby constraining the amount of change of the deformable mesh in multiple frames, reducing high-frequency jitter at the mesh level, reducing the probability of inter-frame jumps in the deformable mesh, making the ship monitoring screen smooth and stable, so that the driver can clearly observe the dock and obstacles, and reduce visual fatigue or misjudgment caused by screen jitter.

[0167] According to some embodiments of this application, determining the first homography matrix corresponding to each fisheye camera based on each bird's-eye image sample may include: determining the motion parameters of the ship based on each bird's-eye image sample; and determining the first homography matrix corresponding to each fisheye camera based on the motion parameters of the ship and the relative poses between the fisheye cameras.

[0168] In some embodiments of this application, the ship's motion parameters can be determined based on bird's-eye view samples, and the first homography matrix corresponding to each fisheye camera can be determined based on the ship's motion parameters and the relative poses between the fisheye cameras. The ship's motion parameters describe the ship's attitude and position changes relative to the calibration time, while the relative poses between the fisheye cameras describe the fixed installation geometry between them. Since each fisheye camera is rigidly mounted on the ship, the relative poses between the fisheye cameras remain unchanged during the ship's movement, while the ship's motion parameters change with the ship's real-time attitude. Applying the changes described by the ship's motion parameters to the relative poses between the fisheye cameras yields the actual pose relationship of each fisheye camera at the current moment, and thus the first homography matrix corresponding to each fisheye camera can be calculated. When a ship is berthing, it experiences lateral movement and heave, causing the position of the same object to change in images from different cameras. By jointly estimating the ship's 6-DOF motion (e.g., lateral movement of 0.1m / frame, roll of 0.5° / frame) from bird's-eye view samples from all fisheye cameras, and then deriving the first homography matrix for each fisheye camera using calibration parameters and rigid constraints, globally consistent coarse alignment can be achieved. For example, the homography matrices of C0-C3 from the four starboard cameras are all derived from the same set of 6 motion parameters, thus naturally ensuring consistency and preventing the cascading error problem where C0 is correctly aligned but C3 is increasingly offset.

[0169] Specifically, the ship's motion parameters can be determined first based on the bird's-eye view sample. These motion parameters are physical quantities describing the ship's motion state in three-dimensional space, and may include at least one of the ship's angular velocity and translational velocity. In some embodiments, the ship's motion parameters can be represented as a six-dimensional vector, where three parameters can be angular velocities, corresponding to roll angular velocity, pitch angular velocity, and yaw angular velocity, respectively, used to describe the ship's rotational motion around the roll axis, pitch axis, and yaw axis, i.e., corresponding to the rotational angular velocities of the roll, pitch, and yaw axes; the other three parameters can be translational velocities, corresponding to the translational velocities along the ship's transverse, longitudinal, and vertical directions, respectively, used to describe the ship's positional movement in three-dimensional space, i.e., corresponding to the translational velocities of the x, y, and z axes.

[0170] In some embodiments, bird's-eye view samples corresponding to each fisheye camera can be used as input. A motion estimation network can then extract features and perform regression prediction on these bird's-eye view samples to output the ship's motion parameters. For example, the motion estimation network can perform global feature extraction on each bird's-eye view sample to obtain a global feature vector corresponding to each sample. This global feature vector is then input into a multilayer perceptron or long short-term memory network to regress and obtain the ship's motion parameters. Since the bird's-eye view samples project the fisheye image onto the ground plane, they can reflect the ship's motion relative to the ground. Therefore, the ship's motion parameters determined based on the bird's-eye view samples can accurately describe the ship's actual motion state.

[0171] In some embodiments, the specific process for determining the ship's motion parameters can be as follows: First, multi-camera aggregation is performed. Specifically, the bird's-eye view samples from each fisheye camera can be globally averaged to obtain the average pooling result. The average pooling results from each fisheye camera are then concatenated to obtain a global feature vector. The spatial features of each fisheye camera are compressed into vectors of corresponding dimensions through global average pooling, and the concatenation results in a global feature vector. This global feature vector encodes the scene information observed by all cameras and serves as the input for subsequent motion estimation. Then, the ship's motion parameters can be determined based on the global feature vector. Specifically, the ship's 6-DOF motion parameters can be regressed from the global feature vector using a fully connected network (multilayer perceptron). ,in, For global feature vectors, For a fully connected network, The original output of the fully connected network (i.e., the output of the output layer) is exemplified by the following structure: Input layer → Fully connected layer (512-dimensional) + ReLU (corrected linear unit) + Dropout (random deactivation) → Fully connected layer + ReLU → Output layer (6-dimensional) + Tanh (hyperbolic tangent function). Wherein, the output vector ξ = [ω... x ,ωy ,ω z ,v x ,v y ,v z The first three parameters of ξ are angular velocities (corresponding to the rotational angular velocities of the roll, pitch, and yaw axes), and the last three parameters are translational velocities (corresponding to the translational velocities of the x, y, and z axes). The Tanh activation function is used to limit the output to the range [-1, 1]. Multiplying this by the preset maximum motion velocity yields the actual motion parameters, i.e., ξ = Tanh(xi) × v. max , where v max The preset maximum speed is used. The aforementioned process fully utilizes the redundant information collected by the multi-camera system, improving the accuracy and robustness of ship motion parameter calculations.

[0172] Based on the above, the first homography matrix corresponding to each fisheye camera can be determined according to the ship's motion parameters and the relative poses between the fisheye cameras. Specifically, the relative poses between fisheye cameras can be the relative poses between adjacent fisheye cameras (which can be obtained through pre-calibration, including the rotation matrix and translation vector between adjacent fisheye cameras), and the relative poses between each fisheye camera and the reference camera can be determined based on the relative poses between adjacent fisheye cameras. Alternatively, the relative poses between fisheye cameras can be the pose relationship between each fisheye camera and the reference camera, specifically including the rotation matrix and translation vector of each fisheye camera relative to the reference camera. Since each fisheye camera is rigidly mounted on the ship, the relative position and attitude between the fisheye cameras remain unchanged during the ship's motion. Therefore, the relative poses between the fisheye cameras can be predetermined during the calibration stage, for example, by multi-camera joint calibration under conditions of ship berthing and calm sea surface, and the relative poses between the fisheye cameras can be pre-stored.

[0173] Specifically, rigid constraints can be used to derive the homography matrix of each fisheye camera relative to the reference camera from the ship's motion parameters and calibrated relative pose.

[0174] Specifically, to determine the first homography matrix, the rotation matrix of the reference camera at the current moment can be determined based on the rotation matrix of the reference camera in the fisheye camera at the previous moment and the ship's motion parameters. The rotation matrix of the reference camera at the previous moment is a rotational representation of the reference camera's attitude in a preset reference coordinate system at the previous moment, and the current moment corresponds to two adjacent processing frames. Specifically, the rotation increment of the reference camera can be determined based on the attitude change of the ship from the previous moment to the current moment described by the ship's motion parameters, and this rotation increment is applied to the rotation matrix of the reference camera at the previous moment to obtain the rotation matrix of the reference camera at the current moment.

[0175] Then, the rotation matrix of each fisheye camera (excluding the reference camera) at the current moment can be determined based on its rotation matrix relative to the reference camera. This rotation matrix represents the relative rotation relationship between each fisheye camera and the reference camera. This relative rotation relationship can be predetermined during the calibration phase, or it can be determined based on the relative poses of the fisheye cameras. Specifically, since each fisheye camera is rigidly mounted on the ship, its rotation matrix relative to the reference camera remains unchanged during the ship's movement. The rotation matrix of each fisheye camera at the current moment can be obtained by combining its rotation matrix relative to the reference camera with the rotation matrix of the reference camera at the current moment.

[0176] Subsequently, based on the rotation matrix of each fisheye camera at the current moment, the intrinsic parameter matrix of each fisheye camera, the translation vector of each fisheye camera relative to the reference camera, the normal vector of the reference plane, and the distance from the reference plane to the reference camera, the first homography matrix of each fisheye camera is determined. The translation vector of each fisheye camera relative to the reference camera represents the relative translation relationship between the two cameras. For example, it can be derived from the fixed-mount translation of the fisheye cameras (excluding the reference camera) relative to the reference camera and the motion parameters of the ship. Specifically, it consists of the cumulative translation of the ship as a whole and the displacement caused by the ship's attitude rotation, resulting in the fixed-mount offset in the current coordinate system. The reference plane can be a pre-defined approximate plane, such as the sea level, which is a horizontal plane with zero height in the world coordinate system. The normal vector of the reference plane is the unit normal vector of the reference plane, and the distance from the reference plane to the reference camera is the distance from the reference plane to the optical center of the reference camera.

[0177] Specifically, for the i-th fisheye camera, the first homography matrix corresponding to the i-th fisheye camera can be determined in the following way.

[0178] Step 201: Construct the rotation matrix from the angular velocity.

[0179] Assume the rotation matrix of the reference camera C0 at time t-1 is... Then the rotation matrix at time t can be updated using the angular velocity increment:

[0180]

[0181] in, To utilize the rotational increment determined by angular velocity; Let C0 be the rotation matrix of the reference camera at time t; This is an exponential mapping that transforms a 3D angular velocity vector into a 3×3 rotation matrix. Specifically:

[0182]

[0183] in, The angular velocities of the ship's roll, pitch, and bow are the angle vectors rotated in each frame, specifically including... .

[0184] Step 202: Derive the rotation matrix of other fisheye cameras using rigid constraints.

[0185] Since all fisheye cameras are fixed on the same rigid body, the rotation matrix of the i-th fisheye camera relative to the reference camera C0 is... (Specifically, the rotation matrix at time t) )for: ,in, Let C be the calibration rotation matrix of the i-th fisheye camera relative to the reference camera C0. Specifically, it can be determined based on the relative poses between the fisheye cameras. Let C be the rotation matrix of the i-th fisheye camera relative to the reference camera C0.

[0186] Step 203: Construct the first homography matrix for each fisheye camera.

[0187] Assuming the scene depth is approximately d (which can be the height of the camera above the water surface for the ocean), the first homography matrix of the i-th fisheye camera relative to the reference camera C0 is:

[0188] in, Let be the first homography matrix of the i-th fisheye camera. Let be the intrinsic parameter matrix of the i-th fisheye camera. Let C be the translation vector of the i-th fisheye camera relative to the reference camera C0. The normal vector of the reference plane (e.g., the unit normal vector of the plane being photographed). The distance from the reference plane to the reference camera C0, This is the intrinsic parameter matrix of the reference camera C0.

[0189] In some embodiments, when the scene is at a distance or involves pure rotation ( The first homography matrix can be simplified to:

[0190] Step 204: For a non-reference camera, the translation vector can be derived from the ship's motion parameters:

[0191] in, The cumulative translation of the reference camera, and the translation speed of the cumulative translation. The three velocity components [v] in the above-determined ship motion parameters x ,v y ,v z ] represents the current frame translation velocity in the ship's coordinate system. The cumulative translation amount is determined by... The cumulative translation is obtained by frame-by-frame integration. Assuming the initial cumulative translation is 0, the cumulative translation in frame t is: , This is the cumulative translation amount from the previous frame. This is the cumulative rotation matrix of the previous frame obtained frame by frame accumulation in the first step. This represents the translation velocity of the current frame output by the network. The rotation accumulation and translation integral together constitute the complete pose of camera C_0.

[0192] Let be the cumulative translation amount relative to the i-th fisheye. The local translation from the reference camera C0 obtained during calibration to the i-th fisheye camera is calculated. The calibration parameters are embedded into the network as hard constraints, so the network only needs to learn the motion parameters and does not need to relearn the relative pose.

[0193] Since the first homography matrix of all fisheye cameras comes from the same set of ship motion parameters and fixed relative poses between cameras, the projection relationship between each camera naturally satisfies the global consistency requirement, which can suppress the accumulation of errors and splicing misalignment caused by the independent processing of each splicing in traditional pairwise cascaded splicing.

[0194] By jointly estimating the ship's motion parameters from the features of all fisheye cameras and deriving the homography matrix of each camera using calibration parameters and rigid constraints, globally consistent coarse alignment can be achieved. This suppresses the problems of error accumulation and misalignment in traditional pairwise cascaded stitching, effectively improving the quality of image stitching.

[0195] According to some embodiments of this application, it may further include: determining a motion smoothing loss function based on the ship's motion parameters and the ship's historical motion parameters in historical frames; updating the initial parameters of the deformable mesh offset prediction network based on the deformable mesh geometric loss function, which may include: updating the initial parameters of the deformable mesh offset prediction network based on the deformable mesh geometric loss function and the motion smoothing loss function.

[0196] In some embodiments of this application, considering that ships have continuous roll, pitch and heave motions during berthing, if each frame is stitched independently, the stitching result will flicker and jump. Therefore, the physical continuity of rigid body motion can be used to constrain temporal stability so that the roll angle will not change abruptly between frames, thereby making the berthing monitoring screen smooth and stable. The driver can clearly observe the dock and obstacles, reducing visual fatigue or misjudgment caused by screen shaking.

[0197] As a rigid body, the hull's 6 degrees of freedom motion is continuous and smooth in time sequence—this is a direct deduction of the laws of physics (rigid body motion cannot have instantaneous jumps). Some embodiments of this application utilize this physical prior to constrain temporal stability at the level of motion parameters.

[0198] Specifically, a motion smoothing loss function can be determined based on the ship's motion parameters and its historical motion parameters in historical frames. The motion smoothing loss function uses the continuity of the ship's motion parameters between adjacent moments as a monitoring signal, constraining the ship's motion parameters determined from the bird's-eye view samples to change smoothly over time. Since the ship's motion parameters determine the first homography matrix corresponding to each fisheye camera, and the first homography matrix determines the coarse alignment landing point, the temporal constraint on the motion parameters can suppress frame-by-frame jumps in the coarse alignment result at the source, thereby suppressing the instability of the stitching result in the temporal dimension. It should be noted that the specific explanation of historical frames can be found in the corresponding sections of the aforementioned embodiments, and will not be repeated here. The historical motion parameters of the historical frames are the motion parameters corresponding to the historical frames, and the method for determining the historical motion parameters is the same as the method for determining the ship's motion parameters in the current frame.

[0199] Specifically, the motion smoothing loss function is a loss function determined based on the ship's motion parameters and historical motion parameters, used to measure the drastic change of the ship's motion parameters over time. For example, the differences between each component of the ship's motion parameters and the corresponding components in the historical motion parameters can be defined as motion parameter residuals (e.g., the squares of the corresponding differences). Summation or averaging of these residuals yields the motion smoothing loss function. In some embodiments, considering the different dimensions of angular velocity and translational velocity in the ship's motion parameters, each component can be normalized before calculating the motion parameter residuals. Alternatively, a velocity loss function and an acceleration loss function can be determined based on the ship's motion parameters and historical motion parameters, and then the motion smoothing loss function can be determined based on these functions.

[0200] Based on the above, the initial parameters of the deformable mesh migration prediction network can be updated according to the deformable mesh geometric loss function and the motion smoothing loss function. Specifically, the deformable mesh geometric loss function and the motion smoothing loss function can be weighted and summed, and the initial parameters of the deformable mesh migration prediction network can be updated using the total loss function obtained from the weighted summation, that is: ,in, For the total loss function, Let be the motion smoothing loss function. These are the weighting coefficients corresponding to the motion smoothing loss function, which can be determined using the validation set.

[0201] As described above, in some embodiments, the initial parameters of the deformable mesh migration prediction network can be updated based on the deformable mesh geometric loss function, the sea surface plane constraint loss function, and the motion smoothing loss function. Alternatively, in some embodiments, the initial parameters of the deformable mesh migration prediction network can be updated based on the deformable mesh geometric loss function, the sea surface plane constraint loss function, the mesh temporal smoothing loss function, and the motion smoothing loss function.

[0202] The above method allows for the inclusion of time-dimensional constraints based on motion parameters in the updating of the deformable mesh offset prediction network. This enables the deformable mesh offset prediction network to learn the ability to output smooth and continuous motion parameters, preventing abrupt changes in the ship's roll angle between frames. This reduces the probability of sudden motion changes and drastic shifts in motion trends. Furthermore, the smoothing of motion parameters can drive frame-by-frame smoothing of the first homography matrix and the coarse alignment results, thereby establishing temporal stability at the source of the network input.

[0203] See Figure 7 This is a flowchart illustrating the determination of the first homography matrix in some embodiments of this application. According to some embodiments of this application, determining the motion smoothing loss function based on the ship's motion parameters and historical motion parameters of the ship in historical frames may include: determining a velocity loss function and an acceleration loss function based on the ship's motion parameters and historical motion parameters of the ship in historical frames; and determining the motion smoothing loss function based on the velocity loss function and the acceleration loss function.

[0204] In some embodiments of this application, when determining the motion smoothing loss function based on the ship's motion parameters and historical motion parameters, a velocity loss function and an acceleration loss function can be specifically determined based on the ship's motion parameters and historical motion parameters, thereby determining the motion smoothing loss function. The velocity loss function constrains the first-order change of the ship's motion parameters relative to historical motion parameters, measuring the rate of change of the motion parameters over time. The acceleration loss function constrains the change of the first-order change, measuring whether the rate of change of the motion parameters undergoes abrupt changes. It should be noted that, based on the motion smoothing loss function including the acceleration loss function, the number of historical frames is at least two; that is, historical frames can include the frame preceding the current frame and at least one frame before the previous frame, i.e., historical frames can include at least two frames preceding the current frame.

[0205] Specifically, a velocity loss function can be determined based on the ship's motion parameters in the current frame and its historical motion parameters in the adjacent previous frame. This velocity loss function is a function that measures the first-order change in the ship's motion parameters. In some embodiments, the velocity loss function can be determined based on the difference between corresponding components of the ship's motion parameters in the current frame and its historical motion parameters in the adjacent previous frame. For example: ,in, The motion parameters of the ship in the current frame. These are the historical motion parameters of the ship in the previous frame (t-1) before the current frame (t). This is the raw output of a fully connected network; or, ,in, The motion parameters of the ship in the current frame. ξ represents the ship's historical motion parameters from the previous frame in the current frame, and ξ is the motion parameters obtained by multiplying by a preset maximum motion speed. The velocity loss function constrains the velocity (the change in motion parameters between frames) to prevent abrupt motion changes.

[0206] Furthermore, an acceleration loss function can be determined based on the ship's motion parameters in the current frame and its historical motion parameters in the adjacent previous frame. This acceleration loss function measures the rate of change of the first-order change in the ship's motion parameters. Specifically, a second-order difference can be determined based on the ship's motion parameters in the current frame and its historical motion parameters in the adjacent previous frame, and this second-order difference can be used as the acceleration loss function. For example: ,in, The motion parameters of the ship in the two frames preceding the current frame (i.e., t-2); or, ,in, The motion parameters of the ship in the current frame. This refers to the ship's historical motion parameters from the previous frame in the current frame. These are the motion parameters of the ship in the two frames preceding the current frame (i.e., t-2). The acceleration loss function constrains the acceleration (rate of change of velocity) to reduce drastic changes in the motion trend.

[0207] Based on the above, the motion smoothing loss function can be determined according to the velocity loss function and the acceleration loss function. For example, the velocity loss function and the acceleration loss function can be added together to obtain the motion smoothing loss function. : .

[0208] As can be seen from the above embodiments, temporal stability can be achieved naturally through the physical constraints of rigid body motion. Only the rate of change of 6 motion parameters needs to be constrained, and no additional temporal modeling network is required. Therefore, it is more robust (physical constraints automatically pull abnormal frames back to a reasonable range) and more efficient (no computationally intensive modules such as 3D convolution).

[0209] The above method allows the velocity loss function and the acceleration loss function to participate in the motion smoothing loss function, thereby reducing the abrupt changes in yaw angle between frames and making the monitoring image smoother and more stable.

[0210] According to some embodiments of this application, before determining the first homography matrix corresponding to each fisheye camera based on each bird's-eye image sample, the method may further include: extracting multi-scale features of the bird's-eye image samples using a feature extraction network to obtain feature map samples of the bird's-eye image samples; determining the first homography matrix corresponding to each fisheye camera based on each bird's-eye image sample; and determining the first projected image based on each bird's-eye image sample and the corresponding first homography matrix, which may include: determining the first homography matrix corresponding to each fisheye camera based on the feature map samples of each bird's-eye image sample; and determining the first projected feature image based on the feature map samples of each bird's-eye image sample and the corresponding first homography matrix.

[0211] In some embodiments of this application, before determining the first homography matrix corresponding to each fisheye camera based on each bird's-eye view sample, a feature extraction network can be used to extract multi-scale features of the bird's-eye view samples to obtain feature map samples of the bird's-eye view samples. The feature extraction network is a neural network used to extract multi-scale features of the bird's-eye view samples. Exemplarily, the feature extraction network can be a convolutional neural network, for example, it can employ an encoder-decoder structure or a feature pyramid structure. The feature extraction network performs progressive downsampling and feature extraction on the bird's-eye view samples, outputting multiple feature maps at different scales. The feature map samples are the feature maps obtained after the feature extraction network extracts features from the bird's-eye view samples.

[0212] In some embodiments, for each fisheye camera's bird's-eye view sample, a shared-weights feature extraction network can be used to extract multi-scale features. For example, the feature extraction network can employ the first four residual stages of ResNet, removing the final global pooling layer and fully connected layer to preserve spatial resolution information. Each fisheye camera obtains a feature map of H / 8×W / 8×C, where H×W×3 is the size of the bird's-eye view sample, H and W are the height and width of the bird's-eye view sample, and C is the number of feature channels (e.g., C=256 by default). The advantages of using shared weights are: (1) reducing the number of model parameters, with only one set of parameters required for N fisheye cameras; (2) ensuring that the features extracted by different fisheye cameras are in the same feature space, making subsequent cross-camera attention calculations comparable; and (3) enhancing the generalization ability of the model, with no additional parameters required for adding fisheye cameras.

[0213] The above approach can provide a data foundation with a larger receptive field and stronger contextual expression for coarse alignment. Specifically, the sea surface is a weak texture region. In the original pixels of the bird's-eye view sample, the pixel value difference in the sea surface region is very small and the number of feature points available for matching is scarce. However, the feature map sample can gather structural information from a larger range of surrounding areas, and can form distinguishable features even in the water surface region with weak texture, thereby alleviating the feature matching failure problem caused by weak texture.

[0214] Based on the above, determining the first homography matrix corresponding to each fisheye camera based on each bird's-eye image sample can specifically involve determining the first homography matrix corresponding to each fisheye camera based on the feature map samples of each bird's-eye image sample. In some embodiments, the first homography matrix is ​​a matrix used to describe the mapping relationship between the feature map samples corresponding to the fisheye camera and the coordinate system of the reference camera C0. Determining the first projected image based on each bird's-eye image sample and the corresponding first homography matrix can specifically involve determining the first projected feature image based on the feature map samples of each bird's-eye image sample and the corresponding first homography matrix. The first projected feature image is a projected image in feature map form obtained by mapping the feature map samples to the corresponding coordinate system according to the corresponding first homography matrix.

[0215] In some embodiments, a corresponding first homography matrix can be applied to the feature map sample to obtain a corresponding first projected feature image. In some embodiments, the first homography matrix can be applied to the bird's-eye view sample itself to obtain a projection result in image form.

[0216] The above method can gather the structural context around the weak texture region of the sea surface, so that the solution of the first homography matrix is ​​still relatively accurate in weak texture scenes with few feature points, thereby improving the alignment accuracy and thus improving the accuracy of the local deformation offset and the deformation mesh offset prediction network.

[0217] According to some embodiments of this application, before determining the first homography matrix corresponding to each fisheye camera based on the feature map samples of each bird's-eye image sample, the method may further include: determining weak texture region samples based on the feature map samples of the bird's-eye image sample; performing cross-camera attention interaction on the weak texture region samples in the overlapping region of adjacent fisheye cameras to obtain fused and enhanced weak texture region samples; and obtaining enhanced feature map samples based on the feature map samples of the bird's-eye image sample and the fused and enhanced weak texture region samples.

[0218] Considering that in some ship navigation scenarios, the field of view of a fisheye camera is mostly the sea surface, for example, in a berthing scenario, the sea surface occupies more than 50% of the lateral field of view, and the sea surface area has poor texture, in some embodiments of this application, the sea surface area features of each fisheye camera can be enhanced by obtaining complementary information from adjacent cameras through cross-camera attention (e.g., adjacent cameras have richer water surface textures due to different lighting angles), thereby reducing the failure of the alignment step due to missing features. That is, the feature information enhancement between adjacent cameras can be achieved through the Transformer attention mechanism to enhance the feature representation of the sea surface area.

[0219] Specifically, before determining the first homography matrix corresponding to each fisheye camera based on the feature map samples of each bird's-eye view sample, the corresponding weak texture region samples can be determined based on the feature map samples of each bird's-eye view sample. These weak texture region samples are information used to characterize areas with weak texture in the bird's-eye view sample, determined based on the feature map samples. For example, for each bird's-eye view sample, weak texture region samples can be determined based on the feature statistics of each local region in its feature map sample. For instance, the amplitude, variance, or energy of the feature response in each local region in the feature map sample can be calculated, and local regions with feature statistics below a preset threshold can be identified as weak texture regions. Alternatively, the weak texture region samples can be binary masks indicating whether each location in the bird's-eye view sample belongs to a weak texture region, or confidence maps characterizing the confidence that each location belongs to a weak texture region. For example, the sea surface area is a weak texture region. Under conditions of no wind or waves or uniform lighting, the sea surface lacks matching texture structures, and its feature statistics in the feature map samples are significantly lower than those of structural areas such as docks, hulls, and fenders.

[0220] Specifically, since there are a large number of sea surface areas in the ship's field of view (e.g., lateral view), these areas have poor texture and low feature extraction quality, which can easily lead to the failure of subsequent alignment steps. Therefore, some embodiments of this application can automatically identify weak texture areas through local texture complexity evaluation and enable cross-camera feature enhancement in these areas. Specifically, in some embodiments, for each feature map sample of a bird's-eye view sample, each spatial location of the feature map sample can be analyzed. The method calculates the difference between the given spatial location and its neighboring spatial locations, sums or averages the squares of these differences to obtain the local texture complexity. Based on the relationship between this local texture complexity and a preset weak texture threshold, a weak texture region sample is determined. The weak texture threshold is a pre-defined threshold used to determine whether a pixel belongs to a weak texture region. Specifically, if the local texture complexity is less than the weak texture threshold, it is determined to be a weak texture region; if the local texture complexity is greater than or equal to the weak texture threshold, it is determined to be a non-weak texture region. In other words, a weak texture mask can be generated based on the local texture complexity and the preset weak texture threshold.

[0221] In some embodiments, for each spatial location of a feature map sample The local texture complexity can be calculated using the following formula: ,in, For The local neighborhood centered on the center (e.g., can be 3×3 by default). The total number of pixels in the neighborhood. The spatial location corresponding to the i-th feature map sample , This reflects the degree of drastic change in local pixel data at that location, i.e. This represents the calculated local texture complexity. A smaller TC value indicates a less textured region. Based on this, a mask for the weakly textured region is generated using the TC value. : ,when ; ,when .in, The weak texture threshold can be determined statistically from training data. Weak texture region mask. For binary images, weak texture regions correspond to positions 1, and strong texture regions correspond to positions 0. For example, the sea surface region corresponds to a position 1, while strong texture regions such as docks and buildings correspond to positions 0.

[0222] Building upon the above, cross-camera attention interaction can be performed on weak-texture region samples in the overlapping areas of adjacent fisheye cameras to obtain fused and enhanced weak-texture region samples. Cross-camera attention interaction is an interactive method that uses features from a weak-texture region sample of one fisheye camera as the query, and features from adjacent fisheye cameras in the overlapping area as the key and value, achieving cross-camera information fusion through the calculation and weighted aggregation of attention weights. For example, cross-camera attention interaction can employ a multi-head attention mechanism; alternatively, it can be bidirectional, where two adjacent fisheye cameras act as each other's query and key; or, it can be that only one fisheye camera queries the other.

[0223] Then, an enhanced feature map sample can be obtained based on the feature map sample of the bird's-eye view sample and the fused and enhanced weak texture region sample. The enhanced feature map sample is the feature sample of the weak texture region after fusion enhancement, determined based on the feature map sample of the bird's-eye view sample and the fused and enhanced weak texture region sample. For example, the fused and enhanced weak texture region sample can be fused (e.g., by replacement or weighted fusion) to the corresponding weak texture region in the feature map sample, while keeping the features of the regions outside the weak texture region in the feature map sample unchanged, thus obtaining the enhanced feature map sample.

[0224] In some embodiments, for the i-th fisheye camera and the (i+1)-th fisheye camera, cross-camera attention interaction is performed in their overlapping region, enabling the features of each fisheye camera to "see" the features of the adjacent camera in the overlapping region, thus achieving cross-viewpoint information enhancement. Specifically, a pre-computed overlapping region mask can be used. Attention is calculated only in the overlapping region to reduce unnecessary computation in non-overlapping regions. Assume the overlapping region of the i-th fisheye camera is... The overlapping area of ​​the (i+1)th fisheye camera is The two have the same spatial dimensions (because the extent of the overlapping area in both camera views is determined by calibration parameters). The calculation process for cross-camera attention is as follows: Step 1: Linear projection generates queries, keys, and values:

[0225]

[0226]

[0227] in, , , The learnable projection matrix (dimension C × d_k); , , Let d_k be the bias vector, and d_k be the dimension of the attention head. The features of the i-th fisheye camera are used as the query (corresponding to...). (i.e., query vector / matrix), the feature of the (i+1)th fisheye camera is used as the key (Key, corresponding to...) (i.e., key vectors / matrices) and values ​​(corresponding to...) (i.e., value vector / matrix), meaning that the i-th fisheye camera actively retrieves and acquires information from the (i+1)-th fisheye camera.

[0228] Step 2: Calculate the attention weight matrix:

[0229] in, The dimension is n_overlap×d_k, The dimension is d_k×n_overlap, and the attention matrix is... The dimension is n_overlap×n_overlap. Divide the dimension associated with each query / key by Scaling is applied to prevent the softmax gradient from vanishing due to an excessively large inner product value. n_overlap is the total number of pixels in the overlapping region.

[0230] Step 3: Calculate attention-weighted features:

[0231] The dimension is n_overlap×d_k, and the value at each position is a weighted sum of the features of the (i+1)th fisheye camera overlapping region. The weights are determined by the feature similarity.

[0232] Step 4: Weak texture gating enhances blending:

[0233] in, The learnable enhancement coefficient (initialized to 0.1). For weak texture masks, ⊙ represents element-wise multiplication. The key features of this fusion strategy are: (1) only in weak texture regions ( =1, i.e., sea surface) Enables cross-camera enhancement, strong texture areas ( =0, i.e., docks, buildings, etc.) keep the original features unchanged and reduce unnecessary interference in strong texture areas; (2) Each fisheye camera can "borrow" feature information from adjacent fisheye cameras to make up for its own lack of weak texture. For example, the sea surface area of ​​the i-th fisheye camera has poor texture, but the adjacent i+1-th fisheye camera may have richer texture information in the overlapping area due to different lighting angles; (3) Attention is only calculated in the overlapping area, and the non-overlapping area does not participate in the calculation. The amount of calculation is controllable and will not increase twice with the image size; (4) Learnable coefficients This enables the network to adaptively adjust the intensity of cross-camera augmentation during training.

[0234] Based on the above, in some embodiments, when determining the first homography matrix based on the feature map samples, the first homography matrix can be specifically determined based on the enhanced feature map samples. Similarly, when determining the first projected feature image based on the feature map samples and the corresponding first homography matrix, the first projected feature image can be specifically determined based on the enhanced feature map samples and the corresponding first homography matrix. In some embodiments, when predicting the first local deformation offset based on the bird's-eye view samples, the first projected image, and the deformable mesh offset prediction network, the first local deformation offset can be specifically predicted based on the enhanced feature map samples, the first projected feature image, and the deformable mesh offset prediction network.

[0235] The above method can be used to enhance the features of weak texture regions across cameras, so that each fisheye camera can see the overlapping features of adjacent fisheye cameras. This allows weak texture regions to obtain supplementary information from adjacent cameras, achieving cross-camera information enhancement, while the features of non-weak texture regions can remain unchanged, thus reducing the amount of computation.

[0236] According to some embodiments of this application, performing cross-camera attention interaction on weak texture region samples in the overlapping region of adjacent fisheye cameras to obtain fused and enhanced weak texture region samples may include: performing cross-camera attention interaction on weak texture region samples in the overlapping region of adjacent fisheye cameras in a first direction to obtain fused and enhanced first weak texture region samples; performing cross-camera attention interaction on weak texture region samples in the overlapping region of adjacent fisheye cameras in a second direction to obtain fused and enhanced second weak texture region samples; the first direction and the second direction are opposite in direction.

[0237] In some embodiments of this application, cross-camera attention interaction can be performed in two opposite directions so that the weak texture regions of the two adjacent fisheye cameras in the overlapping area can obtain complementary information from the other's perspective, thereby achieving symmetrical enhancement.

[0238] Specifically, weak-texture region samples in the overlapping areas of adjacent fisheye cameras can undergo cross-camera attention interaction in a first direction to obtain a fused and enhanced first weak-texture region sample. The first direction can be the interaction direction with the i-th fisheye camera as the query party. Specifically, the cross-camera attention interaction in the first direction can be performed by using features in the weak-texture region sample of the i-th fisheye camera located within the overlapping area of ​​adjacent fisheye cameras as the query, and using features in the feature map sample of the (i+1)-th fisheye camera located within the overlapping area as keys and values. The correlation between the query and each key is calculated as the attention weight, and the corresponding values ​​are weighted and aggregated according to the attention weight to obtain the fused and enhanced first weak-texture region sample. The fused and enhanced first weak-texture region sample belongs to the i-th fisheye camera side, that is, it is the result of the weak-texture region of the i-th fisheye camera being enhanced by the observation of the (i+1)-th fisheye camera.

[0239] Furthermore, weak-texture region samples in the overlapping areas of adjacent fisheye cameras can undergo cross-camera attention interaction in a second direction to obtain a fused and enhanced second weak-texture region sample. The first and second directions are opposite. Specifically, the second direction can be the interaction direction with the (i+1)th fisheye camera as the query party. Cross-camera attention interaction in the second direction can be achieved by using features in the weak-texture region sample of the (i+1)th fisheye camera located within the overlapping area of ​​adjacent fisheye cameras as the query, and using features in the feature map sample of the ith fisheye camera within the overlapping area as the key and value, calculating attention weights, and weighted aggregation to obtain the fused and enhanced second weak-texture region sample. The fused and enhanced second weak-texture region sample belongs to the (i+1)th fisheye camera side.

[0240] That is, for N fisheye cameras, cross-camera attention can propagate bidirectionally along the camera chain, allowing each camera to eventually fuse information from all cameras. In other words, forward propagation (transmitting information from the first camera to the last camera) and backward propagation (transmitting information from the last camera to the first camera) are possible. Specifically, forward propagation is as follows:

[0241]

[0242]

[0243]

[0244]

[0245] Backpropagation is:

[0246]

[0247]

[0248]

[0249] in, Indicates forward propagation. Indicates backpropagation, This indicates cross-camera attention interaction computation.

[0250] Based on the above, the enhanced feature map sample is obtained from the feature map sample of the bird's-eye view and the fused and enhanced weak texture region sample. Specifically, the enhanced feature map sample can be obtained as follows: Based on the feature map sample and the fused and enhanced first weak texture region sample, the enhanced feature map sample corresponding to the first direction is obtained; based on the feature map sample and the fused and enhanced second weak texture region sample, the enhanced feature map sample corresponding to the second direction is obtained; and the enhanced feature map samples corresponding to the first and second directions are fused (stitched and linearly transformed) to obtain the enhanced feature map sample. That is, , To enhance the feature map.

[0251] In this approach, the same cross-attention module (parameter sharing) can be used at each step. Forward propagation passes information from the first camera to the last, and backward propagation passes information from the last to the first. Ultimately, the enhanced features of each fisheye camera are fused with information from all fisheye cameras. Furthermore, the effect of this full-chain feature propagation is that even if there is no direct overlap between non-adjacent fisheye cameras, information can still be indirectly passed to the corresponding fisheye camera through adjacent fisheye cameras. For example, fisheye cameras C_0 and C_3 do not have a direct overlap, but information from fisheye camera C_0 can still be indirectly passed to C_3 through C_1 and C_2, and vice versa. Compared to the pairwise stitching method, the above full-chain propagation can utilize indirect information across cameras to improve image stitching quality.

[0252] The above method allows cross-camera attention to propagate bidirectionally along the phase chain, enabling each camera to ultimately fuse information from all cameras and utilize indirect cross-camera information, thereby facilitating improved image stitching quality.

[0253] This application also provides a fisheye image stitching method, see [link to relevant documentation] Figure 8 This is a flowchart of a fisheye image stitching method according to some embodiments of this application, which may include: S81: Acquire fisheye images from multiple fisheye cameras on the ship, and obtain the corresponding bird's-eye view based on each fisheye image.

[0254] It should be noted that the multiple fisheye cameras mentioned here are the same fisheye cameras mentioned in the offset prediction network training method of some embodiments of this application.

[0255] S82: Determine the second homography matrix corresponding to each fisheye camera based on each bird's-eye view, and determine the second projection map based on each bird's-eye view and the corresponding second homography matrix.

[0256] S83: Determine the third deformable mesh based on the bird's-eye view. Based on each bird's-eye view, each second projection map, and the deformable mesh offset prediction network trained using any of the above offset prediction network training methods, predict the second local deformation offset of the third deformable mesh and each fisheye camera.

[0257] S84: Determine the fourth deformable mesh corresponding to each fisheye camera based on the third deformable mesh and the second local deformable offset corresponding to each fisheye camera.

[0258] It should be noted that the specific implementation process of steps S81 to S84 is similar to the specific implementation process in the corresponding embodiments of the offset prediction network training method in some embodiments of this application, and will not be repeated here.

[0259] S85: Determine the panoramic image based on the fourth deformable grid corresponding to each fisheye camera and each bird's-eye view.

[0260] Based on the fourth deformation mesh and the corresponding bird's-eye view for each fisheye camera, a panoramic image can be determined. For example, for each fisheye camera, the bird's-eye view corresponding to that fisheye camera can be deformed using the mapping relationship determined by its corresponding fourth deformation mesh to obtain a deformed image. These deformed images are then superimposed on a unified canvas corresponding to the panoramic coordinate system, and the overlapping areas of adjacent fisheye cameras are fused to obtain the panoramic image.

[0261] The mapping relationship determined in the fourth deformable mesh combines the coarse alignment mapping described by the second homography matrix and the local deformation described by the second local deformation offset. That is, each position in the bird's-eye view can first be mapped to the coordinate system of the reference camera via the second homography matrix, and then locally deformed according to the fourth deformable mesh. The deformed result is in the panoramic coordinate system. In one embodiment, the bird's-eye view can be projected according to the second homography matrix to obtain a second projection image, and then the second projection image can be deformed using the fourth deformable mesh. This application does not limit this approach. For example, the deformation can be achieved through grid bilinear interpolation and pixel-by-pixel resampling.

[0262] The global consistency, temporal smoothness, and geometric accuracy established during the training phase are maintained without loss during the inference phase. The entire stitching process is a one-time parallel process of coarse alignment, local fine deformation of the grid, and fusion of the unified canvas. This can solve the problem of error accumulation caused by independent processing of each stitching round in the traditional pairwise cascaded stitching method, thereby improving the image stitching quality.

[0263] According to some embodiments of this application, determining a panoramic image based on the fourth deformable grid corresponding to each fisheye camera and each bird's-eye view may include: performing image transformation on each bird's-eye view based on the fourth deformable grid corresponding to each fisheye camera to obtain deformed images of each bird's-eye view; determining the coordinate range of a unified canvas based on the coordinate range of the deformed images; and mapping each deformed image to the unified canvas to obtain a panoramic image.

[0264] In some embodiments of this application, when determining the panoramic image based on the fourth deformation grid corresponding to each fisheye camera and each bird's-eye view, specifically, image transformation can be performed on each bird's-eye view based on the fourth deformation grid corresponding to each fisheye camera to obtain deformed images of each bird's-eye view. For each pixel position p in the deformed image, its corresponding position in the bird's-eye view is calculated through bilinear interpolation of the fourth deformation grid, and then pixel values ​​are sampled from the bird's-eye view: ,in, For the operation of bilinear interpolation sampling based on the fourth deformed mesh, This is a bilinear interpolation in the image domain. The fourth deformable mesh naturally guarantees the smoothness of the transformation (the offsets of adjacent mesh vertices change continuously), requiring no additional smoothing regularization.

[0265] The deformed image is the image transformed from the bird's-eye view through the mapping relationship determined by the fourth deformed grid, and is placed in the panoramic coordinate system. Specifically, the mapping relationship determined in the fourth deformed grid combines the coarse alignment mapping described by the second homography matrix and the local deformation described by the second local deformation offset. The image transformation is to relocate each pixel in the bird's-eye view according to this mapping relationship: for each target position in the deformed image, grid bilinear interpolation can be performed based on the positions of the four grid vertices of the grid cell into which the target position falls in the fourth deformed grid to determine the sampling position corresponding to the target position in the bird's-eye view, and then bilinear interpolation can be performed on the image content at the sampling position to obtain the pixel value of the target position.

[0266] The above method can achieve coarse alignment and local deformation in one step to the image content. Specifically, after image transformation, each bird's-eye view is in the panoramic coordinate system, and the positional relationship between each deformed image has been determined, so there is no need to perform alignment calculations again.

[0267] Based on the above, the coordinate range of the unified canvas can be determined according to the coordinate range of the deformed images. The coordinate range of the deformed images is the coordinate range covered by the deformed images in the panoramic coordinate system. For example, it can be the coordinate range corresponding to the smallest bounding rectangle of the deformed images, that is, the rectangular range determined by the minimum x-coordinate, maximum x-coordinate, minimum y-coordinate, and maximum y-coordinate of each position in the deformed images. Specifically, the coordinate range of the unified canvas can be obtained by taking the union of the coordinate ranges of all deformed images; that is, the coordinate range of the unified canvas can be determined by the minimum x-coordinate, maximum x-coordinate, minimum y-coordinate, and maximum y-coordinate of the coordinate ranges of all deformed images. The unified canvas is the image area used to carry the panoramic image, based on the determined coordinate range.

[0268] The aforementioned process allows the coordinate range of the unified canvas to adaptively cover the actual content of the entire deformed image, ensuring that the entire content is not cropped and reducing redundant white space.

[0269] Based on a defined coordinate range of a unified canvas, each deformed image can be mapped onto the unified canvas to obtain a panoramic image. For example, each deformed image can be translated to its corresponding position within the unified canvas according to its own coordinate range for drawing. Alternatively, before drawing, all deformed images can be scaled as a whole according to the coordinate range of the unified canvas and a preset output resolution, so that the resolution of the drawn panoramic image matches the preset output resolution. This overall scaling applies the same scaling to all deformed images without changing their relative positional relationships.

[0270] The above method can achieve the acquisition of complete panoramic images with consistent geometric scales with extremely low computational overhead, thereby improving the image stitching quality.

[0271] According to some embodiments of this application, mapping each deformed image to a unified canvas to obtain a panoramic image may include: if a single deformed image is mapped to a pixel position on the unified canvas, determining the pixel data of the pixel position based on the pixel data of the deformed image at the pixel position; if multiple deformed images are mapped to pixel positions on the unified canvas, determining the weight of each deformed image at the pixel position based on the position of the pixel position in each deformed image, and performing weighted fusion of the pixel data of each deformed image at the pixel position based on the weight of each deformed image at the pixel position to obtain the pixel data of the pixel position.

[0272] In some embodiments of this application, when mapping each deformed image to a unified canvas, if a single deformed image is mapped to a pixel position on the unified canvas, the pixel data at that pixel position can be determined based on the pixel data of the deformed image at that pixel position. That is, for a pixel position on the unified canvas that is covered by only one deformed image, the pixel data of the deformed image at that pixel position can be determined as the pixel data of that pixel position on the unified canvas. For example, the pixel value of the deformed image at that pixel position can be directly used as the pixel value of that pixel position on the unified canvas. Wherein, if each deformed image has been scaled as a whole when mapped to the unified canvas, the pixel data at that pixel position can be obtained by interpolating and sampling the pixel data at the corresponding position in the deformed image.

[0273] If multiple deformable images are mapped to pixel positions on a unified canvas, the weight of each deformable image at a pixel position can be determined based on its location within each deformable image. Then, the pixel data at each deformable image's pixel position can be weighted and fused according to these weights to obtain the pixel data at that pixel position. Here, the location of a pixel within each deformable image is its relative position within the coverage area of ​​each deformable image, and the weight can be a numerical value determined based on this relative position, characterizing the contribution of each deformable image to the pixel data at that pixel position. For example, the weight of a deformable image at a pixel position can be determined based on its distance from the effective boundary of the deformable image; the farther the pixel position is from the effective boundary (i.e., closer to the center region of the deformable image), the greater its weight. Alternatively, the weight of each deformable image can be determined based on its relative position within an overlapping region, allowing the weights of each deformable image to gradually transition from one side to the other along the direction of the overlapping region. In some embodiments, the weights of each deformable image at a pixel position can be normalized so that the sum of their weights at that pixel position is 1. Building upon the aforementioned approach, the pixel data at that specific pixel location in each deformed image can be weighted according to their respective weights to obtain the pixel data at that location on a unified canvas. This method enables smooth fusion of overlapping areas, making the fusion result physically closer to the real scene.

[0274] The above method allows all pixel positions on a unified canvas to be processed separately according to their coverage. Non-overlapping areas are directly valued to reduce computational overhead, while overlapping areas are weighted and fused to achieve a smooth transition, thereby improving image stitching quality.

[0275] According to some embodiments of this application, determining the weight of each deformed image at a pixel position based on the position of the pixel position in each deformed image may include: determining the weight of each deformed image at a pixel position based on the distance from the pixel position to the effective area boundary of each deformed image on a unified canvas; wherein the distance is positively correlated with the weight.

[0276] In some embodiments of this application, when determining the weight of each deformed image at a pixel location, the weight of each deformed image at a pixel location can be determined based on the distance from the pixel location to the boundary of the effective region of each deformed image on a unified canvas, and the distance can be positively correlated with the weight. That is, the weight of each deformed image at a pixel location can be determined by the distance from the pixel location to the boundary of the effective region of the deformed image; the farther the pixel location is from the boundary of the effective region of the deformed image, the greater the weight of the deformed image at that pixel location.

[0277] The effective region of the deformed image on the unified canvas is the area on the unified canvas that actually covers the deformed image and contains valid pixel data. Specifically, because the field of view of a bird's-eye view is limited, and the interpolation and resampling during image transformation only generates valid pixel data for locations with defined mapping relationships, the area covered by the deformed image after mapping to the unified canvas may not be a complete rectangular area. The effective region of the deformed image on the unified canvas is the portion of the actual coverage area on the unified canvas that contains valid pixel data. The boundary of the effective region is the dividing line between the effective region and the area on the unified canvas outside the effective region.

[0278] For example, the distance d_i(p) (distance field) from the pixel location to the boundary of the effective region of the i-th fisheye camera can be calculated. This distance reflects the depth of the pixel in the effective region—the closer to the center, the greater the weight; the closer to the boundary, the smaller the weight. In some embodiments, the weight of the corresponding deformed image at the pixel location can be determined based on the square of the distance from the pixel location to the boundary of the effective region of the deformed image of each fisheye camera. For example, it can be utilized... The weights of the deformed image corresponding to the i-th fisheye camera at the pixel positions are calculated, where, For the set of all fisheye cameras covering pixel p, To prevent division by zero by small constants, the weights must satisfy the normalization constraint: The final panoramic image is as follows: Finally, a panoramic image can be obtained. ,in, Let be the pixel data of the i-th deformed shape at the pixel position.

[0279] The above method allows pixel data at all pixel positions on a unified canvas to transition in a position-dependent and continuous manner, achieving a smooth transition, reducing stitching gaps, and improving image stitching quality.

[0280] According to some embodiments of this application, determining the second homography matrix corresponding to each fisheye camera based on each bird's-eye view may include: determining the motion parameters of the ship based on each bird's-eye view; and determining the second homography matrix corresponding to each fisheye camera based on the motion parameters of the ship and the relative poses between the fisheye cameras.

[0281] According to some embodiments of this application, determining the motion parameters of a ship based on various bird's-eye view images may include: performing global average pooling on each bird's-eye view image to obtain an average pooling result; stitching the average pooling result together to obtain a global feature vector; and determining the motion parameters of the ship based on the global feature vector.

[0282] According to some embodiments of this application, determining the second homography matrix corresponding to each fisheye camera based on the ship's motion parameters and the relative poses between the fisheye cameras may include: determining the rotation matrix of the reference camera at the current moment based on the rotation matrix of the reference camera in the previous moment and the ship's motion parameters; determining the rotation matrix of the fisheye cameras other than the reference camera at the current moment based on the rotation matrix of the fisheye cameras other than the reference camera relative to the reference camera; and determining the second homography matrix of each fisheye camera based on the rotation matrix of each fisheye camera at the current moment, the intrinsic parameter matrix of each fisheye camera, the translation vector of the fisheye camera relative to the reference camera, the normal vector of the reference plane, and the distance from the reference plane to the reference camera.

[0283] According to some embodiments of this application, before determining the second homography matrix corresponding to each fisheye camera based on each bird's-eye view, the method may further include: extracting multi-scale features of the bird's-eye view using a feature extraction network to obtain a feature map of the bird's-eye view; determining the second homography matrix corresponding to each fisheye camera based on each bird's-eye view; and determining the second projection map based on each bird's-eye view and the corresponding second homography matrix, which includes: determining the second homography matrix corresponding to each fisheye camera based on the feature map of each bird's-eye view; and determining the second projection feature map based on the feature map of each bird's-eye view and the corresponding second homography matrix.

[0284] According to some embodiments of this application, before determining the second homography matrix corresponding to each fisheye camera based on the feature maps of each bird's-eye view, the method may further include: determining weak texture regions based on the feature maps of the bird's-eye view; performing cross-camera attention interaction on the weak texture regions in the overlapping regions of adjacent fisheye cameras to obtain fused and enhanced weak texture regions; and obtaining enhanced feature maps based on the feature maps and the fused and enhanced weak texture regions.

[0285] According to some embodiments of this application, performing cross-camera attention interaction on weak texture regions in the overlapping region of adjacent fisheye cameras to obtain a fused and enhanced weak texture region may include: performing cross-camera attention interaction on weak texture regions in the overlapping region of adjacent fisheye cameras in a first direction to obtain a fused and enhanced first weak texture region; performing cross-camera attention interaction on weak texture regions in the overlapping region of adjacent fisheye cameras in a second direction to obtain a fused and enhanced second weak texture region; the first direction and the second direction are opposite in direction.

[0286] It should be noted that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, these will not be repeated here.

[0287] As can be seen from the above embodiments, some embodiments of this application provide a method for stitching 360-degree panoramic images from multiple fisheye cameras for ships (in some embodiments, it can be used for ships berthing offshore). This method supports flexible camera configurations, achieving wider field-of-view coverage, better stitching quality, higher real-time performance, and more accurate distance calibration, providing reliable panoramic visual assistance for ship operators and improving the safety and efficiency of berthing operations. The following is a combination of... Figure 9 The above-described embodiments of this application will be further described. Figure 9 This is a flowchart illustrating the process of obtaining panoramic images according to some embodiments of this application.

[0288] Step 1: Camera Layout and Image Acquisition The number N and layout of cameras are determined based on the ship's type, size, and berthing requirements, with N fisheye cameras arranged around the hull. Examples include a six-camera mode, an eight-camera mode, and more camera modes.

[0289] The number of lateral cameras N is determined according to the ship type and size. N fisheye cameras (N≥3) are arranged in an arc along the hull on the port and / or starboard sides of the ship. The field of view overlap rate of adjacent cameras is not less than 30% to ensure the reliability of cross-camera feature matching. The following parameters are obtained through pre-calibration: (1) the intrinsic parameter matrix of each camera. (2) Relative pose between adjacent cameras (4×4 homogeneous transformation matrix); (3) Overlapping area mask between adjacent cameras, which is a binary map that identifies the pixel area that is visible in both the view of camera i and camera i+1, and is pre-generated by projection calculation using calibration parameters.

[0290] Step 2: Cross-camera feature interaction network Input: N-channel bird's-eye view projection images + overlapping region mask The output consists of N sets of cross-camera augmentation features. For details, please refer to Figure 10 This is a cross-camera feature interaction network structure diagram of some embodiments of this application, wherein, Figure 10 The dashed lines in the diagram represent cross-camera attention interactions and indicate bidirectional chain propagation.

[0291] For berthing scenarios, the sea surface occupies more than 50% of the lateral field of view, and the sea surface area has poor texture. In step 2, the sea surface area features of camera C0 can obtain complementary information from the adjacent camera C1 through cross-camera attention (for example, C1 has richer water surface texture due to different lighting angles), thereby enhancing the feature representation of the sea surface area and reducing the probability of subsequent alignment steps failing due to missing features.

[0292] Step 2.1: Independent Feature Extraction For each camera's bird's-eye view, a shared-weights feature extraction network is used to extract multi-scale features. The feature extraction network employs the first four residual stages of ResNet, removing the final global pooling layer and fully connected layer to preserve spatial resolution information. .

[0293] Step 2.2: Enhanced Perception of Weak Textures on the Sea Surface A large area of ​​sea surface exists in the lateral view of a ship. These areas have poor texture and low feature extraction quality, which can easily lead to failure in subsequent alignment steps. This invention automatically identifies weak texture regions through texture complexity evaluation and enables cross-camera feature enhancement in these regions.

[0294] For each spatial location of the feature map Calculate the local texture complexity: A weak texture mask is generated based on the TC value. .

[0295] Step 2.3: Cross-camera attention interaction For adjacent cameras i and i+1, cross-camera attention interaction is performed in their overlapping region. This allows the features of each camera to "see" the features of adjacent cameras in the overlapping region, achieving cross-view information enhancement.

[0296] Using pre-computed overlapping region masks Attention is calculated only in the overlapping region to avoid invalid calculations in the non-overlapping region. Let the overlapping region features of camera i be... The overlapping region features of camera i+1 are The two have the same spatial dimensions (because the extent of the overlapping area in the two camera views is determined by the calibration parameters).

[0297] For details on the calculation process of cross-camera attention, please refer to the corresponding sections above.

[0298] Step 2.4: Full-chain feature propagation For N cameras, cross-camera attention propagates bidirectionally along the camera chain, enabling each camera to eventually fuse information from all cameras. Specifically, this can include forward propagation (passing information from the first camera to the last camera) and backward propagation (passing information from the last camera to the first camera).

[0299] Each step uses the same cross-camera attention module (parameter sharing). Forward propagation passes information from the first camera to the last, and backward propagation passes information from the last back to the first. Ultimately, the enhanced features from each camera are fused with information from all cameras. .

[0300] Step 3: Global Rigid Motion Estimation Network The input is: N sets of cross-camera augmentation features + Calibrate relative pose + Internal Reference The output is: 6-DOF global motion parameters and homography matrix.

[0301] When a ship is berthed, it experiences lateral movement and heave, causing the position of the same object to change in images from different cameras. Step 3 jointly estimates the 6-DOF motion of the ship's hull from the features of all cameras (e.g., lateral movement of 0.1m / frame, roll of 0.5° / frame), and then derives the homography matrix of each camera using calibration parameters and rigid constraints, achieving globally consistent coarse alignment. For example, the homography matrices of C0-C3 for the four cameras on the starboard side are all derived from the same set of 6 motion parameters, naturally ensuring consistency and avoiding the cascading error problem where C0 is correctly aligned but C3 is increasingly offset.

[0302] Step 3.1: Multi-camera feature aggregation Enhance the features of all cameras Perform global average pooling and concatenate the results to construct a global feature vector:

[0303]

[0304] in, For the average pooling result, Indicates global average pooling. This represents the global feature vector.

[0305] Global average pooling compresses the spatial features of each camera into a C-dimensional vector, which is then concatenated to obtain an N·C-dimensional global feature vector. This vector encodes the scene information observed by all cameras and serves as the input for subsequent motion estimation.

[0306] Step 3.2: 6-DOF motion regression The ship's 6-DOF motion parameters are regressed from the global feature vector using a fully connected network (multilayer perceptron):

[0307] Output vector ξ=[ω x ,ω y ,ω z ,v x ,v y ,v z The first three parameters are angular velocities (corresponding to the rotational angular velocities of the roll, pitch, and yaw axes), and the last three parameters are translational velocities (corresponding to the translational velocities of the x, y, and z axes). The Tanh activation function is used to limit the output to the range [-1, 1], and multiplying this by the preset maximum motion speed yields the actual motion parameters.

[0308] Step 3.3: Derive the homography matrix for each camera from global motion. Using rigid constraints, the homography matrix of each camera relative to a reference camera (such as C_0) is derived from the global motion ξ and the calibrated relative pose. For a detailed explanation of the implementation process, please refer to the corresponding sections above; it will not be repeated here.

[0309] Step 4: Rigidly Constrained Deformation Network Input: N sets of enhanced features + Homography matrix, output as: N deformable meshes .

[0310] After the global homography matrix provides coarse alignment, objects such as containers and mooring bollards on the dock are roughly aligned. However, non-rigid factors such as water waves and dynamic objects (such as small boats and floating objects) cause local misalignment. Step 4 performs fine adjustment of the predicted local deformation mesh under rigid constraints: rigid objects such as the dock maintain the alignment result of the global motion prediction, while non-rigid regions such as water waves are allowed to have small-range offsets. However, the degree of offset is limited by the rigid constraint loss and will not destroy global consistency.

[0311] (1) Global rigid coarse alignment The homography matrix derived in step 3 is used to coarsely align the enhanced features of each camera. The feature maps are then mapped to the coordinate system of the reference camera C_0 through homography transformation to obtain the projected feature images. :

[0312] After coarse alignment, the feature maps of each camera are roughly aligned, but there are still local deviations caused by non-rigid factors, which require fine adjustment by a local deformation network.

[0313] (2) Local Deformation Mesh Prediction Define a regular grid on the bird's-eye view. (Resolution and feature map alignment), the local deformation network (i.e., the deformable mesh offset prediction network) takes the concatenation of coarse alignment features and original enhancement features as input to predict the offset of each mesh vertex. : .

[0314] in The local offset predicted by the network has a size of H. grid ×W grid ×2 (one channel each in the x and y directions) represents the offset of each mesh vertex relative to the position of the regular mesh. Final deformed mesh. The sum of the regular grid and the offset:

[0315] in It is a regular grid (uniformly distributed grid vertices). This is the final deformed mesh. The local deformable network uses a 5-layer convolutional structure and outputs 2-channel offsets (x and y directions) to handle local non-rigid factors that cannot be captured by global rigid motion.

[0316] (3) Rigid constraint loss Local deformation The deviation from the global rigid motion cannot be too far, otherwise the rigid constraint will be violated. Therefore, the rigid offset is derived from the global homography. That is, the mapping offset of the homography matrix to the vertices of the regular mesh. The rigid constraint loss function is defined as the difference between the local deformation offset and the rigid offset. The specific form of the rigid constraint loss function can be...

[0317] . It is the offset at vertex k in the mesh predicted by the local deformation network, representing the local offset of the vertex relative to the regular mesh position. Its physical meaning is: local position correction caused by local non-rigid factors (such as water waves, dynamic floating objects, etc.) that cannot be captured by global rigid motion.

[0318] Calculation method: The output is directly from the local deformation network, i.e.:

[0319] Where k is the grid vertex index. ∈R^2 (offsets in the x and y directions). Equivalently, = - That is, the difference between the vertex positions of the final deformed mesh and the vertex positions of the regular mesh.

[0320] This is the offset at mesh vertex k when only global rigid motion is considered (excluding local deformation). Its physical meaning is: if this vertex is completely composed of the global homography matrix... The drive determines where it should move from the regular grid position; this movement amount is the rigid offset. The specific calculation process can be found in the detailed explanation of the corresponding section above, and will not be repeated here.

[0321] The rigid constraint loss function enables fine-tuning of local deformation within a globally rigid framework: for regions of rigid objects (docks, containers, etc.). Maintain global alignment; non-rigid areas (water waves, dynamic objects, etc.). Allowed deviation However, the degree of deviation is affected Constraints do not compromise global consistency. The rigid constraint loss function anchors local deformations to the global rigid motion prediction, and consistency is a priori guaranteed—local deformations can only be fine-tuned within the rigid frame, fundamentally disallowing deviations from global consistency.

[0322] Step 5: Cross-camera consistency constraints Input: N deformable meshes +overlapping area mask The output is a deformed mesh after consistency constraints (which may include overlapping regions mapped by a consistency loss function). Consistency loss function with boundary gradient ).

[0323] In berthing scenarios, the same mooring bollard may appear simultaneously in images C0 and C1 (overlapping area). If the mapping positions of the mooring bollard in the deformed meshes of C0 and C1 are inconsistent, ghosting or breaks will appear in the stitching result. Step 5 enforces that the mapping positions of the same object in the overlapping area are consistent, reducing ghosting. At the same time, gradient consistency constraints ensure a smooth transition at the boundary of the overlapping area, reducing stitching gaps.

[0324] (1) Consistency loss function for overlapping region mapping For the overlapping region of adjacent cameras i and i+1, let pixel p be mapped to panoramic coordinates as follows under the deformed grids of the two cameras: and (Calculated via bilinear mesh interpolation), the consistency loss function for overlapping region mapping requires that both be identical. For the specific calculation process, please refer to the detailed explanation in the corresponding section above. The physical meaning of this constraint is: for the same scene point in the overlapping region, the panoramic coordinates obtained should be the same regardless of which camera's deformed mesh mapping is used.

[0325] (2) Boundary gradient consistency loss function Simply constraining the mapping position to be consistent may lead to discontinuous jumps in the deformed mesh at the boundary of the overlapping region. Therefore, it is necessary to further constrain the gradient continuity of the deformed mesh at the boundary of the overlapping region. Specifically, the detailed calculation process of the boundary gradient consistency loss function can be found in the corresponding section above. This loss ensures a smooth transition of the deformed mesh at the boundary of the overlapping region between adjacent cameras.

[0326] Among them, the overlapping region mapping consistency loss function ensures the local consistency of the overlapping region, the boundary gradient consistency loss function ensures that the consistency is smoothly propagated along the camera chain to global consistency, and all deformed meshes are constrained by the same set of global motion parameters (step 3). The cross-camera consistency constraint and the rigid constraint deformed network work together. The consistency constraint is embedded in the optimization objective of the deformed mesh. The deformed mesh satisfies the cross-camera consistency during the learning process, and no post-processing correction steps are required during inference.

[0327] Step 6: Geometry Priors for the Sea Surface Input: N deformable meshes +projected image +Camera pose parameters Camera pose parameters ( + intrinsic parameter matrix The output is a deformed mesh constrained by the sea surface (including the sea surface constraint loss function). ).

[0328] When berthing, approximately 50%-70% of the lateral field of view is the sea surface, which has poor texture (especially at night or on calm waters). Traditional methods cannot align the sea surface area through feature matching. This step utilizes the geometric prior that the sea surface is approximately horizontal (Z=0) to directly calculate the ideal panoramic coordinates of the sea surface pixels using camera pose parameters. It does not rely on any texture features and can correctly align even when the sea surface is completely textureless. The specific implementation consists of the following three steps: Step 6.1: Sea surface area segmentation Use the enhanced feature map extracted in step 2 Alternatively, use the global rigid coarse alignment obtained in step 4. A lightweight segmentation head with 1x1 convolution and sigmoid activation outputs a binary mask of the sea surface. :

[0329] in, =1 indicates that pixel p belongs to the sea surface area. =0 indicates a non-sea surface region. This design reuses existing features, eliminating the need for an additional feature extraction network.

[0330] Step 6.2: Calculation of Ideal Panoramic Coordinates for Sea Surface Pixels For pixel p in the sea surface region, (Through camera internal parameters) Given the Z=0 plane geometric prior, its ideal coordinates on the panoramic canvas are directly calculated. The calculation principle is: pixel p is processed through - The back projection is a ray in the camera coordinate system. The ray is transformed to the panoramic coordinate system using the camera pose. The intersection with the Z=0 plane is used to obtain the 3D sea surface point, which is then linearly mapped to the panoramic canvas coordinate system.

[0331] Step 6.3: Sea Surface Level Constraint Loss For the pixel p=( in the sea surface region) ), which uses deformable mesh The actual panoramic coordinates mapped are Pixel p falls within a rectangular cell of the deformable mesh G_i, which is bounded by four mesh vertices (top left TL, top right TR, bottom left BL, and bottom right BR). The offset of pixel p is calculated using bilinear interpolation of the offsets of these four vertices; see the detailed explanation in the corresponding section above for the specific calculation process.

[0332] Based on the above, the ideal coordinates calculated geometrically are given. The sea surface plane constraint loss function forces both to be consistent. For the specific calculation method of the sea surface plane constraint loss function, please refer to the detailed explanation in the corresponding section above. It should be noted that the sea surface plane constraint loss function only takes effect during training, forcing the network to output deformed meshes in the sea surface region that are close to the geometrically ideal position. During inference, the network directly outputs deformed meshes that satisfy the constraints.

[0333] Step 7: Timing Rigid Constraints The input is: the deformed mesh of the current frame + the deformed mesh of the previous frame + the motion parameters of the current frame + the motion parameters of the previous frame; the output is the temporally smoothed deformed mesh (which may include a motion smoothing loss function). +Grid temporal smoothing loss function ).

[0334] During berthing, ships experience continuous roll, pitch, and heave movements. If each frame is stitched independently, the stitched result will exhibit flickering and jitter. Step 7 utilizes the physical continuity of rigid body motion to constrain temporal stability: the roll angle will not abruptly change between frames (motion smoothing loss function), and the deformed mesh will not jump between frames (mesh temporal smoothing loss function). This makes the berthing monitoring image smooth and stable, allowing the operator to clearly observe the dock and obstacles without visual fatigue or misjudgment due to image jitter.

[0335] As a rigid body, the hull's 6 degrees of freedom motion is continuous and smooth in time—a direct consequence of physical laws (rigid body motion cannot have instantaneous jumps). This application utilizes this physical prior to constrain temporal stability from two levels. For the specific calculation process of the two loss functions mentioned above, please refer to the detailed explanation in the corresponding sections above.

[0336] Specifically, the motion smoothing loss function constrains velocity (the amount of change in motion parameters between frames) to prevent abrupt motion changes; and it also constrains acceleration (the rate of change of velocity) to prevent drastic changes in motion trends. The grid temporal smoothing loss function directly constrains the changes in deformed grids between adjacent frames, reducing high-frequency jitter at the grid level.

[0337] Temporal stability is naturally achieved through the physical constraints of rigid body motion. Only the rate of change of 6 motion parameters needs to be constrained, eliminating the need for an additional temporal modeling network. This makes it more robust (physical constraints automatically pull abnormal frames back to a reasonable range) and more efficient (no computationally intensive modules such as 3D convolution).

[0338] Regarding the relationship between rigid constraint loss, cross-camera consistency constraint (step 5), sea surface geometric prior (step 6), temporal rigid constraint (step 7), and the final deformed mesh: the aforementioned four constraints are loss functions during the training phase, not post-processing operations during the inference phase. Their relationship with the final deformed mesh... The relationship is as follows: (1) Training phase Deformed mesh offset prediction networks can enhance features and homography matrix Input: Local offset This leads to the formation of deformed meshes. During training, all of the following loss functions are applied simultaneously to the deformable mesh. The network parameters θ are jointly optimized through backpropagation. In some embodiments, the total loss function can be:

[0339] in, The local offset is constrained to not deviate too far from the rigid prediction. and Constraints include consistent mapping in overlapping regions and continuous boundary gradients. The deformation of the constrained sea surface region conforms to planar geometry. and Constrain the temporal continuity of motion parameters and deformable meshes. The weighting coefficients for each loss function can be determined through parameter tuning using the validation set.

[0340] (2) Reasoning stage After training, the network parameters θ have learned to output the optimal deformable mesh under all constraints. During inference, the deformable network directly outputs... No post-processing or constraint correction steps are required. Since the network optimizes all losses simultaneously during training, the output... Naturally satisfies: rigid constraints (local deformation does not deviate too far from the global rigid motion), cross-camera consistency (overlapping areas of adjacent cameras are mapped consistently), sea surface prior (sea surface area alignment conforms to planar geometry), and temporal smoothness (inter-frame deformable meshes are continuous and do not jump).

[0341] Rigid constraint loss (step 4(3)), cross-camera consistency constraint (step 5), sea surface geometric prior (step 6), temporal rigid constraint (step 7) remain unchanged. Instead of a traditional computational process, constraints are embedded into the network parameters θ through the loss function during the training phase, allowing the inference process to directly output the network parameters. All constraints are satisfied.

[0342] Step 8: Unify the canvas and blend Input: N deformable meshes + N-way aerial view Output: Panoramic image .

[0343] Steps 4 through 7 output the deformed mesh for each camera. Step 8's task is to transform the projected image of each camera using the deformed mesh, and then merge all the transformed images onto the same panoramic canvas to output the final panoramic image.

[0344] N deformed images are drawn onto a single panoramic canvas. For areas covered by only one image (such as the center of the camera's field of view), the pixels of that image are used directly. For areas covered by multiple overlapping images (such as overlapping areas of adjacent cameras), a weighted fusion is performed based on distance field weights—pixels closer to the image center have a larger weight, and pixels closer to the boundary have a smaller weight, achieving a smooth transition.

[0345] Step 8.1: Image Transformation Based on Deformed Mesh Using deformable mesh For each camera's bird's-eye view, an image transformation is performed. For each pixel position p in the transformed image, its corresponding position in the original image is calculated using bilinear interpolation of the deformed mesh, and then pixel values ​​are sampled from the original image. Deformable meshes inherently guarantee the smoothness of transformations (the offsets of adjacent mesh vertices change continuously), eliminating the need for additional smoothing regularization.

[0346] Step 8.2: Building a Unified Canvas The union of the coordinate ranges of all deformed images is used to determine the size of a unified canvas. The deformed grid coordinates of each camera are then translated to a unified coordinate system, aligning all deformed images to the same canvas.

[0347] Step 8.3: Distance Field Weight Fusion For each pixel p on the canvas, count all camera images covering that pixel. If only one image covers it, use that pixel value directly; if multiple images overlap, perform weighted fusion based on distance field weights to achieve a seamless transition. For each camera i covering pixel p, calculate the distance d_i(p) (distance field) from that pixel to the boundary of the effective region of camera i, reflecting the depth of the pixel in the effective region—the closer to the center, the larger the weight; the closer to the boundary, the smaller the weight.

[0348] Where C(p) is the set of all cameras covering pixel p, and ε is a small constant to prevent division by zero. The weights satisfy the normalization constraint: The final panoramic image is as follows:

[0349] Finally, a panoramic image can be obtained.

[0350] As can be seen from the above, some embodiments of this application provide a method for stitching panoramic images of ships using multiple fisheye cameras based on rigid constraint joint optimization. By utilizing the physical prior of multiple fisheye cameras fixedly mounted on the same rigid body, and through cross-camera feature interaction, global rigid motion estimation, rigid constraint deformation prediction, and cross-camera consistency constraints, a one-time global joint stitching of all cameras is achieved, fundamentally avoiding the error accumulation problem of pairwise cascaded stitching.

[0351] Unlike the traditional divide-and-conquer method of pairwise cascading stitching (processing each pair independently before cascading), this invention is "unified," specifically utilizing rigid constraints to jointly optimize the stitching of all cameras into a globally consistent problem. Specifically, (1) a cross-camera feature interaction network is adopted: using the Transformer cross-camera attention mechanism, the features of each camera can "see" the overlapping features of adjacent cameras, realizing cross-camera information enhancement, especially enabling weak texture regions to obtain supplementary information from adjacent cameras. (2) a global rigid motion estimation network: jointly estimating the 6-DOF rigid body motion (3-axis rotation + 3-axis translation) of the ship from the features of all cameras, as a global prior for the deformation of all cameras, rather than estimating homography independently pairwise. (3) a rigid constraint deformation network: under the guidance of the global rigid motion prior, predicting the local deformation mesh of each camera, where the global rigid motion provides coarse alignment, and the local deformation network is finely adjusted under rigid constraints. The rigid constraint loss ensures that the local deformation does not deviate too far from the global motion prediction. (4) Cross-camera consistency constraint: In the overlapping area of ​​adjacent cameras, the mapping of the deformed meshes of the two cameras to the overlapping area is forced to be consistent. Global constraints are achieved through the overlap area consistency loss, eliminating the need for post-processing mesh alignment steps. (5) Sea surface planar geometry prior: Utilizing the geometric prior that the sea surface is approximately planar, planar constraints are applied to the deformed meshes of the sea surface area to ensure that the alignment of the sea surface area conforms to the planar geometric relationship, fundamentally solving the problem of difficulty in aligning weak textures on the sea surface. (6) Temporal rigidity constraint: Utilizing the temporal continuity of the rigid body motion of the ship, the stability of the stitching result in time is ensured through motion smoothing loss and acceleration constraints, eliminating the need for an independent temporal stabilization network. Furthermore, the cross-camera feature interaction network propagates bidirectionally along the camera chain, and the global rigid motion estimation network aggregates features from N cameras, naturally supporting any number (N≥3) of fisheye camera configurations. Adding a new camera does not require modifying the network structure; only the calibration parameters of the new camera need to be added to the system. Cross-camera attention automatically propagates along the extended camera chain, and global motion estimation automatically incorporates the feature information of the new camera. This allows the system to flexibly adapt to the berthing needs of different types and sizes of ships (such as bulk carriers, container ships, tankers, cruise ships, etc.).

[0352] Through the above, some embodiments of this application can achieve: (1) Global consistency without error accumulation: By jointly optimizing the deformable mesh of all cameras through rigid constraints, global consistency is naturally guaranteed by the optimization target, fundamentally eliminating the error accumulation problem of cascade stitching. (2) Improve accuracy by using rigid priors: By using the rigid constraints of the camera array, the relative pose obtained by calibration is embedded into the network as a hard constraint, which greatly reduces the degrees of freedom that the network needs to learn and improves the alignment accuracy. (3) Cross-camera information enhancement: Weak texture regions (such as sea surface) can obtain complementary information from adjacent cameras, improving the robustness of feature matching. (4) Global motion prior reduces complexity: Using 6-DOF global motion to replace N-1 independent homography estimates is more robust and efficient. (5) No post-processing error correction is required: Cross-camera consistency constraints are effective during training and inference, and no separate mesh alignment step is required. (6) Sea surface weak texture is aligned by geometric hard constraints: The sea surface plane prior uses the geometric relationship of Z=0 to deterministically calculate the ideal mapping position of sea surface pixels, without relying on feature matching, and can be correctly aligned even if there is no texture at all. (7) Temporal stability without additional network: The physical continuity of rigid body motion naturally ensures temporal stability, eliminating the need for computationally intensive temporal modeling modules such as 3D convolution. Furthermore, it supports configurations with six, eight, or other camera numbers. The specific number of cameras can be flexibly selected based on the ship type (bulk carrier, container ship, tanker, cruise ship, etc.), size, and berthing requirements. Adding a camera does not require modifying the network structure; simply adding calibration parameters will automatically incorporate it into the stitching process.

[0353] This application also provides a fisheye image stitching system, which may include: Fisheye cameras installed at different locations on the ship are used to acquire fisheye images; A processor that is communicatively connected to each fisheye camera is used to implement the steps of any of the above-mentioned fisheye image stitching methods.

[0354] The processor, which is communicatively connected to each fisheye camera, is a device used to receive fisheye images acquired by each fisheye camera and execute any of the aforementioned fisheye image stitching methods. For example, the processor may include a central processing unit, a graphics processing unit, a digital signal processor, a microprocessor, an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA), etc.

[0355] Specifically, the processor can connect to each fisheye camera via wired or wireless communication. For example, the processor and each fisheye camera can communicate via Ethernet, Controller Area Network, Universal Serial Bus, or Bluetooth.

[0356] This application also provides a vessel that may include the aforementioned fisheye image stitching system.

[0357] For descriptions of the fisheye image stitching method, fisheye image stitching system, and related parts of the ship provided in some embodiments of this application, please refer to the detailed description of the corresponding parts in the offset prediction network training method provided in some embodiments of this application, which will not be repeated here.

[0358] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and they should all be covered within the scope of the claims and specification of this application. In particular, as long as there is no structural conflict, the various technical features mentioned in the embodiments can be combined in any way. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for training a shift prediction network, the method comprising: include: Obtain bird's-eye view samples corresponding to multiple fisheye cameras on the ship; The first homography matrix corresponding to each fisheye camera is determined based on each of the bird's-eye image samples, and the first projected image is determined based on each of the bird's-eye image samples and the corresponding first homography matrix. A first deformable mesh is determined based on the bird's-eye view sample. Based on each bird's-eye view sample, each first projected image, and the deformable mesh offset prediction network, the first local deformation offset of the first deformable mesh and each fisheye camera is predicted. The deformable mesh offset prediction network has initial parameters. Based on the first local deformation offset corresponding to each fisheye camera, the first deformation mesh, each of the first homography matrices, and the overlapping area of ​​adjacent fisheye cameras, a deformation mesh geometric loss function is constructed. The initial parameters of the deformable mesh offset prediction network are updated according to the deformable mesh geometric loss function.

2. The offset prediction network training method of claim 1, wherein, Based on the first local deformation offset corresponding to each fisheye camera, the first deformable mesh, each of the first homography matrices, and the overlapping areas of adjacent fisheye cameras, a deformable mesh geometric loss function is constructed, including: The rigid offset corresponding to each fisheye camera is determined based on the first deformable mesh and each first homography matrix, and the rigid constraint loss function is determined based on the rigid offset corresponding to each fisheye camera and the first local deformation offset corresponding to each fisheye camera. Based on the first deformed mesh and the first local deformation offset corresponding to each fisheye camera, determine the second deformed mesh corresponding to each fisheye camera; based on the second deformed mesh corresponding to adjacent fisheye cameras and the overlapping area of ​​adjacent fisheye cameras, determine the cross-camera consistency loss function; The deformable mesh geometric loss function is constructed based on the rigid constraint loss function and the cross-camera consistency loss function.

3. The offset prediction network training method of claim 2, wherein, The cross-camera consistency loss function includes an overlapping region mapping consistency loss function, which is determined based on the second deformed mesh corresponding to adjacent fisheye cameras and the overlapping region of adjacent fisheye cameras, including: Using the second deformable mesh corresponding to the adjacent fisheye cameras, each overlapping pixel in the overlapping area of ​​the adjacent fisheye cameras is mapped to the panoramic coordinate system to obtain the first panoramic pixel and the second panoramic pixel corresponding to each overlapping pixel. Based on the first panoramic pixel and the second panoramic pixel corresponding to each of the overlapping pixels, the mapping consistency loss function of the overlapping region is determined.

4. The offset prediction network training method of claim 3, wherein, The cross-camera consistency loss function further includes a boundary gradient consistency loss function, which is determined based on the second deformed mesh corresponding to adjacent fisheye cameras and the overlapping region of adjacent fisheye cameras, including: Determine the gradient of the second deformable mesh corresponding to the adjacent fisheye camera at the boundary of the corresponding overlapping region; The boundary gradient consistency loss function is determined based on the gradient of the second deformed mesh corresponding to the adjacent fisheye camera at the boundary of the corresponding overlapping region.

5. The offset prediction network training method of claim 1, wherein, Also includes: The corresponding sea surface area image is determined based on each of the aforementioned bird's-eye view samples; The sea surface area image, the intrinsic parameter matrix of each fisheye camera, the sea level, the first deformable mesh, and the first local deformation offset corresponding to each fisheye camera are used to determine the sea surface plane constraint loss function. The initial parameters of the deformable mesh offset prediction network are updated according to the deformable mesh geometric loss function, including: The initial parameters of the deformable mesh offset prediction network are updated based on the deformable mesh geometric loss function and the sea surface plane constraint loss function.

6. The offset prediction network training method of claim 5, wherein, Based on the sea surface region image, the intrinsic parameter matrix of each fisheye camera, the sea level, the first deformable mesh, and the first local deformation offset corresponding to each fisheye camera, the sea surface plane constraint loss function is determined, including: Based on the intrinsic parameter matrix of each fisheye camera and the sea level, each sea surface pixel in the sea surface area image is mapped to the panoramic coordinate system to obtain the ideal panoramic pixel corresponding to each sea surface pixel. Based on the first deformable mesh and the first local deformation offset corresponding to each fisheye camera, the second deformable mesh corresponding to each fisheye camera is determined; using the second deformable mesh corresponding to each fisheye camera, each sea surface pixel in the sea surface area image is mapped to the panoramic coordinate system to obtain the sea surface panoramic pixel corresponding to each sea surface pixel. The sea surface plane constraint loss function is determined based on the ideal panoramic pixels and the sea surface panoramic pixels.

7. The offset prediction network training method of claim 1, wherein, Also includes: Based on the first deformable mesh and the first local deformation offset corresponding to each fisheye camera, determine the second deformable mesh corresponding to each fisheye camera; The mesh temporal smoothing loss function is determined based on the second deformed mesh corresponding to each fisheye camera and the second historical deformed mesh corresponding to each fisheye camera in the historical frame. The initial parameters of the deformable mesh offset prediction network are updated according to the deformable mesh geometric loss function, including: The initial parameters of the deformable mesh offset prediction network are updated based on the deformable mesh geometric loss function and the mesh temporal smoothing loss function.

8. The shift prediction network training method of claim 1, wherein, Determine the first homography matrix corresponding to each fisheye camera based on each of the aforementioned bird's-eye image samples, including: The motion parameters of the ship are determined based on the aforementioned bird's-eye view samples; The first homography matrix corresponding to each fisheye camera is determined based on the motion parameters of the ship and the relative poses between the fisheye cameras.

9. The offset prediction network training method of claim 8, wherein, Also includes: The motion smoothing loss function is determined based on the ship's motion parameters and the ship's historical motion parameters in historical frames. The initial parameters of the deformable mesh offset prediction network are updated according to the deformable mesh geometric loss function, including: The initial parameters of the deformable mesh offset prediction network are updated based on the deformable mesh geometric loss function and the motion smoothing loss function.

10. The offset prediction network training method according to claim 9, characterized in that, The motion smoothing loss function is determined based on the ship's motion parameters and the ship's historical motion parameters in historical frames, including: The velocity loss function and acceleration loss function are determined based on the ship's motion parameters and the ship's historical motion parameters in historical frames. The motion smoothing loss function is determined based on the velocity loss function and the acceleration loss function.

11. The method for training a offset prediction network according to any one of claims 1 to 10, characterized in that, Before determining the first homography matrix corresponding to each fisheye camera based on each of the bird's-eye image samples, the method further includes: Multi-scale features of the bird's-eye view sample are extracted using a feature extraction network to obtain a feature map sample of the bird's-eye view sample; Based on each of the aforementioned bird's-eye image samples, a first homography matrix corresponding to each fisheye camera is determined; based on each of the aforementioned bird's-eye image samples and the corresponding first homography matrix, a first projected image is determined, including: The first homography matrix corresponding to each fisheye camera is determined based on the feature map samples of each bird's-eye image sample, and the first projection feature image is determined based on the feature map samples of each bird's-eye image sample and the corresponding first homography matrix.

12. The offset prediction network training method according to claim 11, characterized in that, Before determining the first homography matrix corresponding to each fisheye camera based on the feature map samples of each of the bird's-eye view samples, the method further includes: Weak texture region samples are determined based on the feature map samples of the bird's-eye view samples; Cross-camera attention interaction is performed on weak texture region samples in the overlapping region of adjacent fisheye cameras to obtain fused and enhanced weak texture region samples. An enhanced feature map sample is obtained based on the feature map of the bird's-eye view sample and the fused and enhanced weak texture region sample.

13. The offset prediction network training method according to claim 12, characterized in that, Perform cross-camera attention interaction on weak texture region samples in the overlapping regions of adjacent fisheye cameras to obtain fused and enhanced weak texture region samples, including: For weak texture region samples in the overlapping region of adjacent fisheye cameras, cross-camera attention interaction is performed in the first direction to obtain the first weak texture region sample after fusion enhancement. For weak texture region samples in the overlapping region of adjacent fisheye cameras, cross-camera attention interaction is performed in the second direction to obtain a fused and enhanced second weak texture region sample; the first direction is opposite to the second direction.

14. A fisheye image stitching method, characterized in that, include: Acquire fisheye images captured by multiple fisheye cameras on the ship, and obtain corresponding bird's-eye views based on each fisheye image; Based on each of the bird's-eye views, determine the second homography matrix corresponding to each of the fisheye cameras, and based on each of the bird's-eye views and the corresponding second homography matrix, determine the second projection image; The third deformable mesh is determined based on the bird's-eye view. The third deformable mesh is then predicted to correspond to the second local deformation offset of each fisheye camera based on each bird's-eye view, each second projection view, and the deformable mesh offset prediction network trained using the offset prediction network training method according to any one of claims 1 to 13. Based on the third deformable mesh and the second local deformation offset corresponding to each fisheye camera, determine the fourth deformable mesh corresponding to each fisheye camera; The panoramic image is determined based on the fourth deformable grid corresponding to each fisheye camera and the bird's-eye view.

15. The fisheye image stitching method according to claim 14, characterized in that, Based on the fourth deformable grid corresponding to each fisheye camera and each bird's-eye view, a panoramic image is determined, including: Based on the fourth deformation grid corresponding to each fisheye camera, image transformation is performed on each bird's-eye view to obtain a deformed image of each bird's-eye view; The coordinate range of the unified canvas is determined based on the coordinate range of the deformed image; Each of the deformed images is mapped onto the unified canvas to obtain a panoramic image.

16. The fisheye image stitching method according to claim 15, characterized in that, Mapping each of the deformed images onto the unified canvas yields a panoramic image, including: If a single deformed image is mapped to a pixel position on the unified canvas, the pixel data of the pixel position is determined based on the pixel data of the deformed image at that pixel position; If multiple deformed images are mapped to pixel positions on the unified canvas, the weight of each deformed image at the pixel position is determined according to the position of the pixel position in each deformed image, and the pixel data of each deformed image at the pixel position is weighted and fused according to the weight of each deformed image at the pixel position to obtain the pixel data at the pixel position.

17. The fisheye image stitching method according to claim 16, characterized in that, Determining the weight of each deformed image at the pixel position based on the pixel position in each deformed image includes: The weight of each deformed image at the pixel position is determined based on the distance from the pixel position to the effective region boundary of each deformed image on the unified canvas; wherein the distance is positively correlated with the weight.

18. The fisheye image stitching method according to claim 14, characterized in that, Determine the second homography matrix corresponding to each fisheye camera based on each of the aforementioned bird's-eye views, including: The motion parameters of the ship are determined based on the aforementioned bird's-eye view; The second homography matrix corresponding to each fisheye camera is determined based on the motion parameters of the ship and the relative poses between the fisheye cameras.

19. The fisheye image stitching method according to claim 18, characterized in that, Determining the motion parameters of the ship based on the aforementioned bird's-eye view includes: Global average pooling is performed on each of the aforementioned bird's-eye views to obtain the average pooling result; The average pooling results are concatenated to obtain the global feature vector; The motion parameters of the ship are determined based on the global feature vector.

20. The fisheye image stitching method according to claim 18, characterized in that, Determine the second homography matrix corresponding to each fisheye camera based on the ship's motion parameters and the relative poses between the fisheye cameras, including: Based on the rotation matrix of the reference camera in the fisheye camera at the previous moment and the motion parameters of the ship, determine the rotation matrix of the reference camera at the current moment; Based on the rotation matrix of the fisheye camera (excluding the reference camera) relative to the reference camera, determine the rotation matrix of the fisheye camera (excluding the reference camera) at the current moment; The second homography matrix of each fisheye camera is determined based on the rotation matrix of each fisheye camera at the current moment, the intrinsic parameter matrix of each fisheye camera, the translation vector of the fisheye camera relative to the reference camera, the normal vector of the reference plane, and the distance from the reference plane to the reference camera.

21. The fisheye image stitching method according to any one of claims 14 to 20, characterized in that, Before determining the second homography matrix corresponding to each fisheye camera based on each of the aforementioned bird's-eye views, the method further includes: Multi-scale features of the bird's-eye view are extracted using a feature extraction network to obtain the feature map of the bird's-eye view; Based on the bird's-eye views, a second homography matrix corresponding to each fisheye camera is determined; based on the bird's-eye views and the corresponding second homography matrix, a second projection image is determined, including: The second homography matrix corresponding to each fisheye camera is determined based on the feature map of each bird's-eye view, and the second projection feature map is determined based on the feature map of each bird's-eye view and the corresponding second homography matrix.

22. The fisheye image stitching method according to claim 21, characterized in that, Before determining the second homography matrix corresponding to each fisheye camera based on the feature maps of each of the aforementioned bird's-eye views, the method further includes: Weak texture areas are determined based on the feature map of the bird's-eye view; Cross-camera attention interaction is performed on the weak texture regions in the overlapping regions of adjacent fisheye cameras to obtain fused and enhanced weak texture regions. An enhanced feature map is obtained based on the feature map and the fused and enhanced weak texture region.

23. The fisheye image stitching method according to claim 21, characterized in that, Perform cross-camera attention interaction on weak-texture regions in the overlapping regions of adjacent fisheye cameras to obtain fused and enhanced weak-texture regions, including: For weak texture regions in the overlapping regions of adjacent fisheye cameras, cross-camera attention interaction is performed in the first direction to obtain the first weak texture region after fusion enhancement. For weak texture regions in the overlapping regions of adjacent fisheye cameras, cross-camera attention interaction is performed in a second direction to obtain a fused and enhanced second weak texture region; the first direction is opposite to the second direction.

24. A fisheye image stitching system, characterized in that, include: Fisheye cameras installed at different locations on the ship are used to acquire fisheye images; A processor communicatively connected to each of the fisheye cameras is used to implement the steps of the fisheye image stitching method as described in any one of claims 14 to 23.

25. A ship, characterized in that, Including the fisheye image stitching system as described in claim 24.