A method, system, and medium for acquiring panoramic surround view images of ships.

By installing multiple fisheye cameras on the ship and using calibration and stitching algorithms to generate panoramic surround-view images, the problem of limited field of view of traditional cameras is solved, enabling stable and safe navigation of the ship in complex environments.

CN117412175BActive Publication Date: 2026-11-10HUAZHONG UNIV OF SCI & TECH +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311275127.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-09-28
Publication Date
2026-11-10
Estimated Expiration
2043-09-28

AI Technical Summary

Technical Problem

In traditional ship perception systems, the detection field of view and scene of cameras are limited, making it impossible to effectively perceive floating objects and nearby objects in complex aquatic environments.

Method used

Multiple fisheye cameras are mounted on a gimbal stabilizer to acquire distortion-free images through calibration and correction. The perspective transformation matrix is ​​calculated using the coordinate system information surrounding the landmark, and the images are stitched together to form a panoramic view. Combined with deep neural networks and feature matching algorithms, a stable panoramic bird's-eye view is generated.

Benefits of technology

It enables real-time acquisition of panoramic images of ships, solves the problem of blind spots in complex environments, improves the stability and safety of ships, and provides new ideas for assisted driving and berthing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117412175B_ABST
    Figure CN117412175B_ABST
Patent Text Reader

Abstract

This invention discloses a method, system, and medium for acquiring panoramic images of ships, belonging to the field of shipbuilding. The invention controls an unmanned calibration vessel to perform a periodic encirclement movement around the main ship. Using onboard feature spheres and image segmentation and recognition technologies, it maps world coordinates in the world coordinate system to pixel coordinates in the image coordinate system. Then, by finding suitable target points to minimize reprojection error, the optimal projection transformation matrix for the patch region is obtained. The homography matrix between each patch region is obtained through image feature point extraction, filtering, and constraint matching to achieve stitching from a specific viewpoint. For overlapping areas of N views, feature point extraction, filtering, and constraint matching are performed to obtain the homography matrix between different views. Finally, the area where the ship itself is located is covered with an icon. Through the above method, a continuous bird's-eye view panoramic image of the ship is acquired.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of shipbuilding, and more specifically, relates to a method, system, and medium for acquiring panoramic surround view images of ships. Background Technology

[0002] In recent decades, ships have been widely used in fisheries, forestry, water quality monitoring, military reconnaissance, and marine exploration due to their ability to operate in vast water environments such as rivers, lakes, and oceans. In these maritime missions, ships need to complete challenging single-operation and group operations in complex environments, which means that ship operators need excellent observation skills regarding the environment around the vessel.

[0003] Traditional shipboard awareness systems typically consist of multiple sensors, including cameras, lidar, millimeter-wave radar, sonar detectors, and marine radar. Due to the physical properties of radar sensors, lidar and millimeter-wave radar are ineffective at detecting floating objects in complex water environments. Sonar detectors are primarily used for underwater environmental sensing. Marine radar has an excessively long detection range, making it unable to detect or perceive nearby objects or surface targets. A single camera has a limited field of view and limited detection capabilities. Summary of the Invention

[0004] To address the shortcomings and improvement needs of existing technologies, this invention provides a method, system, and medium for acquiring panoramic surround-view images of ships, aiming to solve the technical problem that the detection field of view and scene are relatively limited by existing single cameras.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a method for acquiring panoramic surround-view images of a ship, the method comprising:

[0006] Pre-calibration stage:

[0007] Obtain N calibration images captured by N cameras on the main ship, where N≥4;

[0008] Based on the coordinates of the illuminated markers on the unmanned calibration vessel surrounding the main ship in the world coordinate system and the pixel coordinates in the image coordinate system, N different families of perspective transformation matrices are obtained; then, based on the N different families of perspective transformation matrices, N calibration images are transformed into different projected top views, and different sub-regions of each calibration image correspond to different perspective transformation matrices in the matrix family.

[0009] Feature extraction is performed on the projected top view of each sub-region in each calibration image. The first homography matrix corresponding to the projected top view of two adjacent sub-regions is obtained based on the extracted features. Then, the projected top view of each sub-region is stitched together based on the first homography matrix to obtain the complete projected top view.

[0010] Feature extraction is performed on each complete projected top view. The second homography matrix corresponding to two adjacent complete projected top views is obtained based on the extracted features. Then, each complete projected top view is stitched together based on the second homography matrix to obtain the panoramic view image of the main ship calibration stage.

[0011] Real-time operation phase:

[0012] Acquire N images of the operation captured by N cameras on the main ship;

[0013] Based on the N different families of perspective transformation matrices obtained in the pre-calibration stage, the N running images are transformed into different projected top views;

[0014] For each operational image, the projected top view of each sub-region is stitched together based on the first homography matrices obtained in the pre-calibration stage to obtain a complete projected top view; then, each complete projected top view is stitched together based on the second homography matrices obtained in the pre-calibration stage to obtain a panoramic view of the main ship during its operational phase.

[0015] Furthermore, the main ship has N cameras mounted on gimbal stabilizers at the front, rear, left, and right, capable of acquiring three-channel image information in real time and with a field of view >180°.

[0016] Furthermore, after acquiring N operational images captured by N cameras on the main ship, the method further includes: performing distortion correction processing on the N operational images based on the N operational images and the pre-acquired intrinsic parameter matrices and distortion coefficients of each camera.

[0017] Furthermore, the intrinsic parameter matrices and distortion coefficients of each camera are obtained in advance using the Zhang Zhengyou calibration method.

[0018] Furthermore, four luminous spheres are installed on the unmanned calibration hull, and the coordinates μ of the four luminous spheres in the world coordinate system are... l for:

[0019] μ l =[RT]μ o l∈{1,2,3,4}

[0020] Where R and T are the rotation and translation matrices of the luminous sphere relative to the positioning center in the world coordinate system, respectively, and μ o The coordinates of the positioning center in the world coordinate system.

[0021] Furthermore, the coordinates of the four luminous spheres in the world coordinate system are transformed by the perspective transformation matrix. The transformed target pixel coordinates should satisfy the constraint of minimizing reprojection error:

[0022]

[0023] Among them, G ij Z represents the j-th position in the i-th channel. * Let (x, y) represent positive integers, and (x, y) be the chosen coordinates of the luminous sphere in the world coordinate system at z = 0. ′ ,y ′ () represents the target pixel coordinates of the luminous sphere after perspective transformation. Perspective transformation matrix The inverse matrix.

[0024] Furthermore, for each patch area of ​​the view captured by the same camera:

[0025]

[0026]

[0027] in, Represents the position of G in each view. ij (i, j∈Z) * The image of region G after correction by intrinsic parameter matrix and distortion coefficients. ij This represents the j-th position in the i-th channel. The homography matrix represents the projection top view of the (j-1)th patch area of ​​the i-th channel to the projection top view of the j-th patch area of ​​the i-th channel. The homography matrix represents the projection top view of the j-th patch area of ​​the (i-1)-th channel to the projection top view of the j-th patch area of ​​the i-th channel.

[0028] Furthermore, when N=4, the panoramic image The calculation method is as follows:

[0029]

[0030] Among them, I boat The image is of the main ship's own area. This is a complete projected top view resulting from the projection and stitching of the four views. These are the homography matrices from left view to front view, front view to right view, right view to rear view, and rear view to left view, respectively.

[0031] In a second aspect, the present invention provides a ship panoramic surround view image acquisition system, comprising: a computer-readable storage medium and a processor;

[0032] The computer-readable storage medium is used to store executable instructions;

[0033] The processor is used to read executable instructions stored in the computer-readable storage medium and execute the ship panoramic surround view image acquisition method as described in the first aspect.

[0034] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for acquiring panoramic images of ships as described in the first aspect.

[0035] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0036] This invention selects appropriate camera installation positions based on the ship's shape and size. Four fisheye cameras, mounted on a gimbal stabilizer at the front, rear, left, and right, are calibrated to obtain relatively distortion-free images. A family of perspective transformation matrices is then calculated based on the coordinates of luminous markers on a calibration ship surrounding the main ship in the world coordinate system and their pixel coordinates in the image coordinate system. Images captured by the four cameras at different positions are processed through four different families of transformation matrices to obtain different projected top-down views. Different regions of each image correspond to different transformation matrices within the matrix family. Feature detection is performed on each local transformation result of each image, and a matching algorithm stitches the discrete local transformation results into a complete image. Finally, the four images are transformed and combined to form a bird's-eye view from a high-angle perspective. An algorithm to reduce the impact of shaking outputs a stable, panoramic bird's-eye view. This method is simple, effective, logically clear, and easy to implement. It provides new ideas for ship assisted navigation, safe ship navigation, and ship berthing, solving problems such as blind spots during ship navigation, improving the stability and safety of ships in complex operating environments, and contributing to the development of ship applications. Attached Figure Description

[0037] Figure 1 A conceptual diagram of a ship panoramic surround view image acquisition device provided in an embodiment of the present invention;

[0038] Figure 2 A diagram showing the ship's operational status and spatial distribution (N=4) provided for embodiments of the present invention;

[0039] Figure 3 This is a flowchart illustrating a method for acquiring panoramic images of a ship, as provided in an embodiment of the present invention.

[0040] Figure 4 A schematic diagram of the location space and pixel distribution of the Patch area provided in an embodiment of the present invention (N=4);

[0041] Figure 5 This is a structural diagram of an unmanned calibration vessel provided in an embodiment of the present invention;

[0042] Figure 6 This is a splicing mapping sequence diagram provided in an embodiment of the present invention;

[0043] Figure 7 This is a schematic diagram of the field of view and overlapping area of ​​each view provided in the embodiments of the present invention. Detailed Implementation

[0044] To make the objectives, system composition, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0045] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0046] When a ship needs to use panoramic views, the algorithm in this invention must be applied before the ship officially sets sail to install a ship panoramic image acquisition device to obtain a real-time, stable bird's-eye view for use from the ship's perspective, such as... Figure 1 As shown, the device includes:

[0047] 1. N (N≥4) image sensing modules (S1) capable of acquiring three-channel (RGB) image information in real time and having a field of view (FOV) > 180°, and four GPS navigation modules (S2) capable of acquiring the current position coordinates in real time.

[0048] 2. Shipboard central processing units (U) deployed on the main ship and calibration ship for storage, computing, and communication.

[0049] 3. A surround-view algorithm (P) that can be implemented in the ship's central processing unit, comprising two parts: pre-calculated (P1) and real-time calculated (P2).

[0050] 4. Unmanned surface vessel control module (E) including encirclement mission mode and trajectory tracking function.

[0051] The panoramic bird's-eye view algorithm (P) is as follows:

[0052] like Figure 3As shown, this algorithm is divided into a pre-calibration part and a real-time execution part, arranged sequentially. The pre-calibration algorithm includes the coordination, formation, and other cluster behaviors of the main ship and the calibration ships, assisting in completing the calibration part of the panoramic view algorithm. The real-time execution (P2) part acquires the panoramic view in real time. To make the algorithm easier to explain and understand, a detailed description is provided when N=4. It should be noted that when N≥5, the algorithm steps are the same as below, with only slight differences in the symbols and notation.

[0053] Part P1 includes the following steps:

[0054] P1.1: Calculate the intrinsic parameter matrix (K) and distortion coefficient matrix (D) of the four fisheye cameras, and use the intrinsic parameter matrix (K) and distortion coefficient (D) to remove the distortion of the four fisheye cameras.

[0055] P1.2: Obtain the family of perspective transformation matrices from each region of the distortion-free image to the projected top view. Based on the family of perspective transformation matrices and the original image, match and constrain each region's feature points, and then map and stitch them together to obtain their respective projected top views.

[0056] P1.3: By stitching together N (N=4) projected top views, a panoramic bird's-eye view is obtained.

[0057] P1.1 Calculate the intrinsic parameters and distortion coefficients to calibrate the camera:

[0058] First, it should be noted that this step is the same for N (N=4) cameras.

[0059] A fisheye-checkerboard calibration method was adopted, using the OpenCV image processing library. In one example, a 9×6 black and white checkerboard was created, and its Harris corners were detected. The intrinsic parameters and distortion coefficients of the fisheye camera were calculated based on the spatial 3D coordinates of the corners and the pixel coordinates in the image. In one example, the Harris corner detection function in the OpenCV image processing library, `cv2.findChessboardCorners()`, is used to detect the corner coordinates in a black and white checkerboard pattern. `cv2.cornerSubPix()` refines the corner coordinates to sub-pixel level. `cv2.fisheye.calibrate()` obtains the intrinsic parameter matrix K and distortion coefficients D. `cv2.fisheye.estimateNewCameraMatrixForUndistortRectify()` optimizes the intrinsic parameter matrix (3×3 dimension) and distortion coefficients (4×1 dimension). `cv2.fisheye.initUndistortRectifyMap()` obtains the mapping relationship between the distorted and undistorted images (stored in matrix form). `cv2.remap()` corrects the mapping of the original image, resulting in K. G D GAnd G∈{F,B,L,F}.

[0060] P1.2 Find the family of perspective transformation matrices This leads to the various projected top views:

[0061] This step involves a main vessel equipped with N fisheye cameras, and unmanned surface vessels (USVs) with sensing and positioning modules, strong light sources, and capable of trajectory planning and tracking. The number of USVs is p, corresponding to p portions of the images captured by the cameras. On open water, the main vessel moves at approximately uniform speed in a straight line along an unobstructed direction, while the calibration vessels perform periodic encirclement maneuvers around the main vessel. The routes traveled by each USV, the operational status of the main vessel and the calibration vessels (USVs), and their channel positions are as follows: Figure 2 As shown, the digital images captured by N cameras (front-view, rear-view, left-view, right-view, etc.) are divided into 'a' rows. Each row can be further divided into different patch regions by columns. In one example, as shown... Figure 4 Taking the front view as an example, the calibration vessel is equipped with a telescopic mast, on which a strong point light source is installed. Under conditions of high contrast with the environment, the color of the point light source is arbitrary. The abstract structure of the calibration vessel is as follows: Figure 5 As shown.

[0062] The USV operating in the k-th channel corresponds to the following... Figure 2 The k-th row of each camera shown is represented by a block where k ∈ {1, 2, 3…p}. The coordinates of the point light source mounted on the USV in space can be uniquely determined by the positioning center of the positioning module and its installation position and dimensions on the hull, such as… Figure 5 As shown, its coordinates in the world coordinate system are (α, β, γ). When the USV travels along the channel, it appears sequentially at various positions along the channel, such as... Figure 2 As shown. Within the projection range of each camera (F p B p L p R p Select a suitable location on the waterway, and denote the j-th location on the i-th waterway as G. ij For example, when N=4, there are G∈(F, B, L, R) (N=4), for example... Figure 4 The forward-looking camera shown has selected four channels within its field of view, and a different marker position F has been selected in each channel. ij Each F ij Each corresponds to one of the four small balls, and their positions can be represented as follows:

[0063] μ l =(x l ,y l ,z l ),l∈{1,2,3,4} (1)

[0064] The position of the ball is uniquely determined by the world coordinates obtained from the positioning module, using the rotation matrix R and translation matrix T of the ball relative to the ship's positioning center O, i.e.:

[0065] μ l =[RT]μ o (2)

[0066] μ o μ is the world coordinate of the positioning center. l The world coordinates of the four spheres.

[0067] like Figure 4 As shown, let the selected viewpoints within the G range be... ij The set of world coordinates of the l-th ball is For each selected location, the pixel coordinates at that location can be identified in the image. A deep neural network is used to select a backbone with strong feature extraction capabilities for the feature sphere, such as classic networks like Transformer, ResNet, VGG, and MobileNet. The target recognition decoder layer is used to realize the target recognition of the feature sphere, and the network segmentation decoder is used to segment the target sphere. The center of the target recognition result (bounding box) and the segmentation region result (ROI) is taken as the pixel coordinates of the feature sphere in the image.

[0068] Denote the perspective transformation matrix For each camera's field of view, G ij The regions are all based on their Determine the perspective transformation matrix corresponding to the location.

[0069] G within different viewpoints ij The pixel coordinate system in which the l-th luminous sphere appears in the image is set as follows: When the ship travels at a pre-set point position from different perspectives At the same time, identify the coordinates of the ball in the pixel coordinate system in each view. Pre-set the target pixel set for projection as

[0070] Therefore, the perspective transformation matrix of the sphere at the j-th position of the i-th channel in view GG∈{F,B,L,F} satisfies the following:

[0071]

[0072]

[0073]

[0074] To ensure that the target value best represents the top view of this section, the target pixel coordinates should satisfy the constraint of minimizing reprojection error:

[0075]

[0076] Where m 33 =1, therefore m can be solved by the correspondence of the four point pairs. 11 m 12 m 13 m 21 m 22 m 23 m 31 m 32 Z * Let (x, y) be positive integers, (x, y) be the chosen coordinates of the ball in the world coordinate system space at z = 0, and (u, v) be the pixel coordinates of the ball in the source image. (x ′ ,y ′ () are the pixel coordinates of the target image of the ball after perspective transformation and satisfying This is the inverse of the perspective transformation matrix for this region.

[0077] For each region F, B, L, R (N=4), its coordinates in the world coordinate system are... After obtaining the perspective transformation matrix at the position, for example... Figure 4 The image region shown undergoes a perspective projection transformation. Therefore, each G... ij G∈(F, B, L, R) (N=4), the corresponding image portion can be regarded as a Patch, and each Patch has:

[0078]

[0079] in Represents the position of G ij G∈(F、B、L、R)(N=4), the location region image is transformed by perspective projection to obtain a new image of the region.

[0080] After obtaining the perspective transformation matrix from the source image setpoint to the target image setpoint for each patch in each view, when the calibration ship enters... and j∈Z * ,G∈(F,B,L,R)(N=4)), j∈Z * G∈(F, B, L, R)(N=4)) and When the boundaries of the patches occupied by G∈(F, B, L, R) (N=4) are reached, multiple consecutive frames of images are acquired. Feature points are extracted from multiple consecutive frames of the patch and matched to complete the stitching. The specific operation is as follows:

[0081] For each view, feature points are extracted from each patch and concatenated: Feature points are detected for each patch (using methods such as SIFT, SURF, ORB, LoG, DoH, DoG, Harris, FAST, BRIEF, Canny, Sobel, Prewitt, etc.). The extracted feature points are evaluated using distance metrics such as BFMatch and Hamming, and incorrect matching pairs are filtered out using matching methods such as KNN and RANSAC. This can be represented as follows:

[0082]

[0083] A ′ =B ′ *H (9)

[0084] Where A and B are the found feature point pairs, A′ and B′ are the filtered point pairs, and H is the least squares solution that satisfies the above equation (9).

[0085] Solve arrive arrive Homography matrix required for splicing And save it. pass Mapped to Then, the images are stitched together. This will give us the complete projected top views of the four views: F (Front), B (Back), L (Left), and R (Right) (N=4).

[0086] P1.3: By stitching together the top-view projections of N (N=4), a panoramic bird's-eye view is obtained.

[0087] The four projected top views obtained from P1.2 are as follows: Figure 7 The diagram shows four trapezoidal regions F, B, L, and R (N=4). Since there is overlap in the field of view at the boundaries of these four regions, the calibrating vessel continues its periodic encirclement maneuver around the main vessel. When the calibrating vessel is traveling at the boundaries of these four regions, for example... Figure 7The image shown is located at the intersection of FL_Lap and LF_Lap, where cameras on the main ship capture multiple consecutive frames of images. Similarly, key feature points are extracted from each view (using methods such as SIFT, SURF, ORB, LoG, DoH, DoG, Harris, FAST, BRIEF, Canny, Sobel, and Prewitt). The extracted feature points are evaluated using distance metrics such as BFMatch and Hammi, and incorrect matching pairs are filtered out using matching methods such as KNN and RANSAC. The homography matrix between images is then calculated from the matching pairs. By sequentially piecing together L, F, R, and B using a homography matrix, the central area corresponds to the main ship itself. The central area is then filled with ship icons, ultimately resulting in a panoramic bird's-eye view of the entire area.

[0088] Part P2 includes the following steps:

[0089] Call K G D G Given G∈{F,B,L,F} (N=4), correct the image for each camera with a different viewpoint to obtain the corrected image I. G G∈{F,B,L,F}(N=4), for I G (G∈{F,B,L,F}(N=4)) have:

[0090]

[0091]

[0092] in, Represents the position of G in each view. ij (i, j∈Z) * The image of region G after correction by intrinsic parameter matrix and distortion coefficients. ij This represents the j-th position in the i-th channel. The homography matrix represents the projection top view of the (j-1)th patch area of ​​the i-th channel to the projection top view of the j-th patch area of ​​the i-th channel. This represents the homography matrix from the top-view projection of the j-th patch area of ​​the (i-1)-th channel to the top-view projection of the j-th patch area of ​​the ith channel. Therefore, as... Figure 6 As shown, the matching and stitching follows the row-first, column-later principle until the entire image is stitched together.

[0093] Obtain complete projected top views from each of the four cameras.

[0094] Then from:

[0095]

[0096] Where I boat The image is of the main ship's own area. This is a complete projected top view after the four views have been projected and stitched together. These are the homography matrices from left view to front view, front view to right view, right view to rear view, and rear view to left view, respectively. This is the final real-time panoramic view image. It's worth noting that the calculation parameters used in part P2 are pre-calculated and saved from P1.

[0097] Finally, a stable panoramic bird's-eye view is obtained through an image stabilization algorithm. Specifically: using an image matching algorithm, the transformation matrix between two consecutive frames is obtained. This matrix contains a large amount of horizontal displacement, vertical displacement, and rotation angle information. The corresponding values ​​are extracted from the matrix to obtain the motion trajectory curve. The curve is smoothed by using methods such as filtering, fitting, or optimization to eliminate numerous "glitch" points. In one example, corner detection and optical flow detection are used to identify image feature points. The RANSAC method is used to find matching point sets from noisy data, and then the transformation matrix between the two frames is calculated. Kalman filtering is used to filter and fit the camera's motion curve to achieve a shake-free and real-time panoramic bird's-eye view.

[0098] The above algorithm details the algorithmic steps when N=4. It should be noted that the algorithm when N>4 is the same as the above steps, but the view range becomes multiple views such as F, B, L, R, etc., no longer limited to the four perspectives when N=4.

[0099] Ship motion controller E:

[0100] This motion controller should be able to perform periodic encirclement movements on a moving target, with adjustable speed and relative phase angle of the encirclement movement, and should be able to handle (x1,y1), (x2,y2), and (x... … ,y … ), (x n ,y n This type of spatial sequence of positions allows unmanned surface vessels to be controlled to navigate accurately along a route, based on the dynamics model of the unmanned surface vessel.

[0101] The ship panoramic surround view image acquisition device is used to implement the above-mentioned ship panoramic surround view image acquisition method. Each unit in the ship panoramic surround view image acquisition device corresponds to the relevant steps of the ship panoramic surround view image acquisition method. The specific details are described above and will not be repeated here.

[0102] Accordingly, this application also relates to an electronic device including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above. Specifically, this electronic device can be a hierarchical processor installed on a ship, used to assist in guiding and maneuvering the ship to acquire surround view images according to the method described above when the ship needs to monitor the surrounding environment in real time and perform tasks such as berthing in port.

[0103] Accordingly, this application also relates to a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method. Specifically, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0104] The ship panoramic surround view image acquisition device of this invention provides a new perception solution for ships when environmental perception and assisted driving are required. It solves for the camera parameters and perspective transformation matrix of the spatial relationship of the ship panoramic surround view image acquisition device. Based on this, it acquires the coordinates of feature points in the world coordinate system by controlling an unmanned surface vessel (USV) to complete periodic encirclement movements. The world coordinates of the feature objects in the world coordinate system are used to establish the coordinates of calibration points in the pixel coordinate system. A deep neural network is used to identify and detect the feature objects in the world coordinate system. When the USV reaches a preset point in the pixel coordinate system, four target points within the patch area are selected, and the perspective transformation matrix of the current area is solved and saved. Then, by extracting feature points and performing matching and filtering, the homography matrix between regions is solved, and a projected top view from that viewpoint is obtained. Finally, feature points are extracted from the overlapping areas between views, and the homography matrix under the transformation relationship of the projected top view from each viewpoint is solved, thus completing the full-domain stitching. The above method is simple and effective, with clear logic and easy implementation. It provides a new approach to ship environmental detection and perception, solves navigation safety issues when ships are performing berthing tasks or accidentally entering complex waters with intertwined weeds, improves the stability of ship mission execution, and contributes to the development of ship applications.

[0105] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for acquiring panoramic surround view images of a ship, characterized in that, The method includes: Pre-calibration stage: Acquire the main ship Data collected by each camera A calibration image, of which, ≥4; The coordinates of the illuminated markers on the unmanned calibration vessel surrounding the main ship in the world coordinate system and the pixel coordinates in the image coordinate system are obtained. Different families of perspective transformation matrices; then according to Different families of perspective transformation matrices will Each calibration image is transformed into a different projected top view, and different sub-regions of each calibration image correspond to different perspective transformation matrices in the matrix family; Feature extraction is performed on the projected top view of each sub-region in each calibration image. The first homography matrix corresponding to the projected top view of two adjacent sub-regions is obtained based on the extracted features. Then, the projected top view of each sub-region is stitched together based on the first homography matrix to obtain the complete projected top view. Feature extraction is performed on each complete projected top view. The second homography matrix corresponding to two adjacent complete projected top views is obtained based on the extracted features. Then, each complete projected top view is stitched together based on the second homography matrix to obtain the panoramic view image of the main ship calibration stage. Real-time operation phase: Acquire the main ship Data collected by each camera A running image; Based on the data obtained during the pre-calibration phase Different families of perspective transformation matrices will The running image is transformed into different projected top views; For each operational image, the projected top view of each sub-region is stitched together based on the first homography matrices obtained in the pre-calibration stage to obtain a complete projected top view; then, each complete projected top view is stitched together based on the second homography matrices obtained in the pre-calibration stage to obtain a panoramic view of the main ship during its operational phase.

2. The method for acquiring panoramic surround view images of a ship according to claim 1, characterized in that, The main ship has a total of [number] sections at the front, rear, left, and right. A camera mounted on a gimbal stabilizer that can acquire three-channel image information in real time and has a field of view >180°.

3. The method for acquiring panoramic surround view images of a ship according to claim 2, characterized in that, Acquire the main ship Data collected by each camera After running the image, the method further includes: based on the The images taken during operation, along with the pre-acquired intrinsic parameter matrices and distortion coefficients of each camera, are used to evaluate the performance of the images. The image is subjected to distortion correction processing.

4. The method for acquiring panoramic surround view images of a ship according to claim 3, characterized in that, The intrinsic parameter matrices and distortion coefficients of each camera were obtained in advance using the Zhang Zhengyou calibration method.

5. The method for acquiring panoramic surround view images of a ship according to claim 1, characterized in that, The unmanned calibration vessel is equipped with four luminous spheres. The coordinates of the four luminous spheres in the world coordinate system are... for: , in, These are the rotation and translation matrices of the luminous sphere relative to the positioning center in the world coordinate system. The coordinates of the positioning center in the world coordinate system.

6. The method for acquiring panoramic surround view images of a ship according to claim 5, characterized in that, The coordinates of the four glowing spheres in the world coordinate system after perspective transformation matrix The transformed target pixel coordinates should satisfy the constraint of minimizing reprojection error: in, Indicates the first The first of the waterways One location, Represents positive integers. It is the space of the luminous sphere in the world coordinate system. Selected coordinates These are the target pixel coordinates of the luminous sphere after perspective transformation. Perspective transformation matrix The inverse matrix.

7. The method for acquiring panoramic surround view images of a ship according to claim 1, characterized in that, The various sub-regions of the view captured by the same camera are: in, Represents the position of each view. The image of the region after correction by intrinsic parameter matrix and distortion coefficients. Indicates the first The first of the waterways One location, , Representing the The first waterway The sub-region projection top view to the first The first waterway Homography matrix of the top view of each sub-region projection. Representing the The first waterway The sub-region projection top view to the first The first waterway Homography matrix of the top view of each sub-region projection.

8. The method for acquiring panoramic surround view images of a ship according to claim 1, characterized in that, At that time, the panoramic image The calculation method is as follows: in, The image is of the main ship's own area. This is a complete projected top view resulting from the projection and stitching of the four views. These are the homography matrices from left view to front view, front view to right view, right view to rear view, and rear view to left view, respectively.

9. A system for acquiring panoramic surround-view images of ships, characterized in that, include: Computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is used to read executable instructions stored in the computer-readable storage medium and execute the ship panoramic surround view image acquisition method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method for acquiring panoramic images of ships as described in any one of claims 1-8.

Citation Information

Patent Citations

  • Method and system for obtaining panoramic image of ship based on machine vision

    CN113191974A

  • Image processing method and device, equipment and storage medium

    CN114820396A