Ship surround view system, method and device

Abstract

The invention relates to a ship surround view system, method and device. The method comprises the following steps: receiving an environment image and three-dimensional point cloud data; aligning the environment image with the three-dimensional point cloud data; distortion correction is carried out on the environment image, and the environment image after distortion correction is converted into a top view by using a homography matrix; mapping the depth information of the radar to an image coordinate system of the environment image to generate a depth map; using weighted fusion to eliminate seams in the overlapped region, and combining the top view and the depth map to obtain a panoramic top view; the panoramic top view is displayed on a visual parameter adjustment interface, the panoramic top view displayed on the visual parameter adjustment interface comprises equidistant lines around the ship, and the method has the advantages that by providing the 360-degree real-time panoramic aerial view with distance information around the ship, a clear visual field is provided for sailors to berth and leave the ship; and sailors are helped to accurately master the position and the motion state of the ship.

Description

Technical Field

[0001] The present invention relates to the field of ship technology applications, and in particular to a ship surround view system, method and device. Background Art

[0002] Modern shipbuilding technology has enhanced the intelligence of modern ships by adding visual technology devices. The images provided by these devices can guide ships in navigating ports and / or route environments. However, the following problems often arise during the operation of visual technology devices:

[0003] 1. Blind spot problem in image stitching. Since cameras with an angle of >180 degrees will lose edge information during dedistortion, the ship is not a regular shape when stitching images around it, so there will be a certain blind spot.

[0004] 2. It is difficult for a ship to remain still in the water, and the external parameters between cameras have a significant impact on the stitching results. Any calibration error may lead to unsatisfactory final results and even introduce new distortion.

[0005] 3. Complex geometric transformations are involved in image processing, projection, transformation, splicing, fusion, etc., which will lead to increased latency.

[0006] 4. Due to the large variety and complexity of ship types, the same image processing solution is difficult to adapt to multiple types of ships.

[0007] Currently, there is no effective solution to the problem that due to defects in the process of acquiring and processing images around the ship in related technologies, the final images cannot meet the needs of ship assisted driving. Summary of the Invention

[0008] The purpose of the present invention is to provide a ship surround view system, method and device to address the deficiencies in the prior art, so as to solve the technical problem in the related art that the final image cannot meet the requirements of ship assisted driving due to defects in the process of acquiring and processing images around the ship.

[0009] To achieve the above object, the technical solution adopted by the present invention is:

[0010] The present invention provides a ship surround view system, comprising: an image acquisition array, a modular retractable bracket, a radar, a foam calibration plate, and an industrial control computer, wherein the image acquisition array is installed around the hull of the ship according to a polygonal axisymmetric layout, covers the entire field of view, is connected to the industrial control computer, and is used to acquire an environmental image around the ship and a calibration image of the foam calibration plate, and transmit the environmental image to the industrial control computer; the modular retractable bracket is connected to the image acquisition array, and is used to adjust the number, installation position, and installation angle of the image acquisition devices in the image acquisition array according to the size of the ship; the radar is connected to the industrial control computer, and is used to acquire three-dimensional point cloud data around the ship, and transmit the three-dimensional point cloud data to the industrial control computer. The data is transmitted to the industrial computer; one end of the foam calibration plate is connected to the ship and distributed around the ship, and is used to calibrate the foam calibration plate with the image acquisition array and the radar to obtain a calibration image; the industrial computer is used to receive the environmental image and the three-dimensional point cloud data for alignment; the environmental image is corrected for distortion, and the distortion-corrected environmental image is converted into a bird's-eye view by applying the homography matrix, the depth information of the radar is mapped to the image coordinate system of the environmental image to generate a depth map, and the overlapping area is eliminated by weighted fusion to obtain a panoramic bird's-eye view; the panoramic bird's-eye view is displayed on the visual parameter adjustment interface, wherein the panoramic bird's-eye view displayed on the visual parameter adjustment interface includes the equidistant lines around the ship.

[0011] Optionally, the image acquisition array includes: at least four wide-angle fisheye cameras, and the installation positions and installation angles of the at least four wide-angle fisheye cameras are determined according to the size of the ship; wherein, the at least four wide-angle fisheye cameras correspond to the foam calibration plate respectively; the at least four wide-angle fisheye cameras are also used to obtain the calibration image of the foam calibration plate during the calibration process, so that the industrial computer calculates the internal and external parameters of the at least four wide-angle fisheye cameras based on the calibration image; the at least four wide-angle fisheye cameras are also used to collect wide-angle images around the ship in real time during the image stitching process, determine the wide-angle images as environmental images, and transmit the environmental images to the industrial computer; the at least four wide-angle fisheye cameras are also used to collect multi-angle fisheye images around the ship in real time during the manual parameter adjustment for display optimization, and transmit the multi-angle fisheye images to the industrial computer; the at least four wide-angle fisheye cameras are also used to collect images around the ship in real time during system maintenance and optimization, detect anomalies, and send an alarm signal to the industrial computer if an anomaly occurs.

[0012] Optionally, the modular telescopic bracket includes: a multi-stage telescopic rod, a universal ball head joint, a positioning hole, a cable management system and a clamping interface, wherein the multi-stage telescopic rod includes: at least two-stage telescopic rods and a locking mechanism corresponding to each telescopic rod in the at least two-stage telescopic rods, and the locking mechanism is used to adjust the length of the corresponding telescopic rod; the universal ball head joint is located at the top of the multi-stage telescopic rod, and the universal ball head joint includes: a dual-axis universal ball head and a scale ruler, wherein the dual-axis universal ball head is used to adjust the rotation angle and pitch angle of the wide-angle fisheye camera, and the scale ruler is used to further adjust the pitch angle of the wide-angle fisheye camera; the positioning hole is located at a specific position of the multi-stage telescopic rod, and is used to align multiple modular telescopic brackets to the same plane according to the positioning hole through an infrared transmitter; the cable management system is located inside the multi-stage telescopic rod, and is used to prevent the cables of the multi-stage telescopic rod from being entangled or pulled during the extension and retraction process through the built-in spiral cable groove and the cable tie fixing point; the clamping interface is located at the horizontal top of the universal ball head joint, and is used to disassemble and assemble the wide-angle fisheye camera.

[0013] Optionally, the radar is also used to scan the foam calibration plate to obtain three-dimensional point cloud data during the calibration process, so that the industrial computer can optimize the external parameters of the wide-angle fisheye camera based on the fusion of the three-dimensional point cloud data of the foam calibration plate and the environmental image; the radar is also used to obtain three-dimensional point cloud data around the ship during the image stitching process, and transmit the three-dimensional point cloud data around the ship to the industrial computer, so that the industrial computer can map the depth information of the radar to the image coordinate system of the environmental image based on the three-dimensional point cloud data to generate a depth map, and stitch the depth map and the top view into a panoramic image; the radar is also used to obtain the three-dimensional point cloud data of the ship during the manual parameter adjustment for display optimization, so that the industrial computer can generate equidistant lines based on the three-dimensional point cloud data and the image feature points collected by the image acquisition array; the radar is also used to continuously scan the three-dimensional point cloud data and coverage range around the ship during system maintenance and optimization. If an abnormality is triggered, the industrial computer is triggered to recalibrate.

[0014] Optionally, the industrial computer is further used to calculate the internal parameters of at least four wide-angle fisheye cameras based on the calibration images transmitted by at least four wide-angle fisheye cameras during the calibration process; generate a distortion correction mapping table, wherein the distortion correction mapping table is used to verify the linear error; in the case of joint external parameter calibration of at least four wide-angle fisheye cameras and radar, extract the pixel coordinates of the calibration image and the three-dimensional point cloud data obtained by the radar scanning foam calibration plate, calculate the external parameters of at least four wide-angle fisheye cameras through a specific algorithm; and calculate the homography matrix based on the homography transformation; the industrial computer is also used to perform distortion correction on the environmental image according to the internal parameters during the image stitching process, and calculate the external parameters according to the homography matrix. The environmental image is converted into a bird's-eye view according to the homography matrix, and the depth information of the radar is mapped to the image coordinate system of the environmental image to generate a depth map; the depth map and the bird's-eye view are aligned, and weighted fusion is used to eliminate seams through overlapping areas. A panoramic bird's-eye view is generated by combining the depth map and the bird's-eye view, and equidistant lines are superimposed on the panoramic bird's-eye view; the industrial computer is also used to generate equidistant lines based on multi-angle fisheye images and the ship's three-dimensional point cloud data during manual parameter adjustment for display optimization; the industrial computer is also used to perform dynamic optimization based on alarm signals and the three-dimensional point cloud data and coverage around the ship during system maintenance and optimization, and to initiate automatic calibration or manual intervention.

[0015] Furthermore, optionally, the industrial computer is also used to store calibration parameters, parameter adjustment history and / or system alarm logs; and is connected to the ship navigation system to output panoramic bird's-eye view and equidistant lines.

[0016] The present invention provides a modular retractable bracket, which is applied to a ship surround view system, comprising: a multi-stage telescopic rod, a universal ball joint, a positioning hole, a cable management system and a clamping interface, wherein the multi-stage telescopic rod comprises: at least two-stage telescopic rods and a locking mechanism corresponding to each of the at least two-stage telescopic rods, and the locking mechanism is used to adjust the length of the telescopic rod of the corresponding stage; the universal ball joint is located at the top of the multi-stage telescopic rod, and the universal ball joint comprises: a dual-axis universal ball head and a scale ruler, wherein the dual-circumference universal ball head is used to adjust the rotation angle and pitch angle of a wide-angle fisheye camera, and the scale ruler is used to further adjust the pitch angle of the wide-angle fisheye camera; the positioning hole is located at a specific position of the multi-stage telescopic rod, and is used to align multiple modular retractable brackets on the same plane according to the positioning hole through an infrared transmitter; the cable management system is located inside the multi-stage telescopic rod, and is used to prevent cables from being entangled or pulled during the extension and retraction of the multi-stage telescopic rod through a built-in spiral cable groove and a cable tie fixing point; the clamping interface is located at the horizontal top of the universal ball joint, and is used to disassemble and assemble the wide-angle fisheye camera.

[0017] The present invention provides an image processing method for ship surround view, which is applied to a ship surround view system, comprising: receiving an environmental image and three-dimensional point cloud data; aligning the environmental image and the three-dimensional point cloud data; performing distortion correction on the environmental image, and converting the distortion-corrected environmental image into a bird's-eye view by applying a homography matrix; mapping depth information of a radar to an image coordinate system of the environmental image to generate a depth map; eliminating seams by using weighted fusion in overlapping areas, and combining the bird's-eye view and the depth map to obtain a panoramic bird's-eye view; and displaying the panoramic bird's-eye view on a visual parameter adjustment interface, wherein the panoramic bird's-eye view displayed on the visual parameter adjustment interface includes equidistant lines around the ship.

[0018] Optionally, the method also includes: running a specific tool of ROS to calculate the internal parameters of at least four wide-angle fisheye cameras based on the calibration image, and generate a distortion correction mapping table; calculating the external parameters of at least four wide-angle fisheye cameras through a specific algorithm based on the calibration image and the three-dimensional point cloud data obtained by radar scanning the foam calibration plate; calculating the homography matrix based on the homography transformation based on the calibration image and the three-dimensional point cloud data obtained by radar scanning the foam calibration plate.

[0019] Furthermore, optionally, performing distortion correction on the environmental image and applying the homography matrix to convert the distortion-corrected environmental image into a top view also includes: performing distortion correction on the environmental image based on internal parameters to obtain the distortion-corrected environmental image; and converting the distortion-corrected environmental image into a top view based on the homography matrix.

[0020] Optionally, weighted fusion is used to eliminate seams in overlapping areas, and the bird's-eye view and depth map are combined to obtain a panoramic bird's-eye view, including: optimizing stitching weights in overlapping areas in combination with depth information; extracting feature points of adjacent images of the bird's-eye view and depth map, eliminating stitching seams and chromatic aberration by screening matching pairs, and using weighted fusion in overlapping areas to introduce wide-angle camera data in far-end blind areas or retain undistorted edge areas to generate a panoramic bird's-eye view, and superimposing equidistant lines on the panoramic bird's-eye view.

[0021] The present invention adopts the above technical solution, by receiving environmental images and three-dimensional point cloud data; aligning the environmental images and the three-dimensional point cloud data; performing distortion correction on the environmental images, and applying the homography matrix to convert the distortion-corrected environmental images into a bird's-eye view; mapping the depth information of the radar to the image coordinate system of the environmental image to generate a depth map; using weighted fusion to eliminate seams in overlapping areas, combining the bird's-eye view and the depth map to obtain a panoramic bird's-eye view; displaying the panoramic bird's-eye view on a visual parameter adjustment interface, wherein the panoramic bird's-eye view displayed on the visual parameter adjustment interface includes equidistant lines around the ship. Compared with the existing technology, the present invention has the following technical effects: by providing a real-time panoramic bird's-eye view of 360 degrees around the ship with distance information, a clear field of view is provided for the crew when berthing and leaving the berth, helping the crew to accurately grasp the position and movement status of the ship. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a schematic diagram of a ship surround view system according to a first embodiment of the present invention;

[0023] Figure 2 Schematic diagram of the positional relationship between a foam calibration plate, a wide-angle fisheye camera, and a ship in a ship surround view system according to Embodiment 1 of the present invention;

[0024] Figure 3 is a schematic diagram of a modular retractable bracket in a ship surround view system according to a first embodiment of the present invention;

[0025] Figure 4 is a schematic diagram of image processing in a ship surround view system according to Embodiment 1 of the present invention;

[0026] Figure 5 FIG. 1 is a flow chart of an image processing method for ship surround view according to a third embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely used to explain this application and are not intended to limit this application. Based on the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without making any creative efforts are within the scope of protection of this application.

[0028] Obviously, the drawings described below are merely examples or embodiments of the present application. Those skilled in the art can, without inventive effort, apply the present application to other similar scenarios based on these drawings. Furthermore, it is also understood that, although the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, changes in design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as an insufficiency of the content disclosed in this application.

[0029] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0030] Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by a person of ordinary skill in the technical field to which this application belongs. The words "a", "an", "a", "the" and the like used in this application do not indicate a limit on quantity and may indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof used in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or units (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units that are inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The words "multiple" / "several" used in this application refer to two or more. "And / or" describes the association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, or B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0031] Example 1

[0032] An exemplary embodiment of the present invention is as follows Figure 1 As shown, Figure 1 Schematic diagram of a ship surround view system according to embodiment 1 of the present invention. The ship surround view system provided by the embodiment of the present application includes:

[0033] The image acquisition array 10, the modular retractable bracket 12, the radar 14, the foam calibration plate 16 and the industrial control computer 18, wherein the image acquisition array 10 is installed around the hull of the ship according to a polygonal axisymmetric layout, covering the full field of view, and is connected to the industrial control computer 18 to collect environmental images around the ship and calibration images of the foam calibration plate 16, and transmit the environmental images to the industrial control computer 18; the modular retractable bracket 12 is connected to the image acquisition array 10 to adjust the number, installation position and installation angle of the image acquisition devices in the image acquisition array 10 according to the size of the ship; the radar 14 is connected to the industrial control computer 18 to obtain three-dimensional point cloud data around the ship, and transmit the three-dimensional point cloud data to the industrial control computer 18. The image is input to the industrial computer 18; one end of the foam calibration plate 16 is connected to the ship and is distributed around the ship, and is used for the image acquisition array 10 and the radar 14 to calibrate the foam calibration plate 16 to obtain a calibration image; the industrial computer 18 is used to receive the environmental image and the three-dimensional point cloud data for alignment; the environmental image is corrected for distortion, and the distortion-corrected environmental image is converted into a top view by applying the homography matrix, the depth information of the radar 14 is mapped to the image coordinate system of the environmental image to generate a depth map, and the overlapping area is eliminated by weighted fusion to obtain a panoramic top view; the panoramic top view is displayed on the visual parameter adjustment interface, wherein the panoramic top view displayed on the visual parameter adjustment interface includes the equidistant lines around the ship.

[0034] The hardware deployment and installation phase of a ship surround view system provided in the embodiment of the present application is as follows:

[0035] In the embodiment of the present application, the image acquisition array 10 is taken as an array composed of multiple wide-angle fisheye cameras, and the radar 14 is described as a laser radar.

[0036] In the layout of multiple wide-angle fisheye cameras and lidars: determine the installation positions of at least 4 fisheye cameras and lidars according to the size and height of the ship, follow the principle of axial symmetry (the bow and stern are located on the central axis, and the cameras on both sides are symmetrically distributed), and ensure that the fields of view of adjacent cameras overlap by ≥30%. In the embodiment of the present application, at least 4 wide-angle fisheye cameras (field of view angle ≥180°) are set, and are installed around the hull in an axially symmetrical layout of a regular polygon, covering a 360° field of view; one or more 360° mechanical lidars (such as 16-line or 32-line) are used to obtain three-dimensional point cloud data around the ship.

[0037] Wherein, each wide-angle fisheye camera is mounted on a corresponding modular retractable bracket 12, and the installation parameters are pre-deployed according to the 3D modeling;

[0038] In the deployment of the foam calibration board 16, a rectangular foam board (such as 8m×2.5m) is placed on the water surface, and an adjustable attitude pole is fixed on the edge to ensure that the foam calibration board 16 is located directly below or in the overlapping area of ​​the wide-angle fisheye camera for calibration.

[0039] The industrial computer 18 in the embodiment of the present application is equipped with a similar high-performance GPU and is responsible for real-time image processing, point cloud fusion and stitching operations.

[0040] In summary, if Figure 2 As shown, Figure 2 This is a schematic diagram of the relationship between the foam calibration plate, wide-angle fisheye camera and ship position in a ship surround view system according to the first embodiment of the present invention. Taking a 20-meter ship as an example, the camera angle of the wide-angle fisheye camera is greater than or equal to 180 degrees for explanation. The angle between the installation angle and the edge of the ship is 45 degrees downward. The wide-angle fisheye camera is powered by a PoE switch. The foam calibration plate 16 (specification: 8 meters × 2.5 meters) is fixed at two corner points of one of the long sides with a long wooden stick of moderate length (i.e., the adjustable posture rod in the embodiment of the present application) to facilitate adjustment of the foam calibration plate 16 when standing on the ship. The foam calibration plate 16 is placed on the water surface to float, and the foam calibration plate 16 is placed in a position parallel to the ship and directly below the wide-angle fisheye camera. At the same time, the wide-angle fisheye camera obtains the picture at this time (i.e., the calibration image in the embodiment of the present application).

[0041] Optionally, the modular retractable bracket 12 includes: a multi-stage telescopic rod 121, a universal ball joint 122, a positioning hole 123, a cable management system and a clamping interface 124, wherein the multi-stage telescopic rod 121 includes: at least two stages of telescopic rods and a locking mechanism corresponding to each stage of the at least two stages of telescopic rods, the locking mechanism is used to adjust the length of the corresponding stage of telescopic rod; the universal ball joint 122 is located at the top position of the multi-stage telescopic rod 121, the universal ball joint 122 includes: a dual-axis universal ball head and a scale ruler, wherein the dual-circle universal ball head is used to adjust the wide-angle fisheye camera The rotation angle and pitch angle, the scale is used to further adjust the pitch angle of the wide-angle fisheye camera; the positioning hole 123 is located at a specific position of the multi-stage telescopic rod 121, and is used to align multiple modular telescopic brackets 12 on the same plane according to the positioning hole 123 through the infrared transmitter; the cable management system is located inside the multi-stage telescopic rod 121, and is used to prevent the cables of the multi-stage telescopic rod 121 from being entangled or pulled during the extension and retraction process through the built-in spiral cable groove and cable tie fixing point; the clamping interface 124 is located at the horizontal top of the universal ball head joint 122, and is used to disassemble and assemble the wide-angle fisheye camera.

[0042] Specifically, such as Figure 3 As shown, Figure 3 This is a schematic diagram of a modular retractable bracket in a ship surround view system according to Example 1 of the present invention. The multi-stage telescopic rod 121 adopts a three-stage aluminum alloy telescopic rod (maximum extension length 2.5 meters). The locking mechanism of each stage supports independent adjustment to adapt to different floor heights (such as exhibition hall ceiling height, factory building trusses, etc.).

[0043] Universal ball joint 122: The top of the multi-stage telescopic rod 121 is integrated with a dual-axis universal ball head, which supports 360° horizontal rotation and 90° pitch adjustment, and can achieve precise positioning (angle error ≤ 0.5°) with the help of a scale ruler.

[0044] Through the cable management system ( Figure 3 (not shown), with built-in spiral cable troughs and tie-wrap fixing points to prevent cables from getting tangled or pulled during telescoping.

[0045] Clamping interface 124: Adopts Manfrotto standard quick release plate (501PL), compatible with mainstream camera models, and supports device disassembly and assembly within 30 seconds.

[0046] During the pre-deployment phase, use 3D modeling software (such as SketchUp) to simulate the scene and determine the initial installation height (e.g., 2.8 meters) and angle parameters for each wide-angle fisheye camera. Based on the simulation results, pre-configure the length of the modular retractable bracket 12 (e.g., 1.8 meters for camera A and 2.3 meters for camera B) and the universal ball head angle (e.g., pitch +5°).

[0047] During on-site installation and commissioning, the modular retractable bracket 12 was set up and locked, and the verticality was calibrated using a spirit level (deviation ≤ 1°). After installing the wide-angle fisheye camera, the universal ball joint was fine-tuned to ensure that the fields of view of adjacent wide-angle fisheye cameras overlapped by 30%-40% (to avoid excessive overlap and resulting in stitching distortion).

[0048] In the dynamic optimization mechanism, during operation, the brightness difference at the splicing seam is detected by image analysis software (such as OpenCV). If the difference is greater than 15%, the height (±5cm) or angle (±2°) of the wide-angle fisheye camera corresponding to the modular retractable bracket 12 is remotely adjusted to compensate.

[0049] The modular retractable bracket 12 in the embodiment of the present application is adjusted through environmental adaptability. In areas where there are structural obstructions or special observation requirements (such as curved walls and around pillars), the camera pitch angle (±15°) and horizontal offset angle (±30°) are dynamically adjusted through the retractable bracket to avoid blind spots in the field of view.

[0050] A ship surround view system provided in an embodiment of the present application adopts a modular retractable bracket, ensures spatial coverage integrity through a regular polygon layout, utilizes multi-stage telescopic rods and universal ball joints to achieve flexible installation, and combines 3D modeling pre-deployment with a dynamic optimization mechanism to significantly improve installation flexibility and system adaptability.

[0051] Optionally, the image acquisition array 10 includes: at least four wide-angle fisheye cameras, and the installation positions and installation angles of the at least four wide-angle fisheye cameras are determined according to the size of the ship; wherein, the at least four wide-angle fisheye cameras correspond to the foam calibration plate 16 respectively; the at least four wide-angle fisheye cameras are also used to obtain the calibration image of the foam calibration plate 16 during the calibration process, so that the industrial computer 18 calculates the internal and external parameters of the at least four wide-angle fisheye cameras based on the calibration image; the at least four wide-angle fisheye cameras are also used to collect wide-angle images around the ship in real time during the image stitching process, determine the wide-angle images as environmental images, and transmit the environmental images to the industrial computer 18; the at least four wide-angle fisheye cameras are also used to collect multi-angle fisheye images around the ship in real time during the manual parameter adjustment for display optimization, and transmit the multi-angle fisheye images to the industrial computer 18; the at least four wide-angle fisheye cameras are also used to collect images around the ship in real time during system maintenance and optimization, detect anomalies, and send an alarm signal to the industrial computer 18 if an anomaly occurs.

[0052] The image acquisition array 10 in the embodiment of the present application adheres to the principle of spatial coverage integrity, namely, employing a regular polygonal layout (e.g., a hexagon or octagon) to ensure that the cameras' fields of view have no overlapping blind spots. For example, six cameras are installed at 120° intervals around the center, with a single camera having a field of view of ≥60°, providing 360° panoramic coverage.

[0053] In the embodiment of the present application, at least four cameras are installed according to the size of the ship. The larger the ship, the more cameras are installed. The installation position is determined according to the imaging effect, and the installation follows the principle of axisymmetry. Since the existing research on ship perception is mainly concentrated on long-distance water areas, open waters have AIS, X-band radar, etc. to provide decision-making conditions for ships and help crews avoid risks in time. The embodiment of the present application mainly studies the perception of waters close to ships, targeting narrow sections, entering and leaving ports, passing locks, crossing bridges, berthing and leaving, and low-speed driving, so as to reduce the defects of manual lookout in transmitting information.

[0054] Optionally, the radar 14 is also used to scan the foam calibration plate 16 to obtain three-dimensional point cloud data during the calibration process, so that the industrial computer 18 can optimize the external parameters of the wide-angle fisheye camera based on the fusion of the three-dimensional point cloud data of the foam calibration plate 16 and the environmental image; the radar 14 is also used to obtain three-dimensional point cloud data around the ship during the image stitching process, and transmit the three-dimensional point cloud data around the ship to the industrial computer 18, so that the industrial computer 18 can map the depth information of the radar 14 to the image coordinate system of the environmental image based on the three-dimensional point cloud data, generate a depth map, and stitch a panoramic image based on the depth map and the top view; the radar 14 is also used to obtain the three-dimensional point cloud data of the ship during the manual parameter adjustment for display optimization, so that the industrial computer 18 can generate equidistant lines based on the three-dimensional point cloud data and the image feature points collected by the image acquisition array 10; the radar 14 is also used to continuously scan the three-dimensional point cloud data and coverage range around the ship during system maintenance and optimization. If an abnormality is triggered, the industrial computer 18 is triggered to recalibrate.

[0055] Optionally, the industrial computer 18 is also used to calculate the internal parameters of at least four wide-angle fisheye cameras based on the calibration images transmitted by the at least four wide-angle fisheye cameras during the calibration process; by generating a distortion correction mapping table, wherein the distortion correction mapping table is used to verify the linear error; in the case of the external parameter calibration of at least four wide-angle fisheye cameras and the radar 14, the pixel coordinates of the calibration image and the three-dimensional point cloud data obtained by the radar 14 scanning the foam calibration plate 16 are extracted, and the external parameters of the at least four wide-angle fisheye cameras are calculated by a specific algorithm; and based on the homography transformation, the homography matrix is ​​calculated; the industrial computer 18 is also used to perform distortion correction on the environmental image according to the internal parameters during the image stitching process , convert the environmental image into a bird's-eye view based on the homography matrix, map the depth information of the radar 14 to the image coordinate system of the environmental image, and generate a depth map; align the depth map with the bird's-eye view, eliminate seams through weighted fusion in the overlapping area, combine the depth map and the bird's-eye view to generate a panoramic bird's-eye view, and superimpose equidistant lines on the panoramic bird's-eye view; the industrial computer 18 is also used to generate equidistant lines based on multi-angle fisheye images and the three-dimensional point cloud data of the ship during manual parameter adjustment for display optimization; the industrial computer 18 is also used to perform dynamic optimization based on alarm signals and the three-dimensional point cloud data and coverage around the ship during system maintenance and optimization, and to initiate automatic calibration or manual intervention.

[0056] Furthermore, optionally, the industrial computer 18 is also used to store calibration parameters, parameter adjustment history and / or system alarm logs; and is connected to the ship navigation system to output a panoramic bird's-eye view and equidistant lines.

[0057] Specifically, in the embodiment of the present application, the industrial computer 18 includes the following in the hardware deployment and initialization phase: the first part, device connection and driver loading; and the second part, system self-test and parameter preloading; wherein,

[0058] Part 1, device connection and driver loading: connect the wide-angle fisheye camera array (i.e., at least four wide-angle fisheye cameras in the embodiment of the present application) through a ROS node (such as usb_cam), initialize parameters such as resolution and frame rate; load the lidar SDK, set the scanning mode (for example, scanning range and operating frequency); configure the PoE switch to power the wide-angle fisheye camera.

[0059] The second part is system self-test and parameter preloading: In sensor status detection, OpenCV's VideoCapture is used to verify that the camera image stream is normal, and the radar SDK interface is called to detect the integrity of the point cloud data; in parameter loading, the pre-calibrated intrinsic parameters (K, D), extrinsic parameters (R, T) and homography matrix (H) are loaded from local storage. If they are not available, the calibration phase is entered; in time synchronization: the wide-angle fisheye camera and lidar clocks are aligned through the PTP protocol.

[0060] In the embodiment of the present application, the industrial computer 18 is combined with at least four wide-angle fisheye cameras and the radar 14. The calibration phase includes three parts: the first part is the internal parameter calibration of the wide-angle fisheye camera; the second part is the external parameter calibration of the laser radar combined with the wide-angle fisheye camera; and the third part is the calibration of the homography matrix.

[0061] Part 1: Calibrate the internal parameters of the wide-angle fisheye camera. Run the ROS tool cameracalibrator.py, combined with the foam calibration plate 16, to capture multi-angle images (i.e., the calibration images in this embodiment). Calculate the distortion parameters (K, D), verify the linear error, and generate the distortion correction map (cv2.fisheye.initUndistortRectifyMap).

[0062] Specifically, the acquired environmental image is first subjected to distortion correction. The calibration tool provided by ROS (Robot Operating System) is used. Before running, a chessboard calibration board must be prepared. Run rosrun camera_calibrationcameracalibrator.py --size 11x8 --square 0.02image:= / usb_cam / image_raw to perform internal calibration on the wide-angle fisheye camera.

[0063] Among them, --size refers to the number of inner corner points of the calibration plate, --square refers to the side length of each small square, and image is the topic name published by the driver of the wide-angle fisheye camera.

[0064] After running, a window with the camera image will appear. The default value of camera_type is 0 (0: pinhole camera, 1: wide-angle fisheye camera). Set camera_type to 1. Then use the foam calibration plate 16 to slowly move in various directions. You will find that the progress bar on the right side of the window changes until the progress bar turns green.

[0065] X represents left and right movement, Y represents up and down, Size represents distance, and Skew represents tilt. When the progress bar is all green, the CALIBRATE button changes color. Click until the calibration information appears on the console to complete the calibration.

[0066] After testing the calibration results, align the foam calibration plate 16 with the wide-angle fisheye camera. The linear error will be displayed on the right side of the window. Generally, if the linear error is less than 0.1 or if acc is displayed, the calibration result is acceptable. In addition, click the SAVE button to obtain a compressed package in the / tmp directory. This package contains the calibration results and images, and the .yaml file is the calibration result.

[0067] The obtained calibration results are corrected for distortion through the cv2.fisheye.initUndistortRectifyMap(K,D,R,P,size,m1type) function. Since the wide-angle fisheye camera will lose a lot of images when performing distortion correction, the parameter P (new intrinsic parameter matrix) in the function is used to display the lost images as needed. The image frame size after dedistortion of the wide-angle fisheye camera is the same as the original image, but the visible range is reduced. The fx and fy parameters in the image camera parameters after dedistortion in the code are reduced to 1 / 4 of the original. The image frame size after distortion correction remains unchanged, and the visible range is increased.

[0068] The second part involves calibrating the lidar and wide-angle fisheye camera using external parameters. The system simultaneously acquires images and 3D point cloud data from the foam calibration plate 16 and extracts corner coordinates (image pixel coordinates and point cloud 3D coordinates). The direct_visual_lidar_calibration algorithm (i.e., the specific algorithm in this embodiment) is called to optimize the rotation matrix R and translation vector T to minimize the reprojection error.

[0069] Specifically, in the process of fusing the lidar and the wide-angle fisheye camera (i.e., external parameter calibration of the lidar combined with the wide-angle fisheye camera), since the input data is usually the pixel coordinates extracted from the image, the foam calibration plate 16 can be drawn in a checkerboard manner, adding multiple groups of points, 8 unknown quantities, and fitting by more than 8 constraint equations.

[0070] Solving Ax = b becomes an overdetermined equation fitting problem using argmin||Ax - b||. Because more data provides more constraints, even if one or two data points have large errors, they can be corrected using other points.

[0071] Since solving SVD for large matrices is time-consuming, OpenCV calculates H by selecting 4 points each time and using ransac to eliminate outliers, and then selects 4 points from the remaining inliers to continue calculating H. Through iteration, more data is referenced for fitting.

[0072] The lidar and wide-angle fisheye camera are calibrated using the direct_visual_lidar_calibration algorithm to obtain the position of each camera in the lidar coordinate system based on the lidar.

[0073] Method 1: The laser radar has distance and azimuth information, so it scans the size and position of the foam calibration plate 16. Through the difference in reflection intensity, the algorithm is used to obtain the data of the foam calibration plate 16 to optimize the error caused by manual extraction. Since there is only one laser radar, the external parameters between the cameras are optimized.

[0074] Method 2: Convert the LiDAR point cloud (x, y, z) to the camera coordinate system by rotating the coordinate system. camera =R·P lidar +T, select the top-down perspective (z=0 plane), extract the x and y coordinates of the 3D point cloud data, adjust the focal length and principal point of the wide-angle fisheye camera internal parameter K according to the distribution of the 3D point cloud data, and generate a top-down projection.

[0075] The external parameter calibration of the LiDAR combined with the wide-angle fisheye camera optimizes the external parameters between the cameras. In close-range scenes, the image information has low accuracy in spatial perception, depth, and distance. Therefore, the LiDAR and camera are added for fusion to supplement the camera's three-dimensional information and obtain better external parameters. The use of 360-degree LiDAR can better obtain the camera's external parameters and the fusion of the stitching.

[0076] The third part is the calibration of the homography matrix. Based on the homography transformation (Homography), the physical size and image coordinates of the foam calibration board 16 are used to solve the homography matrix H through getPerspectiveTransform.

[0077] Specifically, the four corner points of the foam calibration plate 16 are obtained from the distortion-corrected image, and the coordinates of the four corner points of the foam calibration plate 16 in the image to be imaged are obtained by measurement. The top view around the ship is regarded as 1 cm × 1 cm as a pixel point, and the target imaging size is ((target display distance around the ship × 2 + ship length) × (target display distance around the ship × 2 + ship width)). The greater the distance around the target, the lower the clarity in the distance. The size of the foam calibration plate 16 in the target canvas is converted into pixel values ​​mapped to the target canvas with the ship as the center point. The 8 parameters of the homography matrix require at least 4 groups of points to solve. Use opencv getPerspectiveTransform: 4 groups of points provide 8 constraint equations, solve 8 unknown quantities, and obtain the homography matrix (3x3) for converting the image into a top view.

[0078] In the embodiment of the present application, the industrial computer 18 performs the following steps in the real-time processing and splicing phase:

[0079] Step 1: Data synchronization and input: receiving the environmental image from the wide-angle fisheye camera and the 3D point cloud data from the lidar, and ensuring temporal and spatial consistency through timestamp alignment;

[0080] Step 2, GPU accelerated pipeline, through distortion correction, uses cv2.remap function to correct the distortion of the environment image in real time; through bird's-eye view projection, the homography matrix H is applied to convert the environment image into a bird's-eye view; and through point cloud fusion, the lidar point cloud is converted into GPU Tensor format, mapped to the image coordinate system of the environment image, and a depth map is generated. The stitching weights (such as occlusion compensation) are optimized by combining the depth information in the overlapping area.

[0081] Step 3, panoramic stitching and fusion, extracts feature points of adjacent images through feature matching, and screens matching pairs through RANSAC; and uses linear gradient weights or Laplacian pyramid fusion in overlapping areas through weighted fusion to eliminate stitching seams and color differences; through blind spot processing, wide-angle fisheye camera data is introduced in the far blind area or the undistorted edge area is retained.

[0082] Step 4: Generate a 360° panoramic bird's-eye view in real time, overlaying the equidistant lines generated by the lidar.

[0083] In the embodiment of the present application, the industrial computer 18 includes the following in the manual parameter adjustment and display optimization stage: the first part, the visual parameter adjustment interface; and the second part, the parameter persistence and verification; wherein,

[0084] The first part is a visual parameter adjustment interface that displays a bird's-eye view of each wide-angle fisheye camera and supports manual parameter adjustment. It provides real-time feedback on the adjustment effect and updates the stitched image through GPU acceleration.

[0085] The second part is parameter persistence and verification, which saves the optimized parameters to the configuration file and supports fast loading and rollback; it also verifies the adjusted geometric consistency (such as equidistant line accuracy verification) in combination with the lidar point cloud.

[0086] In the embodiment of the present application, the industrial computer 18 includes the following in the system maintenance and dynamic optimization phase: the first part, real-time monitoring and exception handling; the second part, dynamic parameter optimization; and the third part, maintenance log and early warning; wherein,

[0087] The first part involves real-time monitoring and exception handling; detecting differences in brightness at the joints through image quality monitoring; and calculating the vibration amplitude of the bracket through point cloud analysis.

[0088] The second part is dynamic parameter optimization. Through adaptive exposure, the camera parameters are dynamically adjusted according to the ambient light (HDR / low-light mode). Through external parameter compensation, if the lidar detects the bracket offset, the external parameters (R, T) are automatically updated.

[0089] The third part is maintenance log and warning. It records sensor performance data (such as frame rate drop and point cloud noise ratio) and generates maintenance reports. It also issues graded alarms (automatic repair, manual confirmation, emergency shutdown) to prompt crew members to deal with hardware failures.

[0090] In the embodiment of the present application, when the splicing effect of the industrial computer 18 is poor due to physical reasons after running for a period of time, the parameters are manually adjusted through the visual interface, and the parameters are displayed on the screen, which is convenient for the user to manually adjust the parameters and display the 360-degree surround image; and by using the GPU to obtain sensor data, screen synchronization, image processing, projection, transformation, splicing, fusion and other operations, the real-time and smoothness of the image are guaranteed.

[0091] In the embodiment of the present application, the industrial computer 18 stores calibration parameters, parameter adjustment history, and alarm logs through log records in data storage and communication, and supports fault backtracking; by docking with external systems, it communicates with ship navigation systems (such as AIS, ECDIS), outputs panoramic images and equidistant line data, and assists decision-making.

[0092] The industrial computer 18 in this embodiment uses GPU acceleration to ensure real-time performance. Multi-sensor fusion, combining visual and radar data, improves geometric consistency. Finally, an adaptive algorithm dynamically optimizes parameters to adapt to complex sea conditions.

[0093] In summary, Figure 4 FIG. 1 is a schematic diagram of image processing in a ship surround view system according to the first embodiment of the present invention. Figure 4 As shown, the image processing performed by the ship surround view system in the embodiment of the present application is as follows:

[0094] The first step is to determine the wide-angle fisheye camera (i.e. Figure 4 The number, installation position and angle of the cameras) and radar are determined. The installation positions follow the principle of axial symmetry. The wide-angle fisheye camera installed at the bow and the wide-angle fisheye camera installed at the stern are both on the central axis of the ship, and there is an overlapping area between the two adjacent cameras. A rectangular foam calibration plate 16 is placed on the water surface, and an operable rod is installed on the edge of the foam calibration plate 16. The posture of the foam calibration plate 16 floating on the water surface is adjusted and placed directly below the wide-angle fisheye camera and in the overlapping area of ​​the two wide-angle fisheye cameras, and the data collected by the wide-angle fisheye camera is obtained at the same time. Then, the projection transformation matrix of each camera image that has been corrected for distortion is calculated to be converted to a top view.

[0095] In the second step, based on the pre-calibrated camera and radar parameters, the LiDAR system selects a 360-degree mechanical radar. One or more radars can be installed depending on actual needs. Using the LiDAR data as a reference, the camera positions in the radar coordinate system are calculated. Using the 3D radar data, the extrinsic parameters between the cameras are optimized. The projected images are then stitched and fused to achieve a smooth transition within the overlapping area.

[0096] The third step is to display the information required for manual parameter adjustment on the screen, allowing the user to manually adjust the up, down, left, right, and right translation of each image as well as the zoom in and out according to the imaging effect to optimize the final imaging effect.

[0097] In the fourth step, the camera image is acquired through the GPU, and the intrinsic parameters, extrinsic parameters, and stitching parameters obtained in the first, second, and third steps are processed into an icon parameter. The transformation of each pixel is obtained by looking up the table, and the top view after stitching and the equidistant lines around the ship are displayed on the screen.

[0098] The present invention adopts the above technical solution, by receiving environmental images and three-dimensional point cloud data; aligning the environmental images and the three-dimensional point cloud data; performing distortion correction on the environmental images, and applying the homography matrix to convert the distortion-corrected environmental images into a bird's-eye view; mapping the depth information of the radar to the image coordinate system of the environmental image to generate a depth map; using weighted fusion to eliminate seams in overlapping areas, combining the bird's-eye view and the depth map to obtain a panoramic bird's-eye view; displaying the panoramic bird's-eye view on a visual parameter adjustment interface, wherein the panoramic bird's-eye view displayed on the visual parameter adjustment interface includes equidistant lines around the ship. Compared with the existing technology, the present invention has the following technical effects: by providing a real-time panoramic bird's-eye view of 360 degrees around the ship with distance information, a clear field of view is provided for the crew when berthing and leaving the berth, helping the crew to accurately grasp the position and movement status of the ship.

[0099] Example 2

[0100] In an exemplary embodiment of the present invention, a modular retractable bracket provided in the embodiment of the present application is applied to the ship surround view system in embodiment 1, such as Figure 3 As shown, a modular retractable bracket provided in an embodiment of the present application includes:

[0101] A multi-stage telescopic rod, a universal ball head joint, a positioning hole, a cable management system and a clamping interface, wherein the multi-stage telescopic rod includes: at least two-stage telescopic rods and a locking mechanism corresponding to each of the at least two-stage telescopic rods, and the locking mechanism is used to adjust the length of the corresponding-stage telescopic rod; the universal ball head joint is located at the top of the multi-stage telescopic rod, and the universal ball head joint includes: a dual-axis universal ball head and a scale ruler, wherein the dual-circumference universal ball head is used to adjust the rotation angle and pitch angle of the wide-angle fisheye camera, and the scale ruler is used to further adjust the pitch angle of the wide-angle fisheye camera; the positioning hole is located at a specific position of the multi-stage telescopic rod, and is used to align multiple modular telescopic brackets on the same plane according to the positioning hole through an infrared transmitter; the cable management system is located inside the multi-stage telescopic rod, and is used to prevent the cables of the multi-stage telescopic rod from being entangled or pulled during the extension and retraction process through the built-in spiral cable groove and the cable tie fixing point; the clamping interface is located at the horizontal top of the universal ball head joint, and is used to disassemble and assemble the wide-angle fisheye camera.

[0102] Example 3

[0103] An exemplary embodiment of the present invention is as follows Figure 5 As shown, Figure 5 1 is a flow chart of a method for processing image of a ship surround view according to a third embodiment of the present invention, which is applied to the ship surround view system in the first embodiment. The method for processing image of a ship surround view provided in the embodiment of the present application includes:

[0104] Step S500, receiving an environment image and three-dimensional point cloud data;

[0105] Step S502, aligning the environment image and the three-dimensional point cloud data;

[0106] Optionally, an image processing method for ship surround view provided in an embodiment of the present application also includes: running a specific tool of ROS, calculating the internal parameters of at least four wide-angle fisheye cameras based on the calibration image, and generating a distortion correction mapping table; calculating the external parameters of at least four wide-angle fisheye cameras through a specific algorithm based on the calibration image and the three-dimensional point cloud data obtained by radar scanning the foam calibration plate; calculating the homography matrix based on the homography transformation based on the calibration image and the three-dimensional point cloud data obtained by radar scanning the foam calibration plate.

[0107] Step S504, performing distortion correction on the environment image, and applying a homography matrix to convert the distortion-corrected environment image into a top view;

[0108] Furthermore, optionally, performing distortion correction on the environmental image and applying the homography matrix to convert the distortion-corrected environmental image into a top view also includes: performing distortion correction on the environmental image based on internal parameters to obtain the distortion-corrected environmental image; and converting the distortion-corrected environmental image into a top view based on the homography matrix.

[0109] Step S506 , mapping the depth information of the radar to the image coordinate system of the environment image to generate a depth map;

[0110] Step S508, eliminating seams by weighted fusion in overlapping areas, combining the top view and the depth map to obtain a panoramic top view;

[0111] Optionally, weighted fusion is used to eliminate seams in overlapping areas, and the bird's-eye view and depth map are combined to obtain a panoramic bird's-eye view, including: optimizing stitching weights in overlapping areas in combination with depth information; extracting feature points of adjacent images of the bird's-eye view and depth map, eliminating stitching seams and chromatic aberration by screening matching pairs, and using weighted fusion in overlapping areas to introduce wide-angle camera data in far-end blind areas or retain undistorted edge areas to generate a panoramic bird's-eye view, and superimposing equidistant lines on the panoramic bird's-eye view.

[0112] Step S510: Display the panoramic bird's-eye view on the visualization parameter adjustment interface, wherein the panoramic bird's-eye view displayed on the visualization parameter adjustment interface includes equidistant lines around the ship.

[0113] The present invention adopts the above technical solution, by receiving environmental images and three-dimensional point cloud data; aligning the environmental images and the three-dimensional point cloud data; performing distortion correction on the environmental images, and applying the homography matrix to convert the distortion-corrected environmental images into a bird's-eye view; mapping the depth information of the radar to the image coordinate system of the environmental image to generate a depth map; using weighted fusion to eliminate seams in overlapping areas, combining the bird's-eye view and the depth map to obtain a panoramic bird's-eye view; displaying the panoramic bird's-eye view on a visual parameter adjustment interface, wherein the panoramic bird's-eye view displayed on the visual parameter adjustment interface includes equidistant lines around the ship. Compared with the existing technology, the present invention has the following technical effects: by providing a real-time panoramic bird's-eye view of 360 degrees around the ship with distance information, a clear field of view is provided for the crew when berthing and leaving the berth, helping the crew to accurately grasp the position and movement status of the ship.

[0114] The above description is only a preferred embodiment of the present invention and does not limit the implementation mode and protection scope of the present invention. For those skilled in the art, it should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the description and illustrations of the present invention should be included in the protection scope of the present invention.

Claims

1. A ship surround view system, characterized in that: include: Image acquisition array, modular retractable bracket, radar, foam calibration plate and industrial computer, among which, The image acquisition array is installed around the hull of the ship according to a polygonal axisymmetric layout, covers the full field of view, is connected to the industrial computer, and is used to collect environmental images around the ship and calibration images of the foam calibration plate, and transmit the environmental images to the industrial computer; The modular retractable bracket is connected to the image acquisition array and is used to adjust the number, installation position and installation angle of the image acquisition devices in the image acquisition array according to the size of the ship; The radar is connected to the industrial computer and is used to obtain three-dimensional point cloud data around the ship and transmit the three-dimensional point cloud data to the industrial computer; One end of the foam calibration plate is connected to the ship and is distributed around the ship, and is used for the image acquisition array and the radar to calibrate the foam calibration plate to obtain the calibration image; The industrial computer is used to receive the environmental image and the three-dimensional point cloud data for alignment; perform distortion correction on the environmental image, and apply a homography matrix to convert the distortion-corrected environmental image into a bird's-eye view, map the depth information of the radar to the image coordinate system of the environmental image to generate a depth map, and use weighted fusion to eliminate seams in overlapping areas to obtain a panoramic bird's-eye view; display the panoramic bird's-eye view on a visual parameter adjustment interface, wherein the panoramic bird's-eye view displayed on the visual parameter adjustment interface includes equidistant lines around the ship.

2. The ship surround view system according to claim 1, characterized in that: The image acquisition array includes: at least four wide-angle fisheye cameras, wherein the installation positions and the installation angles of the at least four wide-angle fisheye cameras are determined according to the size of the vessel; wherein the at least four wide-angle fisheye cameras correspond to the foam calibration plates respectively; The at least four wide-angle fisheye cameras are further used to obtain the calibration image of the foam calibration plate during the calibration process, so that the industrial computer calculates the internal parameters and external parameters of the at least four wide-angle fisheye cameras based on the calibration image; The at least four wide-angle fisheye cameras are further configured to collect wide-angle images of the surroundings of the ship in real time during the image stitching process, determine the wide-angle images as the environmental images, and transmit the environmental images to the industrial computer; The at least four wide-angle fisheye cameras are further used to collect multi-angle fisheye images around the ship in real time during the process of manual parameter adjustment for display optimization, and transmit the multi-angle fisheye images to the industrial computer; The at least four wide-angle fisheye cameras are also used to collect images around the ship in real time during system maintenance and optimization, detect anomalies, and send an alarm signal to the industrial computer if an anomaly occurs.

3. The ship surround view system according to claim 2, characterized in that: The modular retractable bracket includes: a multi-stage telescopic rod, a universal ball joint, a positioning hole, a cable management system and a clamping interface, wherein: The multi-stage telescopic rod comprises: at least two stages of telescopic rods and a locking mechanism corresponding to each stage of the at least two stages of telescopic rods, wherein the locking mechanism is used to adjust the length of the corresponding stage of telescopic rod; The universal ball joint is located at the top of the multi-stage telescopic rod, and the universal ball joint includes: a dual-axis universal ball head and a scale ruler, wherein the dual-axis universal ball head is used to adjust the rotation angle and pitch angle of the wide-angle fisheye camera, and the scale ruler is used to further adjust the pitch angle of the wide-angle fisheye camera; The positioning holes are located at specific positions of the multi-stage telescopic rod, and are used to align the plurality of modular telescopic brackets on the same plane according to the positioning holes through an infrared transmitter; The cable management system is located inside the multi-stage telescopic rod and is used to prevent the cables from being entangled or pulled during the extension and retraction of the multi-stage telescopic rod through the built-in spiral cable groove and the cable tie fixing point; The clamping interface is located at the horizontal top of the universal ball joint and is used for disassembling and assembling the wide-angle fisheye camera.

4. The ship surround view system according to claim 2, characterized in that: The radar is further configured to scan the foam calibration plate to obtain three-dimensional point cloud data during the calibration process, so that the industrial computer can optimize the external parameters of the wide-angle fisheye camera based on the three-dimensional point cloud data of the foam calibration plate and the environmental image; The radar is further configured to obtain the three-dimensional point cloud data around the ship during the image stitching process, and transmit the three-dimensional point cloud data around the ship to the industrial computer, so that the industrial computer maps the depth information of the radar to the image coordinate system of the environment image based on the three-dimensional point cloud data to generate a depth map, and stitch the depth map and the bird's-eye view into a panoramic image; The radar is further configured to obtain the three-dimensional point cloud data of the ship during the manual parameter adjustment for display optimization, so that the industrial computer generates equidistant lines based on the three-dimensional point cloud data and image feature points acquired by the image acquisition array; The radar is also used to continuously scan the three-dimensional point cloud data and coverage area around the ship during the system maintenance and optimization process, and if an abnormality is triggered, it triggers the industrial computer to recalibrate.

5. The ship surround view system according to claim 2, characterized in that: The industrial computer is further configured to, during the calibration process, calculate the intrinsic parameters of the at least four wide-angle fisheye cameras based on the calibration images transmitted by the at least four wide-angle fisheye cameras; generate a distortion correction mapping table, wherein the distortion correction mapping table is used to verify linear errors; in the case of joint extrinsic parameter calibration of the at least four wide-angle fisheye cameras and the radar, extract pixel coordinates of the calibration image and three-dimensional point cloud data obtained by the radar scanning the foam calibration plate, calculate the extrinsic parameters of the at least four wide-angle fisheye cameras by a specific algorithm; and calculate the homography matrix based on the homography transformation; The industrial computer is further configured to, during the image stitching process, perform distortion correction on the environment image based on the internal parameters, convert the environment image into a top view based on the homography matrix, map the depth information of the radar to the image coordinate system of the environment image, and generate a depth map; align the depth map with the top view, eliminate seams through weighted fusion of the overlapping areas, combine the depth map and the top view to generate the panoramic top view, and superimpose equidistant lines on the panoramic top view; The industrial computer is further configured to generate the equidistant lines based on the multi-angle fisheye image and the three-dimensional point cloud data of the ship during manual parameter adjustment for display optimization; The industrial computer is also used to perform dynamic optimization according to the alarm signal and the three-dimensional point cloud data and coverage around the ship during system maintenance and optimization, and to initiate automatic calibration or manual intervention.

6. The ship surround view system according to claim 5, characterized in that: The industrial computer is also used to store calibration parameters, parameter adjustment history and / or system alarm logs; and is connected to the ship navigation system to output the panoramic bird's-eye view and the equidistant lines.

7. A modular retractable bracket, characterized in that: Applied to ship surround view systems, including: Multi-stage telescopic rod, universal ball joint, positioning hole, cable management system and clamping interface, among which, The multi-stage telescopic rod comprises: at least two stages of telescopic rods and a locking mechanism corresponding to each stage of the at least two stages of telescopic rods, wherein the locking mechanism is used to adjust the length of the corresponding stage of telescopic rod; The universal ball joint is located at the top of the multi-stage telescopic rod, and the universal ball joint includes: a dual-axis universal ball head and a scale ruler, wherein the dual-axis universal ball head is used to adjust the rotation angle and pitch angle of the wide-angle fisheye camera, and the scale ruler is used to further adjust the pitch angle of the wide-angle fisheye camera; The positioning holes are located at specific positions of the multi-stage telescopic rod, and are used to align the plurality of modular telescopic brackets on the same plane according to the positioning holes through an infrared transmitter; The cable management system is located inside the multi-stage telescopic rod and is used to prevent the cables from being entangled or pulled during the extension and retraction of the multi-stage telescopic rod through the built-in spiral cable groove and the cable tie fixing point; The clamping interface is located at the horizontal top of the universal ball joint and is used for disassembling and assembling the wide-angle fisheye camera.

8. A method for processing images of a ship's surrounding view, characterized in that: Applied to ship surround view systems, including: Receive environmental images and 3D point cloud data; Aligning the environment image with the three-dimensional point cloud data; Performing distortion correction on the environment image, and converting the distortion-corrected environment image into a top view by applying a homography matrix; Mapping the depth information of the radar to the image coordinate system of the environment image to generate a depth map; Eliminating seams by weighted fusion in overlapping areas, and combining the top view and the depth map to obtain a panoramic top view; The panoramic bird's-eye view is displayed on a visualization parameter adjustment interface, wherein the panoramic bird's-eye view displayed on the visualization parameter adjustment interface includes equidistant lines around the ship.

9. The image processing method for ship surround view according to claim 8, characterized in that: The method further comprises: Running a specific ROS tool to calculate the internal parameters of the at least four wide-angle fisheye cameras based on the calibration image and generate a distortion correction mapping table; Calculating the external parameters of the at least four wide-angle fisheye cameras using a specific algorithm based on the calibration image and the three-dimensional point cloud data obtained by radar scanning the foam calibration plate; The homography matrix is ​​calculated based on the calibration image and the three-dimensional point cloud data obtained by radar scanning the foam calibration plate and based on the homography transformation.

10. The image processing method for ship surround view according to claim 9, characterized in that: The step of performing distortion correction on the environment image and converting the distortion-corrected environment image into a top view by applying a homography matrix further comprises: Performing distortion correction on the environmental image according to the internal reference to obtain the distortion-corrected environmental image; The distortion-corrected environment image is converted into the top view according to the homography matrix.

11. The image processing method for ship surround view according to claim 10, characterized in that: Eliminating seams by weighted fusion in overlapping areas and combining the top view and the depth map to obtain a panoramic top view includes: Optimizing the stitching weight in the overlapping area in combination with the depth information; Feature points of adjacent images of the top view and the depth map are extracted, and by screening matching pairs, weighted fusion is used for the overlapping areas to eliminate stitching seams and chromatic aberrations, wide-angle camera data is introduced into the far-end blind area or the undistorted edge area is retained to generate the panoramic top view, and equidistant lines are superimposed on the panoramic top view.

Images