Shipborne panoramic berthing auxiliary device and method for large ship

Abstract

The invention provides a shipborne panoramic berthing auxiliary device and method for a large ship, and belongs to the technical field of ship auxiliary navigation. In order to solve the problem that the shipborne panoramic parking auxiliary reliability of the large ship is low, the invention provides a shipborne panoramic parking auxiliary device for the large ship, which comprises an X-band radar sensor, a sonar sensor, a plurality of camera sensor groups and a data processing and panoramic view display platform, according to the invention, the data of the sea level and obstacles above and below the sea level are obtained through the X-band radar sensor and the sonar sensor respectively, the image data of the surrounding environment of the ship in different light scenes are obtained through the multiple camera sensor groups, and the data processing and panoramic view display platform is matched. And the shipborne panoramic parking auxiliary reliability of the large ship is improved.

Description

Technical Field

[0001] The present application relates to the technical field of ship navigation assistance, and in particular to a panoramic mooring assistance device and method for large ships. Background Art

[0002] When a ship is moored at a port or anchorage, it is usually necessary to avoid collisions with surrounding ships, buoys, or fixed facilities such as the shore and docks. Otherwise, it may cause damage to the hull, loss of cargo, and even fuel leakage, pollution of the marine environment, and casualties, resulting in major safety accidents and economic losses. In particular, large ships carry more people and cargo, more fuel and chemicals, and have stronger kinetic energy and impact force. If a collision occurs, it may cause more serious consequences, and may cause serious damage to the hull structure, larger areas of environmental pollution, and more serious economic losses. Therefore, ships, especially large ships, need to take effective anti-collision measures when moored to ensure the safety of the ship and the surrounding environment, avoid accidents, and protect the integrity of the ship, cargo, and marine environment.

[0003] Traditionally, the anti-collision methods for ships at anchor mainly include the following:

[0004] 1. Use rubber buffer strips, buffer airbags or other buffer devices to absorb the impact energy generated during the ship's mooring process and protect infrastructure such as docks. In extreme cases or high-speed collisions, the airbag may not provide sufficient buffering effect and the ship will still be seriously damaged; the damage to the airbag itself increases the maintenance and repair costs, and the status of the airbag needs to be regularly inspected and maintained.

[0005] Second, use tugboats to assist ships in berthing or unberthing. This method is costly, relies more on external resources, is difficult to operate in a small space or complex environment, and there is a risk of collision and friction between the ship and the tugboat.

[0006] 3. Through the cooperation of multiple crew members, port management agencies and pilots, the position of each party is reported by radio communication, and the control room performs statistical analysis to avoid collisions. This method may cause information transmission delays. The control room needs to process and analyze a large amount of information in real time to avoid collisions. The human factor has a greater impact and there are high human resource costs.

[0007] 4. Identify and avoid obstacles through ship radar systems and Automatic Identification Systems (AIS). It is highly dependent on equipment and signals. Equipment errors and transmission problems can lead to inaccurate data and inability to identify obstacles that are not equipped with AIS.

[0008] 5. Use multi-directional cameras to transmit images to the ship operator's display module in real time to assist the ship operator in avoiding obstacles when mooring. Single camera image data will be affected by external factors, and the camera image segmentation is not coherent, which is not convenient for ship operators to view. In addition, there may be more overlapping areas or blind spots; misoperation, misinterpretation or neglect of system alarms may lead to failure in identifying and avoiding obstacles.

[0009] 6. Use a 360-degree panoramic camera system in conjunction with radar and sonar systems to obtain more comprehensive hydrological and obstacle information in the anchorage area for collision avoidance. This method is currently used more on small ships, using cameras in the front, rear, left, and right directions to stitch panoramic images. If it is directly transplanted to the application scenarios of large ships, a large number of cameras are required, and there may be serious blurring or blind spots at the image stitching, making it impossible to visualize obstacles under the ship. In addition, the camera in the system is easily affected by external factors and displaced, and the camera position cannot be adjusted adaptively. In addition, the hydrological and obstacle information in large ship application scenarios is more complex, and more data processing and analysis capabilities may be required.

[0010] The shortcomings of the above methods put forward high reliability requirements for the ship-borne panoramic mooring assistance device for large ships. Summary of the invention

[0011] The purpose of the present application is to solve the problem of low reliability of panoramic parking assistance for large ships in the prior art. Therefore, the present application provides a panoramic parking assistance device and method for large ships, which respectively obtains sea level and obstacle data above and below the sea level through X-band radar sensors and sonar sensors, obtains image data of the surrounding environment of the ship under different light scenes through multiple groups of camera sensor groups, and cooperates with data processing and panoramic view display platform to improve the reliability of panoramic parking assistance for large ships.

[0012] The embodiment of the present application provides a large-scale ship-borne panoramic mooring auxiliary device, comprising:

[0013] X-band radar sensor, installed high on the bridge, to obtain data on the distance, direction and motion status of obstacles above the sea surface and the ship;

[0014] Sonar sensor, installed on the bow or bottom of the ship, and used to obtain the distance, direction and movement status data of obstacles below the sea surface and the ship;

[0015] Multiple camera sensor groups are installed at intervals at the bow, stern, and both sides of the deck of the ship, including at least a fisheye camera, a telephoto camera, a depth camera, a black and white camera, and an infrared camera, and are used to obtain image data of the surrounding environment of the ship under different light scenes;

[0016] The data processing and panoramic view display platform is used to synchronize the data obtained by the X-band radar sensors, sonar sensors and camera sensor groups in the entire area or the specified area, and process and fuse the images obtained by various types of camera sensors in the camera sensor group in the same direction in the specified area, balance and adjust the images fused by the camera sensor groups in different directions in the specified area to try to synthesize a panoramic view, gradually and adaptively adjust the orientation of the camera sensor group according to the panoramic view results, and finally synthesize a panoramic view that meets the preset requirements, and process obstacle data to form upper and lower obstacle information on the sea surface and inherent hull data to draw auxiliary lines in the panoramic view, perform real-time obstacle detection, dynamically adjust the position of the camera sensor group to track obstacles, and display the comprehensive panoramic view on the display module.

[0017] By adopting the above technical solution, the sea level and obstacle data above and below the sea level are obtained respectively through X-band radar sensors and sonar sensors, and the image data of the ship's surrounding environment in different light scenes are obtained through multiple groups of camera sensors. In combination with data processing and panoramic view display platforms, the reliability of ship-borne panoramic mooring assistance for large ships is improved, so that the crew can obtain all-round and real-time environmental information, better understand the situation around the ship, effectively prevent collision accidents and improve the accuracy and safety of mooring operations.

[0018] In some embodiments, the data processing and panoramic view display platform includes an image acquisition module, a surface obstacle data acquisition module, an underwater obstacle data acquisition module, a device synchronization module, a scene selection module, an orientation control module, an adaptive adjustment module, an image adjustment module, an image fusion module, a panoramic image synthesis module, an obstacle data processing module, an obstacle tracking module, an auxiliary line module, an obstacle warning module, a touch module and a display module; wherein,

[0019] The image acquisition module is used to open the corresponding device in the camera sensor group to obtain image data according to the camera sensor device number of the user-specified area, and to package the obtained image data and the corresponding device parameters and send them to the device synchronization module;

[0020] The surface obstacle data acquisition module is used to obtain the distance, direction, and motion status data of obstacles on the sea surface and above scanned by the X-band radar sensor, and package them with the corresponding device parameters and send them to the device synchronization module.

[0021] The underwater obstacle data acquisition module is used to obtain the distance, direction, and motion status data of obstacles below the sea surface scanned by the sonar sensor, and package them with the corresponding device parameters and send them to the device synchronization module.

[0022] The device synchronization module is used to unify the clock and synchronize the trigger signal, that is, to synchronize the threads of the surrounding environment image data and obstacle data acquired by the image acquisition module, the surface obstacle data acquisition module, and the underwater obstacle data acquisition module;

[0023] The scene selection module is used to switch the device working scene by setting the scene switching switch on the device panel and schedule the opening and closing of the corresponding camera sensor;

[0024] The orientation control module is used to select the field of view orientation by setting an orientation selection button on the device panel to control the opening and closing of the corresponding orientation camera sensor group, and to display the panoramic view of the corresponding orientation in the main area of ​​the image display module, and to adaptively adjust the position of the camera sensor group according to the control of the adaptive adjustment module to adjust the panoramic view to a suitable position or track obstacles;

[0025] An adaptive adjustment module is used to fine-tune the position of the camera group according to the fused images of the camera group in all directions of the specified area content, and to control the azimuth movement of the camera group to focus on the obstacle when tracking the obstacle;

[0026] An image adjustment module is used to perform deformation conversion processing on images of different cameras with the same orientation and synchronous processing, and transmit them to the image fusion module for image fusion, and perform color balance adjustment, brightness balance adjustment and image size adjustment on the images after fusion of multiple cameras in different orientations, and transmit them to the panoramic image synthesis module for panoramic image synthesis, and receive image zoom-in, zoom-out and rotation instructions input by the touch module to perform corresponding transformations on the images;

[0027] An image fusion module is used to fuse the same-direction and same-group multi-type camera images transformed by the image adjustment module into a single single-direction image by overlapping the single-direction multiple camera images on the image pixel coordinate system using the overlap projection matrix parameters in the parameter memory;

[0028] A panoramic image synthesis module is used to synthesize the images fused and adjusted by multiple cameras in the same orientation into a panoramic image in sequence according to the sensor position information;

[0029] The obstacle data processing module is used to process the obstacle information data above the sea surface obtained by the X-band radar sensor, and the obstacle information data below the sea surface obtained by the sonar sensor;

[0030] The obstacle tracking module is used to track obstacles identified in the panoramic image, transmit obstacle tracking data to the adaptive adjustment module, focus on the obstacles, control the camera sensor group to change the azimuth angle, and display the location details of the obstacles on the display module;

[0031] Auxiliary line module, used to draw auxiliary lines of hull contour, auxiliary lines of horizontal distance in visible area, auxiliary lines of vertical distance in visible area, auxiliary lines of 3D imaging below the sea surface, and auxiliary lines of obstacle motion trajectory in the panoramic image;

[0032] The obstacle warning module is used to detect obstacle information above and below the target ship and on the sea level, draw an obstacle warning logo on the panoramic view, and flash the screen for warning;

[0033] A touch module, used to zoom in, zoom out, rotate, etc. the image of a specified area, transmit the operation instructions to the image adjustment module, and to change the display position of each azimuth view on the display module;

[0034] The display module is used to display the processed comprehensive panoramic image view.

[0035] In some embodiments, the X-band radar sensor has at least a 50° viewing angle in each of the upper, lower, left and right directions, can cover a low altitude of 0 to 1000 m, and can rotate 360°.

[0036] In some embodiments, the camera sensor groups are arranged at unit intervals of at most 5 m, and when the camera sensor groups are working, their maximum downward angle can cover the hull, and their maximum upward angle can cover a distance of up to 1000 m on the same horizontal plane at maximum draft;

[0037] The camera sensor group is installed on the vessel through a cabin, and the camera sensor group can move in and out of the cabin to be retracted into the cabin when not in use.

[0038] In some embodiments, the scene switching switch includes a normal mode, a low light mode, and a dim light mode, wherein:

[0039] The Normal gear position controls the camera sensor group to turn on the fisheye camera, telephoto camera, and depth camera;

[0040] The low-light gear position controls the camera sensor group to turn on the fisheye camera, telephoto camera, depth camera, and black-and-white camera;

[0041] The low-light gear position controls the camera sensor group to turn on the fisheye camera, telephoto camera and infrared camera.

[0042] In some embodiments, a movable mounting seat is provided in the cabin, and the camera sensor group is mounted on the mounting seat.

[0043] The mounting base includes a base fixed in the cabin, a telescopic rod is arranged on the base, a U-shaped fixing frame is arranged at the end of the telescopic rod away from the base, a rotating rod is arranged between two arms of the U-shaped fixing frame, and a camera sensor group is arranged on the rotating rod; and,

[0044] The activities of the telescopic rod and the rotating rod are controlled by the data processing and panoramic view display platform.

[0045] In some embodiments, a power controller is also included to control the opening and closing of an X-band radar sensor, a sonar sensor, multiple camera sensor groups, and a data processing and panoramic view display platform.

[0046] The present application also provides a large-scale shipboard panoramic mooring assistance method, which uses any of the above devices and includes:

[0047] Step 401: Start the device and obtain the internal and external parameters of the camera;

[0048] Step 402: The ship operator selects a panoramic view position according to the requirements, and the device turns on the camera sensor group at the corresponding position;

[0049] Step 403: The ship operator determines the current external environment scene and selects normal, weak light, and dark light working scenes, and the device turns on the corresponding camera device in the camera sensor group;

[0050] Step 404: synchronously acquiring obstacle data and image data acquired by the X-band radar sensor, the sonar sensor, and each camera sensor of the designated orientation camera sensor group;

[0051] Step 405: fusing images acquired by different camera sensors of the same-position camera sensor group;

[0052] Step 406: uniformly adjust brightness, color, and size of the images fused by the camera sensor groups at different orientations in step 405;

[0053] Step 407: stitching the images uniformly adjusted in step 406 into a panoramic view through a panoramic image synthesis module;

[0054] Step 408: performing obstacle auxiliary line conversion on the obstacle data acquired by the X-band radar sensor and the sonar sensor, and synchronizing the obstacle data with the panoramic image acquired in step 407;

[0055] Step 409: adding boundary auxiliary lines, hull contour auxiliary lines and obstacle auxiliary lines to the panoramic image through the auxiliary line module;

[0056] Step 410: Track the obstacle in real time through the obstacle tracking module, and dynamically update it to the panoramic view with auxiliary lines obtained in step 409;

[0057] Step 411: The obstacle warning module flashes an alarm on the obstacle image display area of ​​the panoramic view in step 410, and dynamically updates the panoramic view;

[0058] Step 412: Displaying the comprehensive panoramic image;

[0059] Step 413: Adjust the view size by touching, rotate the 3D view to view the detailed information of the corresponding orientation, or swap the position of the sub-image in the view.

[0060] In some embodiments, obtaining camera internal and external parameters includes:

[0061] Step 101: Initialization, allocating storage space for camera parameters and corner points of all images, and laying out the calibration board;

[0062] Step 102: Use a camera sensor to capture at least 10 images of the calibration plate at different positions and angles;

[0063] Step 103: Read the image and use the Shi-Tomasi algorithm to perform corner detection;

[0064] Step 104: Refine the corner point coordinates and extract the four corner points of each square on the calibration plate for each calibration plate image;

[0065] Step 105: for the image from which the corner points have been successfully extracted, the coordinate values ​​of the corner points in the real world coordinate system and the sub-pixel coordinate values ​​in the image coordinate system are stored;

[0066] Step 106: Calculate the internal parameters of the camera using the extracted corner point coordinates, including the focal length, principal point position, and distortion parameters of the camera;

[0067] Step 107: Calculate the external parameters of the camera according to the rotation angle and translation vector of the camera;

[0068] Step 108: Use the acquired internal and external parameters of the camera to obtain a camera image, and check whether the image is consistent with the real scene. If the error is too large, repeat steps 102 to 107 to re-acquire the internal and external parameters of the camera.

[0069] In some embodiments, step 405: fusing images acquired by different camera sensors of the same-position camera sensor group includes:

[0070] The fisheye camera bird's-eye view is used as the base, and other camera images are superimposed on this image. During superposition, feature detection is used to find local areas in the image that can be stably detected and recognizable between different images, so as to establish the correspondence between the images, adjust the image size, and perform image fusion. Specifically:

[0071] The method of fusing the image with the telephoto camera image is as follows: using the translation-invariant discrete wavelet transform to perform multi-layer filtering to form high-frequency sub-images and low-frequency sub-images of the two images; fusing the high-frequency components according to the high-frequency sub-images corresponding to the two images to form high-frequency component fusion coefficients; fusing the low-frequency components according to the low-frequency sub-images corresponding to the two images to form low-frequency component fusion coefficients; performing the translation-invariant discrete wavelet inverse transform on the high-frequency components corresponding to the high-frequency component fusion coefficients and the low-frequency components corresponding to the low-frequency component fusion coefficients to generate a fused image to achieve the effect of optical zoom;

[0072] The method of fusing an image with a depth camera image is as follows: the depth region is obtained by using the pixel values ​​of adjacent pixels of the depth camera image, and then the subscript of the region contour is obtained by using the approximation method, and the ratio of the value of the subscript pixel under the contour line to the maximum value of the subscript is set as the weight value, and the weight of the pixel point in the non-contour area is set to 1 to obtain a new pixel point weight matrix; the pixel point weight matrix is ​​multiplied by the value of the corresponding point of the original image pixel matrix and then integerized to obtain an image after the original image and the depth camera image are fused. Compared with the original image, the image increases the sharpness of the edge line of the object in the image and highlights the three-dimensional sense;

[0073] The method of fusing the image with the black-and-white camera image is as follows: first, the brightness component and chromaticity component of the fisheye camera image are obtained, and the brightness component and the black-and-white image are fused. The brightness component pixel matrix value and the black-and-white image pixel matrix value of the corresponding point are averaged and replaced to obtain the brightness fusion component pixel matrix; the target image is obtained by combining the brightness fusion component and the chromaticity component. Compared with the original image, the image brightness in the dark scene can be significantly improved, and the noise can be reduced at the same time;

[0074] Method for fusing the image with the infrared camera image: In a dark scene, turn on the infrared fill light to obtain the infrared image;

[0075] Step 407: stitching the images uniformly adjusted in step 406 into a panoramic view through a panoramic image synthesis module, including:

[0076] The images uniformly adjusted in step 406 are arranged according to the image position information, the overlapping areas are merged, and then the images are subjected to image brightness balance processing and color balance processing; wherein,

[0077] The fusion method used for the overlapping area of ​​the image is to set a weight coefficient for each pixel point. The weight coefficient changes with the pixel value of the point and changes continuously with the distance between each pixel point and the median line.

[0078] The method used for balancing the brightness of multiple groups of images is to calculate a coefficient for each of the three RGB channels corresponding to the panorama with brightness differences after fusion, and then multiply it with the original channel value. The over-bright channel is dimmed with a coefficient less than 1, and the over-dark channel is brightened with a coefficient greater than 1, and finally a picture with adjusted brightness is formed;

[0079] The method used for color balancing of each group of images is to obtain the mean of the total value of the RGB three channels in the pixel matrix for the panoramic image after the image brightness is balanced. The value of the RGB three channels of each pixel point is obtained by multiplying the original value with the mean and then dividing it by the mean of the channel of the pixel matrix.

[0080] Other features and corresponding beneficial effects of the present application are described in the latter part of the specification, and it should be understood that at least some of the beneficial effects become obvious from the records in the specification of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is a schematic structural block diagram of the panoramic mooring auxiliary device for large ships in the present invention;

[0082] Figure 2 An exemplary schematic diagram of the layout of the panoramic view and obstacle tracking view in all directions displayed by the display module of the present invention;

[0083] Figure 3 is an exemplary diagram of a comprehensive panoramic view displayed by the display module when obstacles are detected in the left front, left, and rear of the vessel in the present invention;

[0084] Figure 4 is a simplified diagram of an example of installation positions of an X-band radar sensor, a sonar sensor, and a camera sensor group in the present invention;

[0085] Figure 5 It is a structural schematic diagram of the camera sensor group and its mounting base in the present invention;

[0086] Figure 6 is a schematic flow chart of a large vessel onboard panoramic mooring assistance method in the present invention;

[0087] Figure 7 This is an exemplary schematic diagram of laying out the calibration plate for the camera sensor group in the present invention.

[0088] Description of reference numerals:

[0089] 10. Base; 20. Telescopic rod; 30. U-shaped fixing bracket; 40. Rotating rod; 50. Fisheye camera; 60. Telephoto camera; 70. Depth camera; 80. Black and white camera; 90. Infrared fill light. DETAILED DESCRIPTION

[0090] The following specific embodiments illustrate the implementation of the present application, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Although the description of the present application will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this application are limited to the implementation. On the contrary, the purpose of introducing the application in conjunction with the implementation is to cover other options or modifications that may be extended based on the claims of the present application. In order to provide a deep understanding of the present application, the following description will include many specific details. The present application can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present application, some specific details will be omitted in the description. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0091] It should be noted that in this specification, similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0092] The technical solution of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0093] In the description of the present application, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description created by the present application, unless otherwise specified, "multiple" means two or more.

[0094] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0095] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0096] Embodiment 1:

[0097] See also Figure 1 , Figure 1 It is a schematic block diagram of the structure of the panoramic mooring assistance device for large ships in this embodiment.

[0098] An embodiment of the present application provides a large ship-borne panoramic mooring assistance device, including an X-band radar sensor, a sonar sensor, multiple camera sensor groups, and a data processing and panoramic view display platform.

[0099] The X-band radar sensor is installed high on the bridge and is used to obtain data on the distance, direction and motion status of obstacles above the sea surface and the ship.

[0100] The sonar sensor is installed on the bow or bottom of the ship and is used to obtain the distance, direction and motion status data of obstacles below the sea surface and the ship.

[0101] Multiple camera sensor groups are installed at intervals on the bow, stern, and both sides of the deck of the ship, including at least fisheye cameras, telephoto cameras, depth cameras, black and white cameras, and infrared cameras, and are used to obtain image data of the surrounding environment of the ship under different light scenes. The fisheye camera preferably uses a fisheye camera with an ultra-telephoto of more than 180° and high resolution. It is understandable that multiple camera sensor groups can be installed at a longer distance in the same straight line, and the image pictures obtained by all camera sensors are required to completely cover the surrounding environment of the ship. It is understandable that fisheye cameras are usually used to obtain views with large telephoto differences, telephoto cameras are usually used to obtain clear images of objects at a long distance, depth cameras are usually used to obtain depth information of objects through images, and black and white cameras are usually used to enhance sensitivity and reduce noise levels to provide clearer images. In this embodiment, the infrared camera is provided with an infrared fill light for the camera group to obtain clear images in a dark environment.

[0102] The data processing and panoramic view display platform is used to synchronize the data obtained by the X-band radar sensors, sonar sensors and camera sensor groups in the entire area or the specified area, and process and fuse the images obtained by various types of camera sensors in the camera sensor group in the same direction in the specified area, balance and adjust the images fused by the camera sensor groups in different directions in the specified area to try to synthesize a panoramic view, gradually and adaptively adjust the orientation of the camera sensor group according to the panoramic view results, and finally synthesize a panoramic view that meets the preset requirements, and process obstacle data to form upper and lower obstacle information on the sea surface and inherent hull data to draw auxiliary lines in the panoramic view, perform real-time obstacle detection, dynamically adjust the position of the camera sensor group to track obstacles, and display the comprehensive panoramic view on the display module.

[0103] This method uses X-band radar sensors and sonar sensors to obtain sea level and obstacle data above and below the sea level respectively, and uses multiple camera sensor groups to obtain image data of the surrounding environment of the ship under different light scenes, so as to sample the complex hydrological information and obstacle information around large ships, thereby improving the reliability of ship-borne panoramic mooring assistance for large ships; and cooperates with the data processing and panoramic view display platform to quickly process and analyze data, splice and fuse the images of each camera, add distance auxiliary lines to the panoramic image, dynamically adjust the angle of the camera sensor group according to the obstacle distance in real time, track the dangerous area, etc., further improving the reliability of ship-borne panoramic mooring assistance for large ships, so that the crew can obtain all-round and real-time environmental information, better understand the situation around the ship, effectively prevent collision accidents and improve the accuracy and safety of mooring operations. That is, it solves the technical problems in the prior art that the application scenarios of the ship-borne panoramic surround view are small, the panoramic surround view coverage is incomplete and there are visual blind spots, the ship operator cannot obtain the sea surface and seabed obstacle information in real time and synchronously, the obstacle is tracked at a very close distance, and the single distance measurement sensor has low measurement robustness.

[0104] In one embodiment, the data processing and panoramic view display platform includes an image acquisition module, a surface obstacle data acquisition module, an underwater obstacle data acquisition module, a device synchronization module, a scene selection module, an orientation control module, an adaptive adjustment module, an image adjustment module, an image fusion module, a panoramic image synthesis module, an obstacle data processing module, an obstacle tracking module, an auxiliary line module, an obstacle warning module, a touch module and a display module. Preferably, the data processing and panoramic view display platform is built on the AGX Xavier computing platform.

[0105] In one embodiment, the image acquisition module is used to open the corresponding device in the camera sensor group to obtain image data according to the camera sensor device number of the user-specified area, and to package the obtained image data with the corresponding device parameters and send them to the device synchronization module.

[0106] In one embodiment, the surface obstacle data acquisition module is used to obtain the distance, direction, and motion status data of obstacles on and above the sea surface scanned by the X-band radar sensor, and package them with corresponding device parameters and send them to the device synchronization module.

[0107] In one embodiment, the underwater obstacle data acquisition module is used to obtain the distance, direction, and motion status data of obstacles below the sea surface scanned by the sonar sensor, and package them with corresponding device parameters and send them to the device synchronization module.

[0108] In one embodiment, the device synchronization module is used to unify the clock and synchronize the trigger signal, that is, to synchronize the threads of the surrounding environment image data and obstacle data acquired by the image acquisition module, the surface obstacle data acquisition module, and the underwater obstacle data acquisition module. A counter is used to count each time a set of data is acquired, and the current thread is placed in a sleep pool. When the count reaches the number of sensors running in the specified area, all threads are awakened to enter the next round of task cycles, thereby ensuring the stability, real-time and synchronization of data transmission and preventing confusion during image processing.

[0109] In one embodiment, the scene selection module is used to switch the device working scene by setting a scene switching switch on the device panel, for example, sliding the switch to schedule the opening and closing of the corresponding camera sensor to meet the actual needs of the user in different lighting scenes.

[0110] In one embodiment, the scene switching switch includes a normal mode, a low light mode, and a dark light mode. The normal mode is usually used in daytime or well-lit scenes, the low light mode is used in scenes with dim light but still with a certain amount of visible light, and the dark light mode is used in very dim or completely dark scenes such as at night.

[0111] In a specific implementation, the normal gear position controls the camera sensor group to turn on the fisheye camera, the telephoto camera, and the depth camera.

[0112] The low-light gear position controls the camera sensor group to turn on the fisheye camera, telephoto camera, depth camera and black and white camera.

[0113] The low-light gear position controls the camera sensor group to turn on the fisheye camera, telephoto camera and infrared camera.

[0114] In one embodiment, the orientation control module is used to select the field of view orientation by setting an orientation selection button on the device panel to control the opening and closing of the corresponding orientation camera sensor group, and display the panoramic view of the corresponding orientation in the main area of ​​the image display module.

[0115] For example, the orientation control module may include 6 selection buttons, button 1 is used to select the full range, button 2 is used to select the bow orientation of the target ship, button 3 is used to select the stern orientation of the target ship, button 4 is used to select the left side orientation of the target ship, button 5 is used to select the right side orientation of the target ship, and button 6 is used to display the bottom orientation of the target ship. When button 1 is selected, the full range camera sensor group is turned on, and the initial azimuth is set to the maximum azimuth, at which time the environment image of the farthest area within the field of view is obtained; when button 6 is selected, the full range camera sensor group is turned on, and the initial azimuth is set to the minimum azimuth, at which time the environment image of the nearest area close to the ship body within the field of view is obtained; when only a partial orientation panoramic ring view is selected to be displayed, each area on the display module also only displays the panoramic ring view under the specified orientation.

[0116] It is understandable that the display orientation can be further subdivided, mainly by selecting the corresponding sensor to be turned on through the marking of the orientation sensor device number, and synthesizing the panoramic image of the specified orientation. Alternatively, multiple sensor devices with different orientation angles can be combined, and the image display is performed in the form of merging adjacent orientation images and displaying non-adjacent orientation images separately when synthesizing the panoramic image.

[0117] In one embodiment, the orientation control module is further used to adaptively adjust the position of the camera sensor group according to the control of the adaptive adjustment module, so as to adjust the panoramic view to a suitable position or track an obstacle target.

[0118] In one embodiment, the adaptive adjustment module is used to fine-tune the position of the camera group according to the fusion image of the camera group in each direction of the specified area content, so that the synthesized panoramic image is coherent and smooth. Figure 5 The rotating rod 40 in the image processing apparatus is adjusted at an angle of 1° at a time to gradually check the panoramic imaging effect, so that the synthesized panoramic image is continuous and smooth.

[0119] In one embodiment, the adaptive adjustment module is also used to control the azimuth movement of the camera group to focus on the obstacle when tracking the obstacle, so that the obstacle is always in the center of the obstacle tracking window until it completely leaves the panoramic view field.

[0120] In one embodiment, the image adjustment module is used to perform deformation conversion processing on images of different cameras with the same orientation and synchronous processing, so that the images of each camera meet the fusion condition and are transmitted to the image fusion module for image fusion.

[0121] In one embodiment, the image adjustment module is further used to perform color balance adjustment, brightness balance adjustment and image size adjustment on the image after fusion of multiple camera groups in different orientations, and transmit it to the panoramic image synthesis module for panoramic image synthesis.

[0122] In one embodiment, the image adjustment module is further used to receive image zoom-in, zoom-out, and rotation instructions input by the touch control module, and perform corresponding transformations on the image.

[0123] In one embodiment, the image fusion module is used to fuse the multi-type camera images of the same orientation and group transformed by the image adjustment module into a single single-orientation image by overlapping the single-orientation multiple camera images on the image pixel coordinate system using the overlapping projection matrix parameters in the parameter memory.

[0124] In one embodiment, the panoramic image synthesis module is used to synthesize the panoramic image in sequence according to the sensor position information by fusing and adjusting the images of multiple cameras in the same orientation.

[0125] In one embodiment, the obstacle data processing module is used to process obstacle information data on and above the sea surface acquired by an X-band radar sensor, and to process obstacle information data below the sea surface acquired by a sonar sensor, so as to facilitate subsequent drawing of field of view auxiliary lines, obstacle tracking, and obstacle warning.

[0126] In one embodiment, the obstacle tracking module is used to track obstacles identified in the panoramic image, transmit obstacle tracking data to the adaptive adjustment module, focus on the obstacles, control the camera sensor group to transform the azimuth angle, and display the location details of the obstacles on the display module.

[0127] In one embodiment, the auxiliary line module is used to draw hull contour auxiliary lines, visible area horizontal distance auxiliary lines, visible area vertical distance auxiliary lines, 3D imaging auxiliary lines below the sea surface, and obstacle motion trajectory auxiliary lines in the panoramic image.

[0128] In one embodiment, the obstacle warning module is used to detect obstacle information above and below the target ship and above the sea level, draw an obstacle warning logo on the panoramic view, and perform a screen flashing warning.

[0129] In one embodiment, the touch module is used to zoom in, zoom out, rotate, etc. the image of the specified area, transmit the operation instructions to the image adjustment module, and is used to exchange the various directional views (such as Figure 2 The display position of the two images (as shown) on the display module means that the ship operator can exchange the image positions by touching the images in two different areas, which helps the ship operator to accurately grasp the details of the specified view in a larger view area.

[0130] Figure 2 It is an exemplary schematic diagram of the layout of the panoramic views and obstacle tracking views in all directions displayed by the display module in this embodiment.

[0131] In one embodiment, the display module is used to display the processed comprehensive panoramic image view (such as Figure 3 shown).

[0132] Figure 3 This is an exemplary diagram of the comprehensive panoramic view displayed by the display module when obstacles are detected in the left front, left, and rear of the vessel in this embodiment.

[0133] Please see again Figure 1 In one embodiment, the device further includes a power controller for controlling the on and off of the X-band radar sensor, the sonar sensor, the multiple camera sensor groups, and the data processing and panoramic view display platform. Usually, the power controller is located on the ship control panel and is controlled by a single button to control the on and off of the device. Pressing the button is on, and popping up the button is off.

[0134] See also Figure 4 , Figure 4 : is a simplified diagram of an example of the installation positions of the X-band radar sensor, sonar sensor and camera sensor group in this embodiment, wherein A represents the installation position distribution of the camera sensor group, that is, they are installed at intervals at the bow, stern and both sides of the deck of the ship; B represents the installation position distribution of the X-band radar sensor, that is, they are installed high at the bridge; C represents the installation position distribution of the sonar sensor, that is, they are installed at the bow or bottom of the ship, so as to comprehensively obtain the data of the surrounding environment of the ship.

[0135] In one embodiment, the X-band radar sensor has at least a 50° viewing angle up, down, left, and right, and can cover a low altitude of 0 to 1000 m, and can rotate 360° to obtain data such as the distance, direction, and motion status of obstacles on the sea surface around the ship and above.

[0136] In one embodiment, the camera sensor groups are arranged at unit intervals of up to 5m, and when the camera sensor groups are working, their maximum downward angle can cover the hull, and their maximum upward angle can cover a distance of up to 1000m on the same horizontal plane at maximum draft.

[0137] See also Figure 5 , Figure 5 Schematic diagram of the structure of the camera sensor group and its mounting base in this embodiment.

[0138] In one embodiment, the camera sensor group is installed on the ship through a cabin, and the camera sensor group can enter and exit the cabin, so as to be retracted into the cabin when not in use, thereby protecting the camera sensor group and reducing the risk of uncontrolled displacement of the camera sensor group.

[0139] In one embodiment, a movable mounting seat is provided in the cabin, and the camera sensor group is mounted on the mounting seat.

[0140] The mounting base includes a base 10 fixed in the cabin, a telescopic rod 20 is arranged on the base 10, a U-shaped fixing frame 30 is arranged at the end of the telescopic rod 20 away from the base 10, a rotating rod 40 is arranged between the two arms of the U-shaped fixing frame 30, and the camera sensor group is arranged on the rotating rod 40.

[0141] The activities of the telescopic rod 20 and the rotating rod 40 are controlled by the data processing and panoramic view display platform to adjust the position of the camera sensor group, such as entering and exiting the cabin and adjusting the angle. This, on the one hand, improves the imaging effect and facilitates the stitching of smooth panoramic images. On the other hand, it realizes angle adjustment when tracking obstacles, improves the tracking effect, and thus improves the reliability of parking assistance.

[0142] In this embodiment, the camera sensor group includes a fisheye camera 50 , a telephoto camera 60 , a depth camera 70 , a black and white camera 80 , and an infrared fill light 90 .

[0143] In summary, the device of this embodiment is configured with multiple groups of cameras, including fisheye, depth, telephoto, infrared and black and white cameras, which can be combined to form camera groups. By using these camera groups, scenes in a larger range can be fully monitored and high-quality panoramic surround views can be obtained.

[0144] In different lighting scenarios, different camera combinations can be selected as needed to obtain a clearer and more recognizable local or global field of view.

[0145] In addition, this embodiment also combines distance measuring devices such as X-band radar and sonar, and uses the information provided by multiple sensors to add auxiliary lines, such as obstacle distance lines and ship outlines. Using X-band radar and sonar ranging, the distance and azimuth information of obstacles can also be obtained and directly drawn on the panoramic image. When an obstacle approaches the ship, the azimuth of the camera group will automatically adjust to track the movement of the obstacle.

[0146] The image display of this embodiment can be switched flexibly, so that the user can observe the image in a specified range more carefully. When it is necessary to observe the image in a specific direction, it is only necessary to start the sensor in the corresponding direction, thereby reducing energy consumption and computing burden and reducing image transmission delay.

[0147] By using the device of this embodiment, the ship operator can accurately understand the conditions of the vessel's surrounding environment, including berthing, unberthing and encountering other ships or obstacles, and obtain vertical depth and horizontal distance perception, and understand the ground environment conditions of the vessel's periphery and bottom areas.

[0148] In addition, this embodiment also provides an obstacle warning function, which can help the operator accurately grasp the distance between the ship and the obstacle and determine the safety range, thereby reducing the risk of accidents.

[0149] Embodiment 2:

[0150] See also Figure 6 , Figure 6 It is a schematic flow chart of the large vessel onboard panoramic mooring assistance method in this embodiment.

[0151] Based on Example 1, the present application embodiment further provides a large-scale shipboard panoramic mooring assistance method, using any of the above devices, and the method includes:

[0152] Step 401: Start the device and obtain the internal and external parameters of the camera.

[0153] In step 401, when the camera calibration is not performed, the switch control module is used to turn on the device. The images around the ship obtained by the camera sensor group need to cover the inner and outer edges of the calibration plate, or at least one layer of the black and white grid pattern of the calibration plate, and the images obtained by all camera sensors can cover the external environment of 1000m and 360° outward from the ship and the environment close to the hull. Set the length and width of the bird's-eye view, the length and width from the inner circle to the outer circle of the calibration plate, the distance from the calibration plate to the target ship, and the values ​​of the four world coordinate system coordinate points measured with the upper left corner of the bird's-eye view as the origin and the length and width of the calibration plate grid. Use the imaging principle of the pinhole camera model to obtain the internal parameters and distortion coefficients of each camera sensor. By adjusting the horizontal and vertical scaling ratios of the image and the rigid body transformation operations such as the center coordinate point, the distorted original pattern of the camera sensor is converted from the world coordinate system to the camera coordinate system image. The transformed image is selected in the image physical coordinate system in turn according to the preset 4 coordinate points, so as to obtain the mapping matrix of the camera sensor, and the perspective projection is performed in combination with the camera sensor internal parameter matrix and the distortion coefficient to convert the image from the camera coordinate system to the image coordinate system. At this time, the bird's-eye view of a single camera sensor is obtained, and the parameters of the severely distorted bird's-eye view are set, and the normal bird's-eye view is stored as the camera internal and external parameters. In addition, according to the preset coordinate points on the image, the image overlay projection matrix of multiple different types of camera sensors in the same camera group converted to the same image pixel coordinate system is solved. For the X-band radar sensor and sonar sensor, the conversion matrix parameters of the data point world coordinate system transformed to the image pixel coordinate system, the depth auxiliary line, the horizontal distance auxiliary line, and the pixel matrix of the ship outline auxiliary line in the preset image pixel coordinate system are obtained by calculating the actual measured distance and the azimuth distance of multiple coordinate points relative to the calibration plate. In addition, it is also necessary to set and measure fixed parameters such as the device number and position of each orientation sensor, the obstacle recognition error distance, etc. and store them in the parameter memory.

[0154] If the camera sensor calibration has been performed, then step 401 can directly proceed to step 402 using the real-time data obtained by the X-band radar sensor, sonar sensor and camera sensor group.

[0155] Step 402: The ship operator selects a panoramic view orientation according to needs, and the device turns on the camera sensor group at the corresponding orientation.

[0156] Preferably, the orientation of the panoramic image can be switched at any time, and at the same time, the images displayed therein are also changed, and only the panoramic image synthesized at the specified orientation is displayed on the overall picture.

[0157] Step 403: The ship operator determines the current external environment scene and selects normal, low-light, or dark-light working scenes, and the device turns on the corresponding camera device in the camera sensor group.

[0158] Preferably, the working scene can also be switched at any time, and the screen after the working scene is switched is synchronously updated to the display module screen.

[0159] Step 404: synchronously obtain obstacle data and image data obtained by the X-band radar sensor, the sonar sensor, and each camera sensor of the designated orientation camera sensor group.

[0160] Preferably, the fisheye camera image needs to be transformed for image distortion correction first, and then other camera images are transformed accordingly to obtain bird's-eye views of each camera image, and the bird's-eye views of multiple different types of cameras in the same camera group are overlapped to obtain a new bird's-eye view.

[0161] Step 405: fuse the images acquired by different camera sensors of the camera sensor group in the same orientation.

[0162] Step 406: uniformly adjust brightness, color, and size of the image fused by the camera sensor groups in different orientations in step 405.

[0163] Step 407: The images uniformly adjusted in step 406 are stitched into a panoramic view through a panoramic image synthesis module.

[0164] Preferably, when images are stitched into a panoramic view, the bird's-eye view is sorted by sensor position and rotated according to a preset angle. The images of non-overlapping areas are mapped to the image pixel matrix of the corresponding position of the panoramic view as they are, and the images of overlapping areas are fused according to a pixel-smoothed weight matrix.

[0165] Step 408: The obstacle data acquired by the X-band radar sensor and the sonar sensor are converted into obstacle auxiliary lines, and synchronized with the panoramic image acquired in step 407.

[0166] Step 409: Add boundary auxiliary lines, hull contour auxiliary lines and obstacle auxiliary lines to the panoramic image through the auxiliary line module.

[0167] Preferably, there are differences in the auxiliary lines added to different views, but they mainly include vertical and horizontal dividing lines, ship contour lines, obstacle contour lines, etc. The ship contour auxiliary lines are scaled proportionally by the bird's-eye view of the original target ship in the world coordinate system, and only the outer edges are drawn.

[0168] Step 410: Track the obstacle in real time through the obstacle tracking module, and dynamically update it to the panoramic view with auxiliary lines obtained in step 409.

[0169] Step 411: The obstacle warning module flashes an alarm on the obstacle image display area of ​​the panoramic view in step 410, and dynamically updates it to the panoramic view.

[0170] Step 412: Display the integrated panoramic image.

[0171] Step 413: Adjust the view size by touching, rotate the 3D view to view the detailed information of the corresponding orientation, or swap the position of the sub-image in the view.

[0172] In one embodiment, obtaining camera internal and external parameters includes:

[0173] Step 101: Initialization, allocating storage space for camera parameters and corner points of all images, and laying out the calibration board;

[0174] Step 102: Use a camera sensor to capture at least 10 images of the calibration plate at different positions and angles;

[0175] Step 103: Read the image and use the Shi-Tomasi algorithm to perform corner detection;

[0176] Step 104: Refine the corner point coordinates and extract the four corner points of each square on the calibration plate for each calibration plate image;

[0177] Step 105: for the image from which the corner points have been successfully extracted, the coordinate values ​​of the corner points in the real world coordinate system and the sub-pixel coordinate values ​​in the image coordinate system are stored;

[0178] Step 106: Calculate the internal parameters of the camera using the extracted corner point coordinates, including the focal length, principal point position, and distortion parameters of the camera;

[0179] Step 107: Calculate the external parameters of the camera according to the rotation angle and translation vector of the camera;

[0180] Step 108: Use the acquired internal and external parameters of the camera to obtain a camera image, and check whether the image is consistent with the real scene. If the error is too large, repeat steps 102 to 107 to re-acquire the internal and external parameters of the camera.

[0181] See also Figure 7 , Figure 7 This is an exemplary schematic diagram of the installation of the camera sensor group calibration plate in this embodiment.

[0182] In step 101, the schematic diagram of laying the calibration plate is as follows Figure 7 As shown, black and white checkered marking plates are laid around the hull when the vessel is at its maximum draft.

[0183] When obtaining the internal and external parameters of the camera in step 107 and step 108, four coordinate systems are predefined, namely the world coordinate system, the camera coordinate system, the image physical coordinate system and the image pixel coordinate system.

[0184] The world coordinate system is the coordinate system of an object in the real world. This coordinate system sets the first checkerboard vertex in the upper left corner of the black and white grid calibration plate as the origin, and the direction parallel to the hull as X w Axis direction, the horizontal direction perpendicular to the hull is Y w Axis direction, the value direction perpendicular to the hull is Z w The camera coordinate system takes the optical center as the origin of the camera coordinate system and the direction parallel to the horizontal line of the image as the X axis. c Axis direction, with the direction parallel to the vertical line of the image as Y c Axis direction, Z c The axial direction is the direction of the straight line parallel to the optical axis.

[0185] The physical coordinate system of the image takes the intersection of the principal optical axis and the image plane as the coordinate origin, and the directions of the X-axis and Y-axis are the same as the horizontal x and vertical y directions of the image.

[0186] The image pixel coordinate system takes the vertex of the image as the coordinate origin, and the u-axis and v-axis directions are parallel to the horizontal x and vertical y directions of the image coordinate system, with pixels as units.

[0187] The scene view transforms a 3D point P of the scene into w Obtained by projecting it onto the image plane. w Both and p are expressed in homogeneous coordinates, that is, they are expressed as 3D and 2D homogeneous vectors respectively. The distortion-free projection transformation formula given by the pinhole camera model is as follows:

[0188] sp=A[R|t]P w

[0189] Where P wis a 3D point expressed relative to the world coordinate system, p is a 2D pixel in the image plane, A is the camera intrinsic matrix, R and t are the rotation and translation describing the change from world coordinates to camera coordinates in the coordinate system (or camera frame), and s is an arbitrary scale for the projective transformation.

[0190] The camera intrinsic matrix A projects a given 3D point in the camera coordinate system to 2D pixel coordinates, namely:

[0191] p=AP c

[0192] The camera intrinsic matrix A is given by the focal length f x and f y and the principal point (c x , c y )composition:

[0193]

[0194] but:

[0195]

[0196] The intrinsic parameter matrix does not depend on the observed scene. Therefore, it can be reused as long as the focal length is fixed. If the image from the camera is scaled by a factor, then all these parameters need to be scaled by the same factor.

[0197] The joint rotation and translation matrix [R|t] is the matrix product of the projective transformation and the homogeneous transformation. The 3×4 projective transformation maps 3D points expressed in camera coordinates to 2D points in the image plane and normalizes the camera coordinates x' and y' as:

[0198]

[0199]

[0200] The extrinsic parameters R and t are related to the homogeneous transformation of the cardinality from the real-world coordinate system w to the camera coordinate system c. Given a point P in the camera coordinate system P c With world coordinates P w The relationship is as follows:

[0201]

[0202] The homogeneous transformation consists of R (3×3 rotation matrix) and t (3×1 translation vector):

[0203]

[0204] but:

[0205]

[0206] Combining the projection transformation and the homogeneous transformation, we get the projection transformation that maps 3D points in world coordinates to 2D points in the image plane and the normalized camera coordinate system:

[0207]

[0208]

[0209] If Z c ≠0, then the above transformation is equivalent to:

[0210]

[0211]

[0212] Adding radial distortion and tangential distortion, the above formula is transformed into:

[0213]

[0214] in:

[0215]

[0216] r2=x′ 2 +y′ 2

[0217] Among them, k1, k2, k3, k4, k5 and k6 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients, and s1, s2, s3 and s4 are thin prism distortion coefficients.

[0218] Through the above formula, we only need to obtain the coordinates of the corresponding points of the calibrated chessboard in a three-dimensional scene and its image in the imaging plane, and then we can solve the homography matrix of the two planes. Among them, the calibrated chessboard is specially made, and the coordinates of its corner points are known; the position of the corner point in the image can be determined by each vector from the center q to the point p in the neighborhood of q being orthogonal to the image gradient at p. The corner point extraction formula used here is as follows:

[0219]

[0220] Among them, DI T pi is the image gradient at a point q. It can be expressed as ∈ i Set it to 0 to create a system of equations to solve for the value of q:

[0221]

[0222] Here, the gradient is summed over a neighborhood of q. Calling the first gradient term G and the second gradient term b gives:

[0223] q=G- 1 *b

[0224] The center of the neighborhood window is set at this new center q, and then iterate until the center remains within a set threshold. After solving the homography matrix through the corresponding points, the pixel point (x, y) in the 2D image is mapped and transformed:

[0225] dst(x, y) = src(f x (x,y),f y (x, y)

[0226] A perspective transformation matrix can be calculated using the coordinates of the four corresponding points:

[0227]

[0228] dst(i)=(x′ i , y′ i )src(i)=(x i ,y i ), i = 0, 1, 2, 3

[0229] By calculating the perspective transformation of each coordinate system, we can obtain the camera's intrinsic parameter matrix, distortion coefficient, camera's projection matrix and translation vector.

[0230] In one embodiment, step 405: fusing images acquired by different camera sensors of the same-position camera sensor group includes:

[0231] The fisheye camera bird's-eye view is used as the base, and other camera images are superimposed on this image. During superposition, feature detection is used to find local areas in the image that can be stably detected and recognizable between different images, so as to establish the correspondence between the images, adjust the image size, and perform image fusion. Specifically:

[0232] The method of fusing the image with the telephoto camera image is as follows: using the translation-invariant discrete wavelet transform to perform multi-layer filtering to form high-frequency sub-images and low-frequency sub-images of the two images; fusing the high-frequency components according to the high-frequency sub-images corresponding to the two images to form high-frequency component fusion coefficients; fusing the low-frequency components according to the low-frequency sub-images corresponding to the two images to form low-frequency component fusion coefficients; performing the translation-invariant discrete wavelet inverse transform on the high-frequency components corresponding to the high-frequency component fusion coefficients and the low-frequency components corresponding to the low-frequency component fusion coefficients to generate a fused image to achieve the effect of optical zoom;

[0233] The method of fusing an image with a depth camera image is as follows: the depth region is obtained by using the pixel values ​​of adjacent pixels of the depth camera image, and then the subscript of the region contour is obtained by using the approximation method, and the ratio of the value of the subscript pixel under the contour line to the maximum value of the subscript is set as the weight value, and the weight of the pixel point in the non-contour area is set to 1 to obtain a new pixel point weight matrix; the pixel point weight matrix is ​​multiplied by the value of the corresponding point of the original image pixel matrix and then integerized to obtain an image after the original image and the depth camera image are fused. Compared with the original image, the image increases the sharpness of the edge line of the object in the image and highlights the three-dimensional sense;

[0234] The method of fusing the image with the black-and-white camera image is as follows: first, the brightness component and chromaticity component of the fisheye camera image are obtained, and the brightness component and the black-and-white image are fused. The brightness component pixel matrix value and the black-and-white image pixel matrix value of the corresponding point are averaged and replaced to obtain the brightness fusion component pixel matrix; the target image is obtained by combining the brightness fusion component and the chromaticity component. Compared with the original image, the image brightness in the dark scene can be significantly improved, and the noise can be reduced at the same time;

[0235] Method for fusing the image with the infrared camera image: In a dark scene, turn on the infrared fill light to obtain an infrared image.

[0236] In one embodiment, step 406: uniformly adjust brightness, color, and size of the image fused by the camera sensor groups in different orientations in step 405. This step can be performed by an image adjustment module.

[0237] The brightness balance adjustment method in the image adjustment module is to calculate a coefficient for each of the three RGB channels of the panorama with brightness differences after fusion, and then multiply them with the original channel values. The over-bright channels are dimmed and the multiplication coefficient is less than 1, and the over-dark channels are brightened and the multiplication coefficient is greater than 1, and finally the picture with brightness adjustment is formed. The calculation is performed by the following formula:

[0238]

[0239]

[0240]

[0241]

[0242] Where M1 is the pixel matrix of the first camera image of two adjacent cameras, M2 is the pixel matrix of the second camera image of two adjacent cameras, W represents the corresponding grayscale and binarization adjustment coefficient matrix, L r is the average brightness ratio of two adjacent camera images, L r' is the average brightness ratio of the images stitched from the current two cameras, L r ” is the average brightness ratio of the images stitched together by the other cameras except the current two cameras, L is the average brightness ratio of the images stitched together in different orientations, L' is the brightness coefficient, P is the pixel matrix of the original image, and P' is the pixel matrix after brightness adjustment.

[0243] The method of color balance adjustment in the image adjustment module is to find the average of the total values ​​of the three RGB channels in the pixel matrix of the panoramic image after the image brightness is balanced. The images sent back by N cameras have a total of 3N channels. Calculate a coefficient for each channel, and then multiply these 3N coefficients by these 3N channels respectively, and then merge them to form the adjusted image. Because there will be color difference in each channel of different images, it is necessary to adjust the channel values ​​that are too bright or too dark. The method is to add the average values ​​of each channel of all image pixels and average them. Multiply the original channel value by the ratio of its mean to the channel mean, and you can get the adjusted value of the pixel under the channel, so that the over-bright channel is multiplied by a coefficient less than 1, and the over-dark channel is multiplied by a coefficient greater than 1. The following formula is used here:

[0244]

[0245]

[0246] Among them, M R 、M B 、M G is the three-primary color matrix of the image after brightness adjustment, R, G, B are the three-primary color values ​​of a pixel on the image, R W , G W , B W It is the three primary color values ​​of the pixel after color adjustment.

[0247] In one embodiment, step 407: stitching the images uniformly adjusted in step 406 into a panoramic view by a panoramic image synthesis module includes:

[0248] The images uniformly adjusted in step 406 are arranged according to the image position information, the overlapping areas are merged, and then the images are subjected to image brightness balance processing and color balance processing; wherein,

[0249] The fusion method used for the overlapping area of ​​the image is to set a weight coefficient for each pixel point. The weight coefficient changes with the pixel value of the point and changes continuously with the distance between each pixel point and the median line.

[0250] The method used for balancing the brightness of multiple groups of images is to calculate a coefficient for each of the three RGB channels corresponding to the panorama with brightness differences after fusion, and then multiply it with the original channel value. The over-bright channel is dimmed with a coefficient less than 1, and the over-dark channel is brightened with a coefficient greater than 1, and finally a picture with adjusted brightness is formed;

[0251] The method used for color balancing of each group of images is to obtain the mean of the total value of the RGB three channels in the pixel matrix for the panoramic image after the image brightness is balanced. The value of the RGB three channels of each pixel point is obtained by multiplying the original value with the mean and then dividing it by the mean of the channel of the pixel matrix.

[0252] In one embodiment, the images uniformly adjusted in step 406 are arranged according to the image position information, and the overlapping areas are merged, including:

[0253] Step 201: extracting the overlapping area in the fused images of two adjacent camera sensor groups;

[0254] Step 202: grayscale and binarize the overlapped area image obtained in step 201;

[0255] Step 203: Use morphological operation to remove noise from the image processed in step 202. The specific operation is to slide the structure element on the original image and set the gray value of the image pixel at the structure element anchor point to the minimum value of the pixel in the image area corresponding to the area where the structure element value is 1. It can be expressed as follows:

[0256]

[0257] Where element is the structural element, (x, y) is the position of the anchor point O, x' and y' are the position offsets of the pixels with structural element value 1 relative to the anchor point O, src is the original image, and dst is the result image.

[0258] Step 204: using the image processed in step 203 to detect the boundary outside the overlapping area using an approximation method;

[0259] Step 205: Calculate the weight of each pixel in the overlapping area to the boundary area, use the target pixel coordinates and the contour coordinates of the nearest overlapping area to solve the distance from the pixel to the two boundaries of the overlapping area, and use the distance ratio to obtain the weight value of each pixel in the overlapping area;

[0260] Step 206: Set the weights of the pixels in the non-overlapping area to obtain a continuously changing weight matrix. The weights of the pixels in the first camera image of the two adjacent cameras that are not in the overlapping area are all set to 1, and the weights of the pixels in the second camera image of the two adjacent cameras that are not in the overlapping area are all set to 0. Combined with the weight value matrix of each pixel in the overlapping area, a continuously changing matrix W with a value range between 0 and 1 is obtained;

[0261] Step 207: Use the weight matrix to fuse the overlapping area images. The pixel matrix of the fused image is:

[0262] I f =W*I1+(1-W)*I2

[0263] Among them, W is the weight matrix corresponding to the pixel point, I f is the pixel matrix of the fused image, I1 is the pixel matrix of the first camera image of two adjacent cameras, and I2 is the pixel matrix of the first camera image of two adjacent cameras.

[0264] In one embodiment, for the processing of X-band radar sensor data, noise points are removed from the acquired data, Gaussian filtering is used to smooth the data, and a clustering algorithm based on Euclidean distance is used to segment the point cloud into different obstacle clusters, and the minimum triangle bounding box is calculated according to the coordinates of the midpoints of the clusters. The obstacle color weight is set from 1000m to 0m, and the color weight is set by blue, yellow, orange, and red segments. The closer the obstacle is to the hull, the more the color weight tends to the red RGB value (255,0,0), and the closer the obstacle is to the edge of the panoramic image, the more the color weight tends to the blue RGB value (0,0,255). A three-dimensional coordinate system with a straight line parallel to the hull as the X-axis, a straight line perpendicular to the hull as the Y-axis, and a vertical straight line as the Z-axis is set, as well as a two-dimensional coordinate system with a straight line parallel to the hull as the X-axis and a straight line perpendicular to the hull as the Y-axis, and obtains a three-dimensional mapping matrix for the conversion of the obstacle coordinate system above the sea surface relative to the three-dimensional coordinate system of the comprehensive panoramic view, as well as a bird's-eye view relative to the surroundings of the hull, the bow, and the stern. Figure 2 Two-dimensional mapping matrix for dimensional coordinate system transformation.

[0265] The specific method for calculating the minimum triangular bounding box based on the coordinates of the cluster midpoint is:

[0266] Step 301: Select the leftmost point in the point set as the starting point P 0。 ;

[0267] Step 302: Starting from P0, traverse all other points in the order of increasing polar angles, and find the point P that is the leftmost turning direction of the line segment formed with the current point and the last edge on the previously determined convex hull boundary relative to the starting point P0. i;

[0268] Step 303: P i Add it to the convex hull and set it as the current point, and continue to find the next point;

[0269] Step 304: Repeat the above steps until returning to the starting point P0 and the convex hull is closed.

[0270] In one embodiment, for the processing of sonar sensor data, the distance from the obstacle to the sensor in the sound wave reflection data is calculated using the speed of sound and the time delay. With the sensor as the origin, a straight line parallel to the hull is set as the X-axis, a straight line perpendicular to the hull is set as the Y-axis, and a three-dimensional map coordinate system with a vertical straight line as the Z-axis is set. According to the distance data and the pixel resolution, the pixel value on the three-dimensional map corresponding to each distance value is calculated. Then, the coordinate points in the three-dimensional map coordinate system are mapped to the obstacle coordinate system part below the sea surface of the comprehensive panoramic view to obtain a three-dimensional mapping matrix, which is used to draw these pixel values ​​on the topographic map below the sea surface. Different colors or brightness on the topographic map represent different distance values.

[0271] Please see again Figure 2 In one embodiment, horizontal demarcation lines of 10m, 50m, 100m, 200m, 500m, and 1000m are added to the views of each area. The demarcation line plane and the screen image are initially at a certain angle to show a three-dimensional effect, and the demarcation line plane is divided into regions using virtual straight lines, which is convenient for ship operators to clearly grasp the environmental conditions of each area. In addition, both areas are 3D views, and the bottom of the two areas is placed with a topographic map below the sea surface drawn by sonar sensor data, which can be rotated through the touch module to view the environmental conditions of the unavailable areas. In addition, vertical demarcation lines of 10m, 40m, 60m, and 100m below the sea surface, demarcation lines of 10m, 40m, 60m, and 100m above the sea surface, and auxiliary lines for the perspective of the three-dimensional outline of the hull are added to area 6, which is convenient for ship operators to grasp the environmental conditions of the area close to the hull and the environmental conditions below the sea surface at the bottom of the ship. For area 3, only the horizontal boundary line of the nearest area is added with the obstacle as the center, so that the ship operator can grasp the detailed location of the nearest obstacle. In areas 2, 4 and 5, all or part of the ship's bird's-eye view contour auxiliary lines are added to facilitate the ship operator to use the contour auxiliary lines as a reference system to check the environmental conditions of each boundary area. Figure 3 As shown, a triangular auxiliary line with a frame is used to indicate the location of obstacles. For obstacles that are relatively close, a flashing frame is added to remind the ship operator to avoid collision.

[0272] In one embodiment, step 412: displaying the composite panoramic image includes:

[0273] The touch screen display module is divided into a main interface and 5 sub-interfaces, such as Figure 2 As shown, area 1 is the main interface, located on the left side of the screen, used to display the 3D panoramic view of the entire ship and the bottom terrain map. Area 2 is the panoramic bird's-eye view of the ship, located on the upper right of the screen, mainly used to display the panoramic images of the four directions of the front, back, left, and right of the ship. Area 3 is the obstacle tracking map, which takes the obstacle closest to the ship as the center and displays the environmental conditions of 20m around it in real time. Area 4 is the panoramic bird's-eye view of the bow, used to display the environment and obstacle conditions of the bow area. Area 5 is the panoramic bird's-eye view of the stern, used to display the environment and obstacle conditions of the stern area. Area 6 is the 3D panoramic view of the depth of the hull bottom and the bottom terrain perspective view, used to display the environmental conditions around the hull and the bottom. The display conditions of each area can be switched at will, and the processed panoramic image is transmitted to the display module at 20 to 30 frames per second for display.

[0274] Please see again Figure 3 In one embodiment, the specific method of marking obstacles is to divide the obstacles on and above the sea surface into four areas at a horizontal distance of 0-1000m from the hull. The obstacles are represented by triangular bounding boxes, and the triangular borders are set to red, orange, yellow, and blue thick borders respectively. The closer the obstacle is to the hull, the more red the triangular border color tends to be, and the closer the obstacle is to the edge of the panoramic view at 1000m, the more blue the triangular border color tends to be. The display module will display the environmental conditions within a square range of 20m around the obstacle, with the obstacle as the center. The topographic map below the sea surface is represented by red, orange, yellow, and blue color areas. The farther away from the hull, the bluer it tends to be, and the closer the distance, the redder it tends to be. When an obstacle is detected, the adaptive adjustment module will adjust the camera sensor to track the obstacle, and the obstacle position will be synchronously updated in the panoramic view. When the obstacle is within 20m of the hull, the obstacle tracking view box displayed therein will flash, alerting the ship operator to pay attention to obstacle avoidance. Figure 3 As shown, obstacles are detected between 10-100m in front of the left side of the ship, between 500-1000m on the left side, and between 200-500m behind the ship, and are displayed in the display module as red-bordered triangles, blue-bordered triangles, and yellow-bordered triangles, respectively.

[0275] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A large-scale shipboard panoramic mooring auxiliary device, characterized in that: include: X-band radar sensor, installed high on the bridge, to obtain data on the distance, direction and motion status of obstacles above the sea surface and the ship; Sonar sensor, installed on the bow or bottom of the ship, and used to obtain the distance, direction and movement status data of obstacles below the sea surface and the ship; Multiple camera sensor groups are installed at intervals at the bow, stern, and both sides of the deck of the ship, including at least a fisheye camera, a telephoto camera, a depth camera, a black and white camera, and an infrared camera, and are used to obtain image data of the surrounding environment of the ship under different light scenes; The data processing and panoramic view display platform is used to synchronize the data obtained by the X-band radar sensors, sonar sensors and camera sensor groups in the entire area or the specified area, and process and fuse the images obtained by various types of camera sensors in the camera sensor group in the same direction in the specified area, balance and adjust the images fused by the camera sensor groups in different directions in the specified area to try to synthesize a panoramic view, gradually and adaptively adjust the orientation of the camera sensor group according to the panoramic view results, and finally synthesize a panoramic view that meets the preset requirements, and process obstacle data to form upper and lower obstacle information on the sea surface and inherent hull data to draw auxiliary lines in the panoramic view, perform real-time obstacle detection, dynamically adjust the position of the camera sensor group to track obstacles, and display the comprehensive panoramic view on the display module.

2. The large-scale shipboard panoramic mooring assistance device according to claim 1, characterized in that: The data processing and panoramic view display platform includes an image acquisition module, a surface obstacle data acquisition module, an underwater obstacle data acquisition module, a device synchronization module, a scene selection module, an orientation control module, an adaptive adjustment module, an image adjustment module, an image fusion module, a panoramic view synthesis module, an obstacle data processing module, an obstacle tracking module, an auxiliary line module, an obstacle warning module, a touch module and a display module; among which, The image acquisition module is used to open the corresponding device in the camera sensor group to obtain image data according to the camera sensor device number of the user-specified area, and to package the obtained image data and the corresponding device parameters and send them to the device synchronization module; The surface obstacle data acquisition module is used to obtain the distance, direction, and motion status data of obstacles on and above the sea surface scanned by the X-band radar sensor, and package it with the corresponding device parameters and send it to the device synchronization module; The underwater obstacle data acquisition module is used to obtain the distance, direction, and motion status data of obstacles below the sea surface scanned by the sonar sensor, and package them with the corresponding device parameters and send them to the device synchronization module; The device synchronization module is used to unify the clock and synchronize the trigger signal, that is, to synchronize the threads of the surrounding environment image data and obstacle data acquired by the image acquisition module, the surface obstacle data acquisition module, and the underwater obstacle data acquisition module; The scene selection module is used to switch the device working scene by setting the scene switching switch on the device panel and schedule the opening and closing of the corresponding camera sensor; The orientation control module is used to select the field of view orientation by setting an orientation selection button on the device panel to control the opening and closing of the corresponding orientation camera sensor group, and to display the panoramic view of the corresponding orientation in the main area of ​​the image display module, and to adaptively adjust the position of the camera sensor group according to the control of the adaptive adjustment module to adjust the panoramic view to a suitable position or track obstacles; An adaptive adjustment module is used to fine-tune the position of the camera group according to the fused images of the camera group in all directions of the specified area content, and to control the azimuth movement of the camera group to focus on the obstacle when tracking the obstacle; An image adjustment module is used to perform deformation conversion processing on images of different cameras with the same orientation and synchronous processing, and transmit them to the image fusion module for image fusion, and perform color balance adjustment, brightness balance adjustment and image size adjustment on the images after fusion of multiple cameras in different orientations, and transmit them to the panoramic image synthesis module for panoramic image synthesis, and receive image zoom-in, zoom-out and rotation instructions input by the touch module to perform corresponding transformations on the images; An image fusion module is used to fuse the same-direction and same-group multi-type camera images transformed by the image adjustment module into a single single-direction image by overlapping the single-direction multiple camera images on the image pixel coordinate system using the overlap projection matrix parameters in the parameter memory; A panoramic image synthesis module is used to synthesize the images fused and adjusted by multiple cameras in the same orientation into a panoramic image in sequence according to the sensor position information; The obstacle data processing module is used to process the obstacle information data above the sea surface obtained by the X-band radar sensor, and the obstacle information data below the sea surface obtained by the sonar sensor; The obstacle tracking module is used to track obstacles identified in the panoramic image, transmit obstacle tracking data to the adaptive adjustment module, focus on the obstacles, control the camera sensor group to change the azimuth angle, and display the location details of the obstacles on the display module; Auxiliary line module, used to draw auxiliary lines of hull contour, auxiliary lines of horizontal distance in visible area, auxiliary lines of vertical distance in visible area, auxiliary lines of 3D imaging below the sea surface, and auxiliary lines of obstacle motion trajectory in the panoramic image; The obstacle warning module is used to detect obstacle information above and below the target ship and on the sea level, draw an obstacle warning logo on the panoramic view, and flash the screen for warning; A touch module, used to zoom in, zoom out, rotate, etc. the image of a specified area, transmit the operation instructions to the image adjustment module, and to change the display position of each azimuth view on the display module; The display module is used to display the processed comprehensive panoramic image view.

3. The large-scale shipboard panoramic mooring assistance device according to claim 1, characterized in that: The X-band radar sensor has a viewing angle of at least 50° up, down, left and right, can cover a low altitude of 0 to 1000m, and can rotate 360°.

4. The large-scale shipboard panoramic mooring assistance device according to claim 1, characterized in that: The camera sensor groups are arranged at intervals of up to 5m, and when the camera sensor groups are working, their maximum downward angle can cover the hull, and their maximum upward angle can cover a distance of up to 1000m on the same horizontal plane at maximum draft; The camera sensor group is installed on the vessel through a cabin, and the camera sensor group can move in and out of the cabin to be retracted into the cabin when not in use.

5. The large-scale shipboard panoramic mooring assistance device according to claim 2, characterized in that: The scene switching switch includes normal mode, low light mode and dim light mode, among which: The Normal gear position controls the camera sensor group to turn on the fisheye camera, telephoto camera, and depth camera; The low-light gear position controls the camera sensor group to turn on the fisheye camera, telephoto camera, depth camera, and black-and-white camera; The low-light gear position controls the camera sensor group to turn on the fisheye camera, telephoto camera and infrared camera.

6. The large-scale shipboard panoramic mooring assistance device according to claim 4, characterized in that: A movable mounting base is provided in the cabin, and the camera sensor group is installed on the mounting base. The mounting base includes a base fixed in the cabin, a telescopic rod is arranged on the base, a U-shaped fixing frame is arranged at the end of the telescopic rod away from the base, a rotating rod is arranged between two arms of the U-shaped fixing frame, and a camera sensor group is arranged on the rotating rod; and, The activities of the telescopic rod and the rotating rod are controlled by the data processing and panoramic view display platform.

7. A large-scale shipboard panoramic mooring assistance device according to any one of claims 1 to 6, characterized in that: It also includes a power controller for controlling the opening and closing of X-band radar sensors, sonar sensors, multiple camera sensor groups, and data processing and panoramic view display platforms.

8. A large-scale shipboard panoramic mooring assistance method, characterized in that: Using the device as claimed in any one of claims 1 to 7, and the method includes: Step 401: Start the device and obtain the internal and external parameters of the camera; Step 402: The ship operator selects a panoramic view position according to the requirements, and the device turns on the camera sensor group at the corresponding position; Step 403: The ship operator determines the current external environment scene and selects normal, weak light, and dark light working scenes, and the device turns on the corresponding camera device in the camera sensor group; Step 404: synchronously acquiring obstacle data and image data acquired by the X-band radar sensor, the sonar sensor, and each camera sensor of the designated orientation camera sensor group; Step 405: fusing images acquired by different camera sensors of the same-position camera sensor group; Step 406: uniformly adjust brightness, color, and size of the images fused by the camera sensor groups at different orientations in step 405; Step 407: stitching the images uniformly adjusted in step 406 into a panoramic view through a panoramic image synthesis module; Step 408: performing obstacle auxiliary line conversion on the obstacle data acquired by the X-band radar sensor and the sonar sensor, and synchronizing the obstacle data with the panoramic image acquired in step 407; Step 409: adding boundary auxiliary lines, hull contour auxiliary lines and obstacle auxiliary lines to the panoramic image through the auxiliary line module; Step 410: Track the obstacle in real time through the obstacle tracking module, and dynamically update it to the panoramic view with auxiliary lines obtained in step 409; Step 411: The obstacle warning module flashes an alarm on the obstacle image display area of ​​the panoramic view in step 410, and dynamically updates the panoramic view; Step 412: Displaying the comprehensive panoramic image; Step 413: Adjust the view size by touching, rotate the 3D view to view the detailed information of the corresponding orientation, or swap the position of the sub-image in the view.

9. The large vessel onboard panoramic mooring assistance method according to claim 8, characterized in that: Get the internal and external parameters of the camera, including: Step 101: Initialization, allocating storage space for camera parameters and corner points of all images, and laying out the calibration board; Step 102: Use a camera sensor to capture at least 10 images of the calibration plate at different positions and angles; Step 103: Read the image and use the Shi-Tomasi algorithm to perform corner detection; Step 104: Refine the corner point coordinates and extract the four corner points of each square on the calibration plate for each calibration plate image; Step 105: for the image from which the corner points have been successfully extracted, the coordinate values ​​of the corner points in the real world coordinate system and the sub-pixel coordinate values ​​in the image coordinate system are stored; Step 106: Calculate the internal parameters of the camera using the extracted corner point coordinates, including the focal length, principal point position, and distortion parameters of the camera; Step 107: Calculate the external parameters of the camera according to the rotation angle and translation vector of the camera; Step 108: Use the acquired internal and external parameters of the camera to obtain a camera image, and check whether the image is consistent with the real scene. If the error is too large, repeat steps 102 to 107 to re-acquire the internal and external parameters of the camera.

10. The large vessel onboard panoramic mooring assistance method according to claim 8, characterized in that: Step 405: fusing images acquired by different camera sensors of the same-position camera sensor group, including: The fisheye camera bird's-eye view is used as the base, and other camera images are superimposed on this image. During superposition, feature detection is used to find local areas in the image that can be stably detected and recognizable between different images, so as to establish the correspondence between the images, adjust the image size, and perform image fusion. Specifically: The method of fusing the image with the telephoto camera image is as follows: using the translation-invariant discrete wavelet transform to perform multi-layer filtering to form high-frequency sub-images and low-frequency sub-images of the two images; fusing the high-frequency components according to the high-frequency sub-images corresponding to the two images to form high-frequency component fusion coefficients; fusing the low-frequency components according to the low-frequency sub-images corresponding to the two images to form low-frequency component fusion coefficients; performing the translation-invariant discrete wavelet inverse transform on the high-frequency components corresponding to the high-frequency component fusion coefficients and the low-frequency components corresponding to the low-frequency component fusion coefficients to generate a fused image to achieve the effect of optical zoom; The method of fusing an image with a depth camera image is as follows: the depth region is obtained by using the pixel values ​​of adjacent pixels of the depth camera image, and then the subscript of the region contour is obtained by using the approximation method, and the ratio of the value of the subscript pixel under the contour line to the maximum value of the subscript is set as the weight value, and the weight of the pixel point in the non-contour area is set to 1 to obtain a new pixel point weight matrix; the pixel point weight matrix is ​​multiplied by the value of the corresponding point of the original image pixel matrix and then integerized to obtain an image after the original image and the depth camera image are fused. Compared with the original image, the image increases the sharpness of the edge line of the object in the image and highlights the three-dimensional sense; The method of fusing the image with the black-and-white camera image is as follows: first, the brightness component and chromaticity component of the fisheye camera image are obtained, and the brightness component and the black-and-white image are fused. The brightness component pixel matrix value and the black-and-white image pixel matrix value of the corresponding point are averaged and replaced to obtain the brightness fusion component pixel matrix; the target image is obtained by combining the brightness fusion component and the chromaticity component. Compared with the original image, the image brightness in the dark scene can be significantly improved, and the noise can be reduced at the same time; Method for fusing the image with the infrared camera image: In a dark scene, turn on the infrared fill light to obtain the infrared image; Step 407: stitching the images uniformly adjusted in step 406 into a panoramic view through a panoramic image synthesis module, including: The images uniformly adjusted in step 406 are arranged according to the image position information, the overlapping areas are merged, and then the images are subjected to image brightness balance processing and color balance processing; wherein, The fusion method used for the overlapping area of ​​the image is to set a weight coefficient for each pixel point. The weight coefficient changes with the pixel value of the point and changes continuously with the distance between each pixel point and the median line. The method used for balancing the brightness of multiple groups of images is to calculate a coefficient for each of the three RGB channels corresponding to the panorama with brightness differences after fusion, and then multiply it with the original channel value. The over-bright channel is dimmed with a coefficient less than 1, and the over-dark channel is brightened with a coefficient greater than 1, and finally a picture with adjusted brightness is formed; The method used for color balancing of each group of images is to obtain the mean of the total value of the RGB three channels in the pixel matrix for the panoramic image after the image brightness is balanced. The value of the RGB three channels of each pixel point is obtained by multiplying the original value with the mean and then dividing it by the mean of the channel of the pixel matrix.

Images