Autonomous resupply method of water surface unmanned vehicle based on fusion of visual and laser radar information

Abstract

This invention relates to an autonomous resupply method for unmanned surface vessels (USVs) based on the fusion of visual and lidar information, belonging to the field of autonomous resupply technology for USVs. This invention utilizes the rich features of visual image information, using visual detection results as prior information to fuse visual positioning information with lidar point cloud spatial information to obtain accurate docking positioning information. Simultaneously, it combines target tracking and prediction search boxes for target fusion detection, shortening detection time. This invention, through a fusion detection and tracking system of the USV's camera and lidar, overcomes the shortcomings of cameras being susceptible to light and weather conditions and having inaccurate spatial information. It provides a maritime resupply strategy for USVs, achieving autonomous detection, tracking, and precise positioning of the resupply cone during maritime docking resupply, effectively improving the resupply decision-making capability of USVs.

Description

Technical Field

[0001] This invention relates to an autonomous resupply method for unmanned surface vessels based on the fusion of visual and lidar information, belonging to the field of autonomous resupply technology for unmanned surface vessels. Background Technology

[0002] The inevitable trend of unmanned equipment replacing manned equipment in missions makes unmanned surface vessels (USVs) play a crucial role in future maritime transport and resupply systems. With the increasingly widespread application of USVs, research on autonomous resupply at sea for USVs by berthing at large replenishment vessels is of great significance. This is crucial to reduce the frequency of USVs entering and leaving ports for resupply during missions, decrease their dependence on fixed bases, and avoid problems such as long resupply times, low efficiency, and operational inconvenience. In the process of automated USV resupply, the active mutual identification and positioning between the USV and the resupply device is a prerequisite for accurately controlling the USV's entry into the resupply position and ensuring precise docking between the resupply device and the target fuel tank. Currently, commonly used environmental perception and positioning devices in the marine field include GPS, marine radar, lidar, and cameras. GPS and marine radar offer high long-range perception accuracy but lower short-range accuracy. LiDAR and cameras are suitable for medium- and short-range perception and positioning, meeting the detection range and accuracy requirements for autonomous USV resupply at sea, and can serve as auxiliary perception and positioning methods in the process of automated USV resupply.

[0003] Currently, unmanned docking and resupply technology mainly focuses on ground vehicles and drones. While research on manned vessels is more extensive, research on unmanned vessel docking and resupply is less common. The challenge of unmanned docking and resupply technology lies in autonomous docking navigation during the resupply process, which currently relies primarily on vision-based navigation. Camera images offer rich feature information and high image resolution and angle measurement accuracy, but are easily limited by lighting and weather conditions, and struggle to acquire precise spatial information. Unmanned surface vessels (USVs) are significantly affected by wind, waves, and currents at sea, and single sensors have limited adaptability in complex weather environments, exhibiting problems such as limited information, insufficient redundancy, and low accuracy, failing to meet the precise perception requirements of autonomous resupply scenarios for USVs. While lidar images lack sufficient feature information, they offer high spatial resolution and accuracy. Based on these issues, this technical field urgently needs to address how to obtain a USV maritime resupply method based on the fusion of vision and lidar information technology to improve the autonomous docking and resupply decision-making capabilities of USVs. Summary of the Invention

[0004] The purpose of this invention is to solve the technical problem of how to obtain a maritime replenishment method for unmanned surface vessels based on visual and lidar information fusion technology, thereby improving the autonomous berthing and replenishment decision-making capability of unmanned surface vessels.

[0005] To address the aforementioned issues, this invention provides an autonomous resupply method for unmanned surface vessels based on the fusion of visual and lidar information. Utilizing the rich features of visual image information, the method uses visual detection results as prior information to fuse visual positioning information with lidar point cloud spatial information to obtain accurate docking positioning information. Simultaneously, it combines target tracking prediction search boxes for target fusion detection, thereby shortening the detection time.

[0006] Preferably, it includes the following steps:

[0007] Step 1: The unmanned surface vessel sends a resupply request. After receiving the resupply request through certain communication means, the large supply ship patrols and waits in the designated area at a certain speed and course.

[0008] Step 2: The unmanned surface vessel (USV) obtains the absolute position information of the large supply ship through its onboard GPS and navigation radar. Relying on its onboard autonomous navigation control system, it reaches the designated area, namely the rendezvous area between the USV and the large supply ship, with a certain planning and control strategy. At the same time, while continuously approaching the large supply ship, it determines whether the USV is currently on the port or starboard side of the large supply ship by the angle between the large supply ship's course and the latest point of the USV's movement. It then controls the USV to approach the large supply ship from the starboard side.

[0009] Step 3: After the unmanned surface vessel arrives at the rendezvous area, it enters the resupply docking phase. The large supply ship lowers the supply cone on the starboard side, and at the same time, the unmanned surface vessel's camera and lidar system are activated. The unmanned surface vessel's navigation mode changes from GPS and navigation radar mode to vision and lidar mode.

[0010] Step 4: During the resupply docking phase, the unmanned surface vessel's camera and lidar continuously capture and scan the resupply cone sleeve at the end of the resupply cone tube, performing target fusion detection and tracking. The camera images are used to determine the type of target objects appearing in the camera's field of view and their position and attitude information. At the same time, the 3D point cloud information scanned by the lidar is mapped onto the camera's 2D plane. The point cloud images are filtered according to the target type and its position information, and then image matching is performed to obtain the position and attitude information of the target objects after the radar scan. The position and attitude information of the target objects obtained by the camera and lidar sensors, as well as the target object type information, are fused and output.

[0011] Step 5: Based on the fusion output of the camera and lidar, the surface unmanned surface vessel is guided to approach the resupply cone opening with a certain trajectory tracking strategy and reach the resupply distance within a specified time. After stabilization, it maintains a following and tracking attitude at a certain speed and heading. The surface unmanned surface vessel extends the resupply plug and, based on the fusion output, guides the resupply plug to insert into the resupply cone to complete the docking.

[0012] Step 6: After the surface unmanned vessel enters the resupply phase, it maintains a companion navigation attitude with the large resupply ship, then opens the resupply valve to transfer the resupply materials to the surface unmanned vessel. After the resupply is completed, the resupply valve is closed.

[0013] Step 7: After resupply is completed, the surface unmanned surface vessel (SUV) switches its navigation mode from vision and lidar mode back to GPS and navigation radar mode. Under the control of the SUV's autonomous navigation control system, it slowly decelerates to separate the resupply plug from the resupply cone. After the connection is broken, it leaves the rendezvous area via GPS and navigation radar to continue performing its designated mission.

[0014] Preferably, the supply cone in step 3 is L-shaped, extends from the starboard side of the large supply ship, and has a conical supply cone sleeve at the stern.

[0015] Preferably, the supply docking stage in step 4 includes target fusion detection and target tracking. That is, after the camera and lidar successfully detect the supply cone, the image tracking algorithm is used to track the supply cone and the target fusion detection of the supply cone is performed in the predicted search box, thereby shortening the detection time.

[0016] Preferably, the target fusion detection in step 4 includes a timing control module, a spatial correction module, a target matching module, and a target fusion detection module. The timing control module primarily controls the timing of matching between visual and radar images at different time steps, synchronizing timestamps from different sensors through a central processing unit. The spatial correction module transforms the spatial feature information of the target in the camera and lidar coordinate systems to a unified coordinate system for spatiotemporal registration. The spatial correction module designs the transformation between the real-world 3D coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system, defining the real-world coordinate system as the lidar coordinate system. The target matching module calculates the intersection-union ratio (IOU) between the center circle of the point family after lidar clustering and the rectangle detected by the visual image under a unified temporal and spatial domain, determining whether they are the same target based on the IOU value. The target fusion detection module, based on the target type, target position, and attitude information acquired by the camera and lidar, performs multi-source heterogeneous target decision-level fusion using evidence theory methods to obtain a consistent interpretation and description of the target.

[0017] Preferably, in step 4, the timing control module synchronizes the timestamps of different sensors. Since the sampling frequency of the LiDAR is about 10Hz and the sampling frequency of the camera is about 30FPS, the sampling frequencies of each sensor are different, making it difficult to achieve true time synchronization. Based on the timestamp synchronization, the LiDAR data with the lower sampling frequency is selected as the benchmark, and a fixed minimum time interval is set to obtain the image data with the minimum time interval for each frame of LiDAR data acquisition.

[0018] Preferably, in step 4, the spatial correction module defines the real-world coordinate system as the lidar coordinate system. Let the coordinates of a point in the lidar coordinate system be Q(X,Y,Z). After translation and rotation, the coordinates of the point in the camera coordinate system are obtained as Q′(X′,Y′,Z′). Q′(X′,Y′,Z′) is projected onto the image coordinate system to obtain q(x,y), as shown in equation (1). q(x,y) is projected onto the pixel coordinate system to obtain q′(x′,y′), as shown in equation (2). The target position and angle information of the camera in the target fusion module are obtained through equations (3) and (4).

[0019]

[0020]

[0021]

[0022]

[0023] Where f is the camera focal length, (u0, v0) are the coordinates of the image center point in the pixel coordinate system, and s x s y This represents the pixel value per millimeter along both axes of the pixel coordinate system; R is the actual radius of the known supply cone, and r is the radius of the supply cone in the pixel coordinate system, which is obtained by target fusion detection from the camera and LiDAR. To compensate for the lateral deflection angle between the cone sleeve and the unmanned surface vessel.

[0024] Preferably, in step 4, the cross-union ratio (CUNR) is used to determine whether the targets are the same; if the CUNR is ≥ 0.8, they are the same target, and if the CUNR is < 0.8, they are different targets.

[0025] Preferably, when using a camera and a lidar for target fusion detection and target tracking in step 4, multiple possible states should be considered, including the initial state, fusion capture state, target tracking state, and target re-tracking state.

[0026] Preferably, in step 4, the initial state is entered after the surface unmanned surface vessel's camera and lidar system are turned on. The camera and lidar begin to acquire images, and target matching is performed after spatiotemporal correction. If the matching is successful, the vessel enters the fusion capture state. If the matching fails but the visual image exists, the surface unmanned surface vessel continues to move towards the target detected in the two-dimensional image, and a second matching is performed during the process. If the matching fails and the visual image does not exist, the detection threshold is lowered and a second detection is performed, and a second matching is performed during the process. After a successful matching, the vessel enters the fusion capture state, outputs the target type, target position, and attitude information, and simultaneously enters the target tracking state. After entering the target tracking state, the target search box is calculated using the target tracking algorithm, and target fusion detection is performed in the target search box. If the fusion detection is successful or the visual image detection is successful, target tracking continues. If the detection fails, the vessel enters the target re-tracking state. After entering the target re-tracking state, the search window is enlarged, and target fusion detection is performed according to the operation steps in the target tracking state. If the detection is successful, the vessel returns to the target tracking state. If the detection fails, the vessel returns to the initial state and begins the next process.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] This invention addresses the shortcomings of using a single sensor in complex weather conditions by providing a method for autonomous resupply and docking of unmanned surface vessels (USVs) based on the fusion of visual and lidar information. This method has a wide range of applications, primarily for USVs to autonomously berth at large supply vessels and perform docking and resupply maneuvers.

[0029] Compared with existing technologies that rely on manual operation or a single camera for maritime docking and resupply, this invention uses a fusion detection and tracking system combining the camera and lidar of the unmanned surface vessel (USV) to overcome the shortcomings of cameras being susceptible to light and weather conditions and having inaccurate spatial information. It provides a maritime resupply strategy for USVs, enabling autonomous detection, tracking, and precise positioning of the resupply cone during the docking and resupply process, effectively improving the resupply decision-making capabilities of USVs. Attached Figure Description

[0030] Figure 1 This diagram illustrates a method for determining whether an unmanned surface vessel is located on the port or starboard side of a large supply ship under GPS and navigation radar modes.

[0031] Figure 2 A schematic diagram of the supply cone and cone sleeve for a large supply ship.

[0032] Figure 3 This is a schematic diagram of the components for target fusion detection.

[0033] Figure 4 This is a flowchart of the target fusion detection process using camera and lidar navigation modes.

[0034] Figure 5 This is a schematic diagram of the target fusion detection and tracking status. Detailed Implementation

[0035] To make the present invention more apparent and understandable, preferred embodiments are described in detail below with reference to the accompanying drawings:

[0036] like Figure 1-5 As shown, the technical solution adopted by this invention is to provide an autonomous resupply and docking method for unmanned surface vessels based on the fusion of visual and lidar information, comprising the following steps:

[0037] Step 1: The unmanned surface vessel sends a resupply request, and after the large supply ship receives the resupply request through certain communication means, it patrols and waits in the designated area at a certain speed and course.

[0038] Step 2: The unmanned surface vessel (USV) obtains the absolute position information of the large supply ship through its onboard GPS and navigation radar. Relying on its onboard autonomous navigation control system, it reaches the designated area, namely the rendezvous area between the USV and the large supply ship, with a certain planning and control strategy. At the same time, while continuously approaching the large supply ship, the USV determines whether it is currently on the port or starboard side of the large supply ship by using the angle between the large supply ship's course (calculated from the positions of two consecutive points on the large supply ship measured by GPS and navigation radar) and the latest point of movement of the USV to the latest point of movement of the large supply ship. The USV is then controlled to approach the large supply ship from the starboard side. If the angle is less than 180°, the USV is on the port side of the large supply ship; if the angle is greater than 180°, the USV is on the starboard side of the large supply ship.

[0039] Step 3: After the unmanned surface vessel arrives at the rendezvous area, it enters the resupply docking phase. The large supply ship lowers the supply cone on the starboard side, and at the same time, the unmanned surface vessel's camera and lidar system are activated. The unmanned surface vessel's navigation mode changes from GPS and navigation radar mode to vision and lidar mode.

[0040] Step 4: During the resupply docking phase, the unmanned surface vessel's camera and lidar continuously capture and scan the resupply cone, performing target fusion detection and tracking. The camera images are used to determine the type of target objects appearing within the camera's field of view and their position and attitude information. Simultaneously, the 3D point cloud information scanned by the lidar is mapped onto the camera's 2D plane. The point cloud images are filtered according to the target type and their position information, and then image matching is performed to obtain the position and attitude information of the target objects after the radar scan. The position and attitude information of the target objects obtained by the camera and lidar sensors, as well as the target object type information, are fused and output.

[0041] Step 5: During the resupply docking phase, the surface unmanned surface vessel (USV) is guided to approach the resupply cone opening with a certain trajectory tracking strategy based on the fusion output results of the camera and lidar. Within a specified time, it reaches a distance of about 10 meters from the resupply cone opening. After stabilizing, it maintains a tracking attitude with a certain speed and heading. The USV extends the resupply plug, and through the fusion output results, guides the resupply plug to insert into the resupply cone to complete the docking.

[0042] Step 6: After the surface unmanned vessel enters the resupply phase, it maintains a companion navigation attitude with the large resupply ship, then opens the resupply valve to deliver fuel and other supplies to the surface unmanned vessel. After the resupply is completed, the resupply valve is closed.

[0043] Step 7: After resupply is completed, the surface unmanned surface vessel (SUV) switches its navigation mode from vision and lidar mode back to GPS and navigation radar mode. Under the control of the SUV's autonomous navigation control system, it slowly decelerates to separate the resupply plug from the resupply cone. After the connection is broken, it leaves the rendezvous area via GPS and navigation radar to continue performing its designated mission.

[0044] The supply cone in step 3 above is L-shaped and extends from the starboard side of the large supply ship. The stern of the supply cone has a conical supply cone sleeve with a diameter of about 2 meters.

[0045] The supply docking stage in step 4 above includes target fusion detection and target tracking. That is, after the camera and lidar successfully detect the supply cone, the image tracking algorithm is used to track the supply cone and perform supply cone target fusion detection in the predicted search box to shorten the detection time.

[0046] The target fusion detection in step 4 above includes a timing control module, a spatial correction module, a target matching module, and a target fusion detection module. The timing control module primarily controls the timing of matching between visual and radar images at different time steps. It synchronizes the timestamps of different sensors through a central processing unit. However, since the LiDAR sampling frequency is approximately 10Hz and the camera sampling frequency is approximately 30fps, true time synchronization is difficult to achieve due to the different sampling frequencies of the sensors. Therefore, based on timestamp synchronization, LiDAR data with a lower sampling frequency is selected as the benchmark, and a fixed minimum time interval is set to acquire image data with the smallest time interval between LiDAR data acquisitions for each frame. The spatial correction module transforms the spatial feature information of the target in the camera and LiDAR coordinate systems to a unified coordinate system for spatiotemporal registration. The spatial correction module designs the transformation between the real-world 3D coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system, defining the real-world coordinate system as the LiDAR coordinate system. The coordinates of a point in the lidar coordinate system are Q(X,Y,Z). After translation and rotation, the coordinates of the point in the camera coordinate system are Q′(X′,Y′,Z′). Q′(X′,Y′,Z′) is projected onto the image coordinate system to obtain q(x,y) (as shown in Equation (1)). q(x,y) is projected onto the pixel coordinate system to obtain q′(x′,y′) (as shown in Equation (2)). The target matching module calculates the intersection-union ratio (IOU) of the center circle of the point cluster after lidar clustering and the rectangle detected by the visual image under the unified time and spatial domain. It judges whether they are the same target based on the size of the IOU. If the IOU is ≥0.8, they are the same target. If the IOU is <0.8, they are different targets. The target fusion module performs multi-source heterogeneous target decision-level fusion based on the target type, target position and attitude information obtained by the camera and lidar through the evidence theory method to obtain the target consistency interpretation and description. The target position and angle information of the camera are obtained through Equations (3) and (4).

[0047]

[0048]

[0049] Where f is the camera focal length, (u0, v0) are the coordinates of the image center point in the pixel coordinate system, and s x s y This represents the pixel value per millimeter along both axes of the pixel coordinate system.

[0050]

[0051]

[0052] Where R is the known actual radius of the supply cone, and r is the radius of the supply cone in the pixel coordinate system (obtained by target fusion detection from camera and LiDAR). To compensate for the lateral deflection angle between the cone sleeve and the unmanned surface vessel.

[0053] In step 4 above, when using a camera and LiDAR for target fusion detection and tracking, several possible states should be considered, including the initial state, fusion acquisition state, target tracking state, and target re-tracking state: After the unmanned surface vessel's camera and LiDAR system are turned on, it enters the initial state, where the camera and LiDAR begin acquiring images. After spatiotemporal correction, target matching is performed. If the matching is successful, it enters the fusion acquisition state. If the matching fails but the visual image exists, the unmanned surface vessel continues to move towards the target detected in the 2D image, performing a second matching during the process. If the matching fails and the visual image does not exist, the detection threshold is lowered for a second detection. The process involves a second matching step during the initial measurement. Upon successful matching, the system enters a fusion capture state, outputting target type, target position, and pose information, and simultaneously enters a target tracking state. In the target tracking state, a target search box is calculated using a target tracking algorithm. Target fusion detection is then performed within the search box. If fusion detection or visual image detection is successful, target tracking continues. If detection fails, the system enters a re-tracking state. In the re-tracking state, the search window is enlarged, and target fusion detection is performed according to the steps outlined in the target tracking state. If detection is successful, the system returns to the target tracking state; otherwise, it returns to the initial state and begins the next step.

[0054] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any form or substance. It should be noted that those skilled in the art can make various improvements and additions without departing from the present invention, and these improvements and additions should also be considered within the scope of protection of the present invention. Any modifications, alterations, and equivalent changes made by those skilled in the art based on the above-disclosed technical content without departing from the spirit and scope of the present invention are equivalent embodiments of the present invention. Furthermore, any modifications, alterations, and evolutions made to the above embodiments based on the essential technology of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. An autonomous resupply method for a water surface unmanned vehicle based on fusion of visual and laser radar information, characterized in that, The visual image information is rich in features, and the visual detection result is used as prior information to fuse the visual positioning information and the laser radar point cloud spatial information to obtain accurate docking positioning information. The method comprises the following steps: Step 1: The water surface unmanned vehicle sends a supply request, and the large supply ship receives the supply request through a certain communication means, and then cruises at a certain speed and heading in the specified area to wait; Step 2: The water surface unmanned vehicle obtains the absolute position information of the large supply ship through the on-board GPS and navigation radar, and reaches the specified area by relying on the on-board autonomous navigation control system and a certain planning and control strategy, that is, the water surface unmanned vehicle and the large supply ship meet in the specified area, and in the process of continuously approaching the large supply ship, the water surface unmanned vehicle is located on the left or right side of the large supply ship by judging the angle between the heading of the large supply ship and the line from the latest point of the water surface unmanned vehicle to the latest point of the large supply ship, and the water surface unmanned vehicle is controlled to approach from the right side of the large supply ship; Step 3: After the water surface unmanned vehicle reaches the meeting area, it enters the supply docking stage, the large supply ship lowers the supply cone pipe on the right side, and the camera and laser radar system of the water surface unmanned vehicle are turned on, and the navigation mode of the water surface unmanned vehicle is changed from GPS and navigation radar mode to visual and laser radar mode; Step 4: In the supply docking stage, the camera and laser radar of the water surface unmanned vehicle continuously shoot and scan the supply cone sleeve at the end of the supply cone pipe, and target fusion detection and tracking are performed, the type and position and posture information of the target object appearing in the camera visual angle are judged through the image shot by the camera, the three-dimensional point cloud information scanned by the laser radar is mapped to the two-dimensional plane of the camera, the point cloud image is screened according to the target type and position information, and then image matching is performed to obtain the position and posture information of the target object scanned by the radar, and the position and posture information of the target object obtained by the camera and laser radar sensors and the target object type information are fused and output; Step 5: The water surface unmanned vehicle guides the water surface unmanned vehicle to approach the supply cone sleeve opening according to the fusion output result of the camera and laser radar, and reaches the supply distance within a specified time, and then keeps a following tracking posture at a certain speed and heading, and the water surface unmanned vehicle extends the supply plug, and the supply plug is inserted into the supply cone sleeve through the fusion output result, and the docking is completed; Step 6: After the water surface unmanned vehicle enters the supply stage, it still keeps a following navigation posture with the large supply ship, then opens the supply valve to deliver the supply materials to the water surface unmanned vehicle, and closes the supply valve after the supply is completed; Step 7: After the supply is completed, the navigation mode of the water surface unmanned vehicle is switched from the visual and laser radar mode to the GPS and navigation radar mode, and the water surface unmanned vehicle slows down slowly under the control of the autonomous navigation control system, so as to realize the separation of the supply plug and the supply cone sleeve, and then the water surface unmanned vehicle drives away from the meeting area through the GPS and navigation radar, and continues to execute the specified task.

2. The autonomous resupply method for a water surface unmanned vehicle based on fusion of visual and laser radar information according to claim 1, characterized in that, The supply cone pipe in step 3 is L-shaped and extends from the right side of the large supply ship, and the stern of the supply cone pipe is provided with a conical supply cone sleeve.

3. The autonomous resupply method for a water surface unmanned vehicle based on fusion of visual and laser radar information according to claim 1, characterized in that, The replenishment docking phase in step 4 includes target fusion detection and target tracking, that is, after the camera and the laser radar successfully detect the replenishment cone, the image tracking algorithm is used to track the replenishment cone, and the target fusion detection is performed in the predicted search box, so as to shorten the detection time consumption.

4. The autonomous resupply method for a water surface unmanned vehicle based on fusion of visual and laser radar information according to claim 1, characterized in that, The target fusion detection in step 4 includes a time sequence control module, a space correction module, a target matching module and a target fusion detection module; the time sequence control module mainly completes matching time control of visual and radar images at different time steps, and synchronizes time stamps of different sensors through a central processing unit; the space correction module converts spatial feature information of a target in a camera coordinate system and a laser radar coordinate system to a unified coordinate system for space-time registration, and designs conversion between a real world three-dimensional coordinate system, a camera coordinate system, an image coordinate system and a pixel coordinate system, and sets the real world coordinate system as the laser radar coordinate system; The target matching module calculates an intersection-over-union (IOU) of a center circle of a point family after laser radar clustering and a rectangular frame detected by a visual image in a unified time domain and space domain, and judges whether they are the same target according to the size of the intersection-over-union; The target fusion detection module obtains target type, target position and attitude information through a camera and a laser radar, and performs multi-source heterogeneous target decision-level fusion through an evidence theory method to obtain target consistency interpretation and description.

5. The autonomous resupply method for a water surface unmanned vehicle based on fusion of visual and laser radar information according to claim 4, characterized in that, The time sequence control module synchronizes time stamps of different sensors in step 4, and since the sampling frequency of the laser radar is about 10HZ and the sampling frequency of the camera is about 30FPS, the sampling frequencies of the sensors are different, so it is difficult to realize real-time synchronization, and on the basis of time stamp synchronization, the laser radar data with a smaller sampling frequency is selected as a reference, a fixed minimum time interval is set, and image data with the smallest time interval of each frame of laser radar collected data is obtained.

6. The autonomous resupply method for a water surface unmanned vehicle based on fusion of visual and laser radar information according to claim 5, characterized in that, In step 4, the spatial correction module defines the real-world coordinate system as the lidar coordinate system. Let the coordinates of a point in the lidar coordinate system be... After translation and rotation, the coordinates of this point in the camera coordinate system are obtained as follows: ,Will Projecting onto the image coordinate system yields As shown in equation (1), Projecting onto the pixel coordinate system yields As shown in equation (2); Camera target position and angle information in the target fusion module is obtained through formula (3) and formula (4); （1） （2） （3） （4） wherein, is the camera focal length, is the coordinate of the image coordinate center point in the pixel coordinate system, , represents the unit millimeter pixel value in the two-axis direction of the pixel coordinate system; is the known actual radius of the supply cone, is the radius of the supply cone in the pixel coordinate system, and the radius is obtained by camera and laser radar target fusion detection, is the lateral angle between the supply cone and the water surface unmanned ship.

7. The autonomous resupply method for a water surface unmanned vehicle based on fusion of visual and laser radar information according to claim 6, characterized in that, In step 4, whether the targets are the same target is judged according to the size of the intersection-over-union; if the intersection-over-union is greater than or equal to 0.8, they are the same target, and if the intersection-over-union is less than 0.8, they are different targets.

8. The autonomous resupply method for a water surface unmanned vehicle based on fusion of visual and laser radar information according to claim 7, characterized in that, In step 4, when the camera and the laser radar are used for target fusion detection and target tracking, multiple possible states should be considered, including an initial state, a fusion capture state, a target tracking state and a target re-tracking state.

9. The autonomous resupply method for a water surface unmanned vehicle based on fusion of visual and laser radar information according to claim 8, characterized in that, In step 4, after the camera and the laser radar system of the surface unmanned ship is turned on, the initial state is entered, the camera and the laser radar start to collect images, target matching is performed after space-time correction, if the matching is successful, the fusion capture state is entered, if the matching is unsuccessful and the visual image exists, the surface unmanned ship continues to move towards the direction of the two-dimensional image detection target, and secondary matching is performed in the process, if the matching is unsuccessful and the visual image does not exist, the detection threshold is reduced for secondary detection, and secondary matching is performed in the process; after the matching is successful, the fusion capture state is entered, and the target type, target position and attitude information are output, and the target tracking state is entered. After entering the target tracking state, a target search frame is calculated by a target tracking algorithm, and target fusion detection is performed in the target search frame. If the fusion detection is successful or the visual image detection is successful, the target tracking is continued. If the detection fails, the target re-tracking state is entered. After entering the target re-tracking state, the search window is enlarged, and target fusion detection is performed according to the operation steps in the target tracking state. If the detection is successful, the target tracking state is returned to. If the detection fails, the initial state is returned to, and the next flow is started.

Images