Unmanned ship intelligent net cage feeding method based on laser radar and camera fusion

Through the intelligent cage feeding method of integrating lidar and camera, the problems of low efficiency and poor accuracy of traditional unmanned ships are solved, efficient and precise feed delivery of marine ranches are achieved, and the stability and efficiency improvement of the breeding environment are ensured.

CN119024350AInactive Publication Date: 2024-11-26GUANGDONG OCEAN UNIVERSITY
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202411201555.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The feed delivery method of traditional unmanned ships in marine ranches is inefficient and difficult to accurately control, which can easily lead to impact cages and feed deviations, resulting in accidents and reduced breeding efficiency.

Method used

Using intelligent cage feeding method based on the fusion of lidar and camera, the precise positioning of cages and feeding position planning is achieved through synchronous data acquisition, image correction, YOLOv8 network detection, radar data processing and tolerance parameter matching fusion algorithm to achieve accurate positioning of cages and feeding position planning to ensure that there is no The man and the ship are accurately distributed.

Benefits of technology

It improves feeding accuracy, avoids cage collisions and fish population interference, stabilizes the breeding environment, and improves breeding efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119024350A_ABST
    Figure CN119024350A_ABST
Patent Text Reader

Abstract

The invention relates to an unmanned ship intelligent net cage feeding method based on laser radar and camera fusion, and the method comprises the steps: combining a laser radar and a camera, carrying out the coarse positioning through the laser radar and the camera, and carrying out the fusion of the coarse positioning results through a matching fusion algorithm based on a tolerance parameter, accurate distance and angle measurement data of net cage reference objects are found from multiple groups of objects, then a driving scheme of the unmanned ship is planned based on the data, finally, the unmanned ship is controlled to arrive at the feeding position according to the driving scheme, the unmanned ship is controlled to feed at the feeding position according to a preset control rule, net cage collision can be avoided as much as possible, and the feeding efficiency is improved. According to the method, the stability of the culture environment and the improvement of the culture efficiency are guaranteed while the operation precision is improved. In addition, due to the fact that the unmanned ship intelligent net cage feeding device based on fusion of the laser radar and the camera applies the method, the problem can be solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field related to autonomous navigation of unmanned ships, and in particular to an unmanned ship intelligent cage feeding method based on the fusion of laser radar and camera. Background Art

[0002] Marine ranching is a way of aquaculture that utilizes natural marine resources, involving large areas of open water and artificial breeding facilities. In the management of marine ranching, precise feed delivery is one of the key factors in maintaining efficient breeding and environmental sustainability. However, traditional feed delivery methods often rely on manual operations or simple mechanical equipment, which is not only inefficient but also difficult to accurately control, especially in large-scale breeding.

[0003] Traditionally, the fixed-point feeding of unmanned boats mainly relies on the GPS positioning system of the unmanned boats and the cages. However, taking into account GPS errors, the drift and swing of the cages due to ocean currents and waves, magnetic compass errors, and the vibrations of the hull and the water body, in the actual feeding process, it is difficult to ensure that the feeding boat is in the best positioning state relative to the cages by relying solely on GPS positioning. During use, it is easy to cause accidents such as the unmanned boat colliding with the cages, feeding deviation, and the unmanned boat losing control. Summary of the invention

[0004] The purpose of the present invention is to solve at least one of the deficiencies of the prior art and to provide an unmanned boat intelligent cage feeding method based on the fusion of laser radar and camera.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: Specifically, an unmanned boat intelligent cage feeding method based on the fusion of laser radar and camera is proposed, including the following: Step 110, controlling the time synchronization of laser radar and camera data acquisition, and performing image data correction on the camera to complete initialization; Step 120: Obtain video frame data collected by the camera, and obtain radar data collected by the laser radar; Step 130: input the video frame data into a pre-trained YOLOv8 network, output the two-dimensional bounding box coordinate positions of multiple cage reference objects, and calculate a first rough distance and a first rough angle from the cage reference object to the unmanned ship based on the two-dimensional bounding box coordinate positions; Step 140: Calculate based on the radar data to obtain a second rough distance and a second rough angle from the cage reference object to the unmanned boat; Step 150: obtaining a precise distance based on the first rough distance and the second rough distance, and obtaining a precise angle based on the first rough angle and the second rough angle; Step 160, repeat step 150 to obtain the precise distance and precise angle from the camera to all cage reference objects, find any two adjacent cage reference objects, and record the key points of the two adjacent cage reference objects as cage feeding reference points, and calculate the distance and angle between the unmanned boat and the cage feeding reference points; Step 170: planning a driving plan of the unmanned boat based on the distance and angle between the camera and the cage feeding reference point, and controlling the unmanned boat to reach the feeding position according to the driving plan; Step 180: Control the unmanned boat to feed materials at the feeding position according to the preset control rules.

[0006] Further, specifically, the initialization process includes: ROS encapsulation is used to synchronize the data streams of the lidar and camera using a message filtering method to ensure that the data from the lidar and camera are aligned in time before processing; after receiving the synchronization data, the callback function will be triggered to process the aligned video frame data and lidar data at the same time; Correct the video frame data collected by the camera. The key camera parameters for measuring the camera's field of view FOV, Use the camera calibration pattern generator to generate a checkerboard calibration plate of a specified size, run the camera driver on the industrial computer, and use the Camera Calibration toolkit under ROS to calibrate the camera's intrinsic parameters. Run ROS to subscribe to the camera topic and use the intrinsic parameter matrix to pre-process and correct the video frames.

[0007] Further, specifically, the step 130 includes: Create a data set of cage reference objects to configure and test the yolov8 detection algorithm: camera image acquisition to provide multiple water surface images for image target detection; use the labelme annotation tool to annotate image data and convert it into a format that meets the yolov8 input data set format, input the processed cage data into yolov8, and after training and testing, screen out the pre-trained YOLOv8 network; Import the pre-trained YOLOv8 network into the unmanned boat to perform target detection on the video frame data collected by the calibrated camera, obtain the two-dimensional frames of multiple cage reference objects in real time, and output the position coordinates, type and confidence of the two-dimensional frames of the cage reference objects; According to the position coordinates of the two-dimensional frames of the output multiple cage reference objects, the position coordinates of the two closest cage reference objects are found, and according to the position coordinates of the two two-dimensional frames, the distance and angle between the unmanned boat and the cage reference object are calculated; The distance calculation formula is as follows: Using the real-time detection frame coordinates to obtain the bounding box height of the cage reference object in the image, the focal length of the camera, the actual height of the object, and the height of the image sensor, we can calculate the distance from the object to the camera.

[0008]

[0009] in, Get the bounding box height of the grid reference object in the image for the detection box coordinates, is the actual height of the cage reference, is the height of the camera, is the focal length of the camera, Image Height is the height of the image captured by the camera, in pixels; The angle is calculated as follows:

[0010] Where W is the pixel width of the image, is the field of view of the camera, and Target Center X is the X coordinate value of the horizontal center point of the target object in the image. It is obtained through the target detection algorithm and is used to indicate the horizontal position of the target object in the image.

[0011] Further, specifically, the step 140 includes: Before acquiring radar data, the random sampling consistent RANSAC algorithm is used to fit the ground data according to the characteristics of the ground point cloud; Apply a conditional removal filter to retain only the points directly in front of the input point cloud of the radar data, set the filter angle value, set a specific angle threshold and traverse each point in the radar data, and determine whether the point is within the defined front angle range based on the calculated angle; if the angle of the point meets the condition, retain the point, otherwise remove the point, and delete all scattered data points in the input point cloud of the radar data that do not meet the clustering requirements to obtain the processed input point cloud; The point cloud of the feeding device of the unmanned ship itself is filtered out from the processed input point cloud to obtain filtered point cloud data to avoid data interference; The filtered point cloud data is input into Kd_tree to simplify the calculation, and then the clustering boxes of multiple cage references are segmented by applying the spatial clustering DBSCAN method based on density-based noise to obtain the cluster center coordinates; The distance and angle from the unmanned boat to the cage reference object are obtained based on the obtained cluster center coordinates.

[0012] Further, specifically, the step 160 includes: Matching fusion algorithm based on tolerance parameters: Set the tolerance of distance and angle to determine the match between the lidar and camera data, find the two sets of distance and angle data that best match the distance and angle of the cage reference identified by the camera from the multiple sets of distance and angle data in the lidar radar data, and output the target category, shape and color information of the two cage reference objects obtained by the camera and the precise distance and angle measurement data obtained by the lidar, that is, the precise distance and precise angle; Then, the midpoint of the edge of the cage is fitted according to the distance and angle measurement data of two adjacent cage reference objects obtained in the above manner and recorded as the cage feeding reference point, and the distance and angle between the unmanned boat and the cage feeding reference point are calculated.

[0013] Further, specifically, the step 180 includes: When the unmanned ship goes to the feeding position according to the driving plan, the real-time distance d and the real-time azimuth A between the unmanned ship and the feeding position are obtained. When d is within the distance threshold range and A is within the angle threshold range, it is judged that the feeding conditions are met, and the unmanned ship maintains the ship posture for feeding. When d is not within the distance threshold range or A is not within the angle threshold range, the PID controller continues to control the navigation of the unmanned ship until the feeding conditions are met.

[0014] Further, specifically, the distance threshold range is (1, 1.5), and the angle threshold range is (-0.5, 0.5).

[0015] The present invention also proposes an unmanned ship intelligent net cage feeding device based on the fusion of laser radar and camera, and applies the unmanned ship intelligent net cage feeding method based on the fusion of laser radar and camera, including: An unmanned ship sensing device includes a laser radar and a camera, wherein the laser radar is used to collect radar data and calculate a second rough distance and a second rough angle from a cage reference object to the unmanned ship according to the radar data, and the camera is used to collect video frame data and input the video frame data into a pre-trained YOLOv8 network, output a two-dimensional bounding box coordinate position of multiple cage reference objects, and calculate a first rough distance and a first rough angle from the cage reference object to the unmanned ship based on the two-dimensional bounding box coordinate position; a data fusion unit, configured to obtain a precise distance based on the first rough distance and the second rough distance, and to obtain a precise angle based on the first rough angle and the second rough angle; The unmanned boat control unit is used to plan the driving plan of the unmanned boat based on the distance and angle between the camera and the cage feeding reference point, control the unmanned boat to reach the feeding position according to the driving plan, and control the unmanned boat to feed at the feeding position according to preset control rules.

[0016] Furthermore, specifically, the camera is a monocular camera, and the camera is installed at the front of the unmanned boat to ensure that its optical axis is parallel to the ground, and the laser radar is installed above the camera so that the line connecting the optical center of the laser radar and the optical center of the camera is perpendicular to the ground; at the same time, under the premise of ensuring the normal operation of the laser radar and the monocular camera, the vertical distance between the laser radar and the monocular camera is minimized.

[0017] Furthermore, specifically, the scanning data of the monocular camera and lidar are transmitted to the Jetson Nano main control board via a USB cable. The main control board uses the Linux operating system and is equipped with ROS robot control software. After data processing, ship attitude estimation, and heading and speed adjustment calculations, the RS232 serial port outputs ship action instructions to the ship-borne controller, which uses the motor drive module to perform PWM speed regulation on the ship propeller, thereby achieving precise positioning and attitude maintenance between the unmanned ship and the cage reference object.

[0018] The beneficial effects of the present invention are: The present invention proposes an unmanned boat intelligent cage feeding method based on the fusion of laser radar and camera. By combining the use of laser radar and camera, coarse positioning is performed by the laser radar and camera respectively, and then the results of the coarse positioning of the two are fused through a matching fusion algorithm based on tolerance parameters, so as to realize accurate distance and angle measurement data of cage reference objects from multiple groups of objects, and then the driving plan of the unmanned boat is planned based on this. Finally, the unmanned boat is controlled to arrive at the feeding position according to the driving plan, and then the unmanned boat is controlled to feed at the feeding position according to preset control rules. It can avoid collision with cages as much as possible and reduce interference with fish schools. This method improves the operating accuracy while also ensuring the stability of the aquaculture environment and the improvement of aquaculture efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features of the present disclosure will become more obvious by describing in detail the embodiments shown in the accompanying drawings. The same reference numerals in the accompanying drawings of the present disclosure represent the same or similar sampling monitoring points. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure. For those skilled in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work. In the accompanying drawings: Figure 1 It is a structural principle diagram of the unmanned boat intelligent cage feeding device based on the fusion of laser radar and camera proposed in the present invention; Figure 2 is a diagram of the installation location of unmanned ship equipment in one embodiment; Figure 3 is a schematic diagram of the installation of a cage reference object in one embodiment; Figure 4 is a diagram of camera and lidar ranging and angle measurement in one embodiment; Figure 5 is a schematic diagram of a real-time visualization interface of an image in video frame data in one embodiment; Figure 6 is a visualization diagram of laser radar ranging and angle measurement in one embodiment; Figure 7 is a schematic diagram of simultaneous visualization of a laser radar and a camera in one embodiment; Figure 8 It is a flow chart of the unmanned boat intelligent cage feeding method based on the fusion of laser radar and camera proposed in the present invention; Fig. 9 It is a schematic diagram of the control rules for controlling the unmanned boat to feed at the feeding position. DETAILED DESCRIPTION

[0020] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other without conflict. The same reference numerals used throughout the drawings indicate the same or similar parts.

[0021] Example 8, reference Figure 8 The present invention proposes an unmanned boat intelligent cage feeding method based on the fusion of laser radar and camera, which includes the following: Step 110, controlling the time synchronization of laser radar and camera data acquisition, and performing image data correction on the camera to complete initialization; Step 120: Obtain video frame data collected by the camera, and obtain radar data collected by the laser radar; Step 130: input the video frame data into a pre-trained YOLOv8 network, output the two-dimensional bounding box coordinate positions of multiple cage reference objects, and calculate a first rough distance and a first rough angle from the cage reference object to the unmanned ship based on the two-dimensional bounding box coordinate positions; Step 140: Calculate based on the radar data to obtain a second rough distance and a second rough angle from the cage reference object to the unmanned boat; Step 150: obtaining a precise distance based on the first rough distance and the second rough distance, and obtaining a precise angle based on the first rough angle and the second rough angle; Step 160, repeat step 150 to obtain the precise distance and precise angle from the camera to all cage reference objects, find any two adjacent cage reference objects, and record the key points of the two adjacent cage reference objects as cage feeding reference points, and calculate the distance and angle between the unmanned boat and the cage feeding reference points; Step 170: planning a driving plan of the unmanned boat based on the distance and angle between the camera and the cage feeding reference point, and controlling the unmanned boat to reach the feeding position according to the driving plan; Step 180: Control the unmanned boat to feed materials at the feeding position according to the preset control rules.

[0022] In this embodiment 1, by combining the use of laser radar and camera, coarse positioning is performed by laser radar and camera respectively, and then the results of the coarse positioning of the two are fused through a matching fusion algorithm based on tolerance parameters, so as to achieve accurate distance and angle measurement data of the cage reference object from multiple groups of objects, and then plan the driving plan of the unmanned boat based on this. Finally, the unmanned boat is controlled to arrive at the feeding position according to the driving plan, and then the unmanned boat is controlled to feed at the feeding position according to the preset control rules. It can avoid collision with the cage as much as possible and reduce interference with the fish school. This method not only improves the operating accuracy, but also ensures the stability of the breeding environment and improves the breeding efficiency.

[0023] As a preferred embodiment of the present invention, specifically, the initialization process includes: ROS encapsulation is used to synchronize the data streams of the lidar and camera using a message filtering method to ensure that the data from the lidar and camera are aligned in time before processing; after receiving the synchronization data, the callback function will be triggered to process the aligned video frame data and lidar data at the same time; Correct the video frame data collected by the camera. The key camera parameters for measuring the camera's field of view FOV, Use the camera calibration pattern generator to generate a checkerboard calibration plate of a specified size, run the camera driver on the industrial computer, and use the Camera Calibration toolkit under ROS to calibrate the camera's intrinsic parameters. Run ROS to subscribe to the camera topic, and use the intrinsic parameter matrix to pre-process and correct the video frames to eliminate distortion and other effects and improve image quality.

[0024] As a preferred embodiment of the present invention, specifically, step 130 includes: Create a data set of cage reference objects to configure and test the yolov8 detection algorithm: camera image acquisition to provide multiple water surface images for image target detection; use the labelme annotation tool to annotate image data and convert it into a format that meets the yolov8 input data set format, input the processed cage data into yolov8, and after training and testing, screen out the pre-trained YOLOv8 network; Import the pre-trained YOLOv8 network into the unmanned boat to perform target detection on the video frame data collected by the calibrated camera, obtain the two-dimensional frames of multiple cage reference objects in real time, and output the position coordinates, type and confidence of the two-dimensional frames of the cage reference objects; According to the position coordinates of the two-dimensional frames of the output multiple cage reference objects, the position coordinates of the two closest cage reference objects are found, and according to the position coordinates of the two two-dimensional frames, the distance and angle between the unmanned boat and the cage reference object are calculated; The distance calculation formula is as follows: Using the real-time detection frame coordinates to obtain the bounding box height of the cage reference object in the image, the focal length of the camera, the actual height of the object, and the height of the image sensor, we can calculate the distance from the object to the camera.

[0025]

[0026] in, Get the bounding box height of the grid reference object in the image for the detection box coordinates, is the actual height of the cage reference, is the height of the camera, is the focal length of the camera, Image Height is the height of the image captured by the camera, in pixels; The angle is calculated as follows:

[0027] W is the pixel width of the image, is the field of view of the camera, and Target Center X is the X coordinate value of the horizontal center point of the target object in the image. It is obtained through the target detection algorithm and is used to indicate the horizontal position of the target object in the image.

[0028] As a preferred embodiment of the present invention, specifically, step 140 includes: Before acquiring radar data, the random sampling consistent RANSAC algorithm is used to fit the ground data according to the characteristics of the ground point cloud; Apply a conditional removal filter to retain only the points directly in front of the input point cloud of the radar data, set the filter angle value, set a specific angle threshold and traverse each point in the radar data, and determine whether the point is within the defined front angle range based on the calculated angle; if the angle of the point meets the condition, retain the point, otherwise remove the point, and delete all scattered data points in the input point cloud of the radar data that do not meet the clustering requirements to obtain the processed input point cloud; The point cloud of the feeding device of the unmanned ship itself is filtered out from the processed input point cloud to obtain filtered point cloud data to avoid data interference; The filtered point cloud data is input into Kd_tree to simplify the calculation, and then the clustering boxes of multiple cage references are segmented by applying the spatial clustering DBSCAN method based on density-based noise to obtain the cluster center coordinates; The distance and angle from the unmanned boat to the cage reference object are obtained based on the obtained cluster center coordinates.

[0029] In this preferred embodiment, the laser radar scans the surrounding environment and performs point cloud processing to measure the distance and angle data of surrounding objects.

[0030] (1) Ground point cloud filtering: According to the characteristics of the ground point cloud, the random sampling consensus (RANSAC) algorithm is used to fit the ground data.

[0031] (2) Angle conditional filtering: Apply a conditional removal filter to retain only the points directly in front of the LiDAR, set the filter angle value, set a specific angle threshold and traverse each point in the cloud, and determine whether the point is within the defined front angle range based on the calculated angle. If the angle of the point meets the condition, the point is retained, otherwise, the point is removed. At the same time, all scattered data points in the input point cloud that do not meet the clustering requirements are deleted.

[0032] (3) Filter out the point cloud of the unmanned ship’s own feeding device to avoid data interference.

[0033] (4) Point cloud clustering: The filtered point cloud data is input into Kd_tree to simplify the calculation, which can save the time required for clustering and meet the requirements of real-time monitoring. Then, the density-based spatial clustering of noise applications (DBSCAN) method is used to segment the clustering boxes of multiple surrounding objects and obtain the coordinates of the cluster center.

[0034] (5) Based on the obtained cluster center coordinates, the distance and angle between the laser radar and the surrounding objects are obtained.

[0035] As a preferred embodiment of the present invention, specifically, step 160 includes: Matching fusion algorithm based on tolerance parameters: Set the tolerance of distance and angle to determine the match between the lidar and camera data, find the two sets of distance and angle data that best match the distance and angle of the cage reference identified by the camera from the multiple sets of distance and angle data in the lidar radar data, and output the target category, shape and color information of the two cage reference objects obtained by the camera and the precise distance and angle measurement data obtained by the lidar, that is, the precise distance and precise angle; Then, the midpoint of the edge of the cage is fitted according to the distance and angle measurement data of two adjacent cage reference objects obtained in the above manner and recorded as the cage feeding reference point, and the distance and angle between the unmanned boat and the cage feeding reference point are calculated.

[0036] Here is a concrete example of how to determine the exact distance as well as the exact angle: Camera data: Assume that the camera recognizes two cage references, A and B.

[0037] The camera estimates that cage A is at an angle of 10 degrees relative to the camera and is 10 meters away (this is just a rough estimate).

[0038] The camera estimates that cage B is at an angle of 20 degrees relative to the camera and at a distance of 12 meters (again, a rough estimate).

[0039] LiDAR data: The LiDAR scans multiple distance and angle data points.

[0040] We need to find the two sets of data that best match cages A and B identified by the camera.

[0041] Assuming that a data point matching cage A is found, the radar measures its precise distance to be 9.8 meters and its precise angle to be 10.2 degrees.

[0042] For cage B, the radar measured its precise distance to be 11.9 meters and the precise angle to be 19.8 degrees.

[0043] Data matching and output: We set a tolerance, for example, the distance tolerance is 0.5 meters and the angle tolerance is 2 degrees.

[0044] For cage A, the distance 10 meters estimated by the camera differs from the distance 9.8 meters measured by the radar by 0.2 meters (within the tolerance) and the angle differs by 0.2 degrees (also within the tolerance), so this is considered a matching set of data.

[0045] For cage B, similarly, the distance 12 m estimated by the camera and the distance 11.9 m measured by the radar differed by 0.1 m and the angle by 0.2 degrees, which were also considered a match.

[0046] Final output: Cage A: target category (assumed to be “cage”), shape (assumed to be “rectangle”), color (assumed to be “blue”), precise distance 9.8 meters, precise angle 10.2 degrees.

[0047] Cage B: target category “cage”, shape “rectangle”, color “blue”, precise distance 11.9 meters, precise angle 19.8 degrees.

[0048] Through this process, we combined the camera and lidar data to obtain more precise location information (distance and angle) of the cage reference objects, as well as their other properties (such as object category, shape and color).

[0049] In this preferred embodiment, the data fusion unit receives data, runs a fusion algorithm, analyzes angles and distances, and generates heading data.

[0050] (1) Run the first ROS node: Receive the distance and angle between the camera and the cage reference object calculated by the camera in real time. Visualize the distance and angle data in real time, and mark the box to display the category, confidence, and distance and angle values.

[0051] (2) Run the second ROS node: Receive the distances and angles of multiple surrounding objects obtained by the lidar in real time, and visualize the distance and angle data in real time, with boxes showing the ID, distance and angle values, and cluster center coordinates.

[0052] (3) Run the third ROS node: Implement a matching fusion algorithm based on tolerance parameters: Set the distance and angle tolerance to determine the match between the lidar and camera data, find the two sets of distance and angle data that best match the cage reference objects identified by the camera from the multiple sets of distance and angle data from the lidar, and output the target category, shape, color and other information of the two cage reference objects obtained by the camera and the precise distance and angle measurement data obtained by the lidar.

[0053] Then, based on the distance and angle measurement data of two adjacent cage reference objects (placed every 45 degrees) obtained in the above steps, the midpoint of the cage edge is fitted and recorded as the cage feeding reference point to prevent the unmanned boat from accidentally hitting the cage when feeding. The distance and angle of the cage feeding reference point are calculated.

[0054] (4) The main control unit transmits the positioning information to the external device through the communication unit to realize the visualization of data images and other information. Figure 4 , Figure 5 , Figure 6 as well as Figure 7 shown.

[0055] (5) Run the fourth ROS node: adjust the ship's heading according to the azimuth of the cage feeding reference point, that is, the direction from the current heading of the ship to the feeding reference point. Adjust the speed of the unmanned ship and other information according to the distance from the unmanned ship to the feeding reference point. If the target is far away, the speed can be increased to reach the target faster, while if the target is nearby, the speed should be reduced to ensure safe maneuvering and stopping distance. This generates heading data.

[0056] Reference Fig. 9 As a preferred embodiment of the present invention, specifically, step 180 includes: When the unmanned ship goes to the feeding position according to the driving plan, the real-time distance d and the real-time azimuth A between the unmanned ship and the feeding position are obtained. When d is within the distance threshold range and A is within the angle threshold range, it is judged that the feeding conditions are met, and the unmanned ship maintains the ship posture for feeding. When d is not within the distance threshold range or A is not within the angle threshold range, the PID controller continues to control the navigation of the unmanned ship until the feeding conditions are met.

[0057] In this preferred embodiment, the unmanned ship sails to the designated cage feeding reference point according to the heading determined by the algorithm. The trajectory of the ship is continuously updated according to the real-time environmental data and the coordinates of the feeding reference point. Until the bow of the unmanned ship is facing the feeding reference point (the azimuth is within the threshold range), and the distance of the cage feeding reference point relative to the unmanned ship is within the preset range. A feedback mechanism is established to monitor the position and heading of the ship using sensors. The system should compare the actual trajectory with the planned route and make necessary adjustments to correct any deviations. After arriving at the feeding position, the feed extractor is started to extract the feed, and the feeding system releases the feed according to the preset program.

[0058] As a preferred implementation manner of the present invention, specifically, the distance threshold range is (1, 1.5), and the angle threshold range is (-0.5, 0.5).

[0059] Reference Figure 1 , Figure 2 as well as Figure 3 The present invention also proposes an unmanned ship intelligent net cage feeding device based on the fusion of laser radar and camera, and applies the unmanned ship intelligent net cage feeding method based on the fusion of laser radar and camera, including: An unmanned ship sensing device includes a laser radar and a camera, wherein the laser radar is used to collect radar data and calculate a second rough distance and a second rough angle from a cage reference object to the unmanned ship according to the radar data, and the camera is used to collect video frame data and input the video frame data into a pre-trained YOLOv8 network, output a two-dimensional bounding box coordinate position of multiple cage reference objects, and calculate a first rough distance and a first rough angle from the cage reference object to the unmanned ship based on the two-dimensional bounding box coordinate position; a data fusion unit, configured to obtain a precise distance based on the first rough distance and the second rough distance, and to obtain a precise angle based on the first rough angle and the second rough angle; The unmanned boat control unit is used to plan the driving plan of the unmanned boat based on the distance and angle between the camera and the cage feeding reference point, control the unmanned boat to reach the feeding position according to the driving plan, and control the unmanned boat to feed at the feeding position according to preset control rules.

[0060] As a preferred embodiment of the present invention, specifically, the camera is a monocular camera, and the camera is installed at the front of the unmanned boat to ensure that its optical axis is parallel to the ground, and the laser radar is installed above the camera so that the line connecting the optical center of the laser radar and the optical center of the camera is perpendicular to the ground; at the same time, under the premise of ensuring the normal operation of the laser radar and the monocular camera, the vertical distance between the laser radar and the monocular camera is minimized.

[0061] As a preferred embodiment of the present invention, specifically, the scanning data of the monocular camera and the laser radar are transmitted to the Jetson Nano main control board through a USB cable. The main control board uses a Linux operating system and is equipped with ROS robot control software. After data processing, ship attitude estimation, and heading and speed adjustment calculation, the RS232 serial port outputs ship action instructions to the ship-borne controller, which performs PWM speed regulation on the ship propeller through a motor drive module, thereby achieving precise positioning and attitude maintenance between the unmanned ship and the cage reference object.

[0062] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of software functional modules.

[0063] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or system that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0064] Although the description of the present invention has been quite detailed and specifically describes several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but should be regarded as providing a broad possible interpretation of these claims in view of the prior art by reference to the appended claims, thereby effectively covering the intended scope of the present invention. In addition, the above description of the present invention is based on the embodiments foreseeable by the inventor, and its purpose is to provide a useful description, and those non-substantial changes to the present invention that have not yet been foreseen may still represent equivalent changes to the present invention.

[0065] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, it should belong to the protection scope of the present invention. Within the protection scope of the present invention, its technical scheme and / or implementation method can have various modifications and changes.

Claims

1. An unmanned boat intelligent cage feeding method based on the fusion of laser radar and camera, characterized in that: Includes the following: Step 110, controlling the time synchronization of laser radar and camera data acquisition, and performing image data correction on the camera to complete initialization; Step 120: Obtain video frame data collected by the camera, and obtain radar data collected by the laser radar; Step 130: input the video frame data into a pre-trained YOLOv8 network, output the two-dimensional bounding box coordinate positions of multiple cage reference objects, and calculate a first rough distance and a first rough angle from the cage reference object to the unmanned ship based on the two-dimensional bounding box coordinate positions; Step 140: Calculate based on the radar data to obtain a second rough distance and a second rough angle from the cage reference object to the unmanned boat; Step 150: obtaining a precise distance based on the first rough distance and the second rough distance, and obtaining a precise angle based on the first rough angle and the second rough angle; Step 160, repeat step 150 to obtain the precise distance and precise angle from the camera to all cage reference objects, find any two adjacent cage reference objects, and record the key points of the two adjacent cage reference objects as cage feeding reference points, and calculate the distance and angle between the unmanned boat and the cage feeding reference points; Step 170: planning a driving plan of the unmanned boat based on the distance and angle between the camera and the cage feeding reference point, and controlling the unmanned boat to reach the feeding position according to the driving plan; Step 180: Control the unmanned boat to feed materials at the feeding position according to the preset control rules.

2. The unmanned boat intelligent cage feeding method based on laser radar and camera fusion according to claim 1 is characterized in that: Specifically, the initialization process includes: Using ROS encapsulation, the data streams of the lidar and camera are synchronized using a message filtering method to ensure that the data from the lidar and camera are aligned in time before processing; After receiving the synchronization data, the callback function will be triggered to process the aligned video frame data and radar data at the same time; Correct the video frame data collected by the camera. The key camera parameters for measuring the camera's field of view FOV, Use the camera calibration pattern generator to generate a checkerboard calibration plate of a specified size, run the camera driver on the industrial computer, and use the Camera Calibration toolkit under ROS to calibrate the camera's intrinsic parameters. Run ROS to subscribe to the camera topic and use the intrinsic parameter matrix to pre-process and correct the video frames.

3. The unmanned boat intelligent cage feeding method based on laser radar and camera fusion according to claim 2 is characterized in that: Specifically, the step 130 includes: Create a data set of cage reference objects to configure and test the yolov8 detection algorithm: camera image acquisition to provide multiple water surface images for image target detection; Use the labelme annotation tool to annotate the image data and convert it into a format that meets the yolov8 input data set. Input the processed cage data into yolov8. After training and testing, the pre-trained YOLOv8 network is screened. Import the pre-trained YOLOv8 network into the unmanned boat to perform target detection on the video frame data collected by the calibrated camera, obtain the two-dimensional frames of multiple cage reference objects in real time, and output the position coordinates, type and confidence of the two-dimensional frames of the cage reference objects; According to the position coordinates of the two-dimensional frames of the output multiple cage reference objects, the position coordinates of the two closest cage reference objects are found, and according to the position coordinates of the two two-dimensional frames, the distance and angle between the unmanned boat and the cage reference object are calculated; The distance calculation formula is as follows: the height of the bounding box of the cage reference object in the image is obtained by using the real-time detection frame coordinates, the focal length of the camera, the actual height of the object and the height of the image sensor, and then the distance from the object to the camera is calculated. in, Get the bounding box height of the grid reference object in the image for the detection box coordinates, is the actual height of the cage reference, is the height of the camera, is the focal length of the camera, Image Height is the height of the image captured by the camera, in pixels; The angle is calculated as follows: Where W is the pixel width of the image, is the field of view of the camera, and Target Center X is the X coordinate value of the horizontal center point of the target object in the image.

4. The unmanned boat intelligent cage feeding method based on laser radar and camera fusion according to claim 1 is characterized in that: Specifically, the step 140 includes: Before acquiring radar data, the random sampling consistent RANSAC algorithm is used to fit the ground data according to the characteristics of the ground point cloud; Apply a conditional removal filter to retain only the points directly in front of the input point cloud of the radar data, set the filter angle value, set a specific angle threshold and traverse each point in the radar data, and determine whether the point is within the defined front angle range based on the calculated angle; if the angle of the point meets the condition, retain the point, otherwise remove the point, and delete all scattered data points in the input point cloud of the radar data that do not meet the clustering requirements to obtain the processed input point cloud; The point cloud of the feeding device of the unmanned ship itself is filtered out from the processed input point cloud to obtain filtered point cloud data to avoid data interference; The filtered point cloud data is input into Kd_tree to simplify the calculation, and then the clustering boxes of multiple cage references are segmented by applying the spatial clustering DBSCAN method based on density-based noise to obtain the cluster center coordinates; The distance and angle from the unmanned boat to the cage reference object are obtained based on the obtained cluster center coordinates.

5. The unmanned boat intelligent cage feeding method based on laser radar and camera fusion according to claim 4 is characterized in that: Specifically, the step 160 includes: Matching fusion algorithm based on tolerance parameters: Set the tolerance of distance and angle to determine the match between the lidar and camera data, find the two sets of distance and angle data that best match the distance and angle of the cage reference identified by the camera from the multiple sets of distance and angle data in the lidar radar data, and output the target category, shape and color information of the two cage reference objects obtained by the camera and the precise distance and angle measurement data obtained by the lidar, that is, the precise distance and precise angle; Then, the midpoint of the edge of the cage is fitted according to the distance and angle measurement data of two adjacent cage reference objects obtained in the above manner and recorded as the cage feeding reference point, and the distance and angle between the unmanned boat and the cage feeding reference point are calculated.

6. The unmanned boat intelligent cage feeding method based on laser radar and camera fusion according to claim 1 is characterized in that: Specifically, the step 180 includes: When the unmanned ship goes to the feeding position according to the driving plan, the real-time distance d and the real-time azimuth A between the unmanned ship and the feeding position are obtained. When d is within the distance threshold range and A is within the angle threshold range, it is judged that the feeding conditions are met, and the unmanned ship maintains the ship posture for feeding. When d is not within the distance threshold range or A is not within the angle threshold range, the PID controller continues to control the navigation of the unmanned ship until the feeding conditions are met.

7. The unmanned boat intelligent cage feeding method based on laser radar and camera fusion according to claim 6 is characterized in that: Specifically, the distance threshold range is (1, 1.5), and the angle threshold range is (-0.5, 0.5).

8. Unmanned boat intelligent cage feeding device based on laser radar and camera fusion, characterized in that: The unmanned boat intelligent cage feeding method based on laser radar and camera fusion as described in any one of claims 1 to 7 above is applied, comprising: An unmanned ship sensing device includes a laser radar and a camera, wherein the laser radar is used to collect radar data and calculate a second rough distance and a second rough angle from a cage reference object to the unmanned ship according to the radar data, and the camera is used to collect video frame data and input the video frame data into a pre-trained YOLOv8 network, output a two-dimensional bounding box coordinate position of multiple cage reference objects, and calculate a first rough distance and a first rough angle from the cage reference object to the unmanned ship based on the two-dimensional bounding box coordinate position; a data fusion unit, configured to obtain a precise distance based on the first rough distance and the second rough distance, and to obtain a precise angle based on the first rough angle and the second rough angle; The unmanned boat control unit is used to plan the driving plan of the unmanned boat based on the distance and angle between the camera and the cage feeding reference point, control the unmanned boat to reach the feeding position according to the driving plan, and control the unmanned boat to feed at the feeding position according to preset control rules.

9. The unmanned boat intelligent cage feeding device based on laser radar and camera fusion according to claim 8 is characterized in that: Specifically, the camera is a monocular camera, and the camera is installed at the front of the unmanned boat to ensure that its optical axis is parallel to the ground, and the laser radar is installed above the camera so that the line connecting the optical center of the laser radar and the optical center of the camera is perpendicular to the ground; at the same time, under the premise of ensuring the normal operation of the laser radar and the monocular camera, the vertical distance between the laser radar and the monocular camera is minimized.

10. The unmanned boat intelligent cage feeding device based on laser radar and camera fusion according to claim 9 is characterized in that: Specifically, the scanning data of the monocular camera and lidar are transmitted to the Jetson Nano main control board via a USB cable. The main control board uses the Linux operating system and is equipped with ROS robot control software. After data processing, ship attitude estimation, and heading and speed adjustment calculations, the RS232 serial port outputs ship action instructions to the ship-borne controller, which uses the motor drive module to perform PWM speed regulation on the ship's propeller, thereby achieving precise positioning and attitude maintenance between the unmanned ship and the cage reference object.

Citation Information

Patent Citations

  • Unmanned ship water surface target detection, identification and positioning method based on monocular camera and lidar information fusion

    CN109444911A

  • River boundary detection and tracking method and system for unmanned ship

    CN114332647A

  • Ship identification method and device based on laser radar and AIS

    CN115308762A

  • Joint target detection system and detection method on unmanned surface ship

    CN116540235A

  • Control method, device and equipment of unmanned feeding equipment and storage medium

    CN118177129A