Sea ice thickness detection device and method based on image recognition
By using Jetson NANO processor and lidar module in the sea ice thickness detection device, combined with neural network and least squares method, the problem of low accuracy of sea ice flip angle measurement is solved, and the thickness recognition of incomplete flip sea ice is achieved. It has improved navigation safety and efficiency and provided important support for environmental protection.
Patent Information
- Application Number
- CN202411178459.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-08
AI Technical Summary
Existing sea ice thickness measurement methods are difficult to ensure accurate measurement of sea ice flip angle, especially when sea ice is not completely flipped, resulting in low recognition accuracy and poor data continuity.
A sea ice thickness detection device based on image recognition is adopted, combined with a Jetson NANO processor, lidar module and camera module, point cloud data is fitted through neural network algorithms and least squares method to realize real-time recognition of sea ice flip angle and thickness.
The thickness detection and identification of incompletely flipped sea ice has been achieved, navigation safety and efficiency have been improved, and important support for environmental protection and scientific research have been provided.
Smart Images

Figure CN118913117A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of ship and ocean engineering, and relates to a sea ice thickness detection device based on image recognition and a detection method thereof. Background Art
[0002] Due to the special geographical location and rich natural resources of the Arctic and Antarctic, countries around the world are actively promoting the development of polar routes. As necessary equipment for the development of Arctic and Antarctic shipping routes and the exploitation of natural resources, polar icebreakers have an important strategic position. Sea ice poses a threat to icebreaker shipping, marine operations, scientific investigations and other related activities. Among them, sea ice thickness is a key parameter for sea ice detection. Sea ice thickness detection can help crews arrange icebreaking operations reasonably, optimize resource allocation, and ensure the smooth completion of tasks.
[0003] The known documents include: A sea ice thickness identification system and sea ice thickness identification method suitable for ship ice zone navigation test CN112614177B, authorized on November 18, 2022; Ship-based video image sea ice thickness measurement method and system based on deep learning CN112700489B, authorized on April 18, 2023.
[0004] The existing sea ice thickness measurement method is only for completely overturned sea ice, that is, the overturn angle is more than 90°. However, it is difficult to ensure the complete overturning of sea ice during actual navigation, resulting in low recognition accuracy. In addition, it is difficult to ensure data continuity by only selecting completely overturned sea ice for thickness identification. Summary of the invention
[0005] In view of the problems existing in the prior art, the present invention provides a sea ice thickness detection device and method based on image recognition.
[0006] The technical problems to be solved by the present invention are:
[0007] (1) The sea ice thickness detection device provides continuous real-time parameters of sea ice thickness during the voyage of the icebreaker.
[0008] (2) In particular, it is capable of measuring the overturning angle of sea ice, including the overturning angle of sea ice that has not been completely overturned, thereby realizing the thickness detection and identification of the sea ice that has not been completely overturned.
[0009] To achieve the above purpose, the technical solution of the present invention is:
[0010] A sea ice thickness detection device based on image recognition comprises a Jetson NANO processor, a device body, and a laser radar module, a camera module and an LED module installed on the device body, wherein the Jetson NANO processor is arranged inside the device body, and the laser radar module, the camera module and the LED module are all communicatively connected with the Jetson NANO processor.
[0011] The LED module is located directly below the camera module and is lit when working to assist the camera module in shooting.
[0012] The laser radar module is used to emit a laser beam to form point cloud data.
[0013] The camera module is located directly below the laser radar module and is used to obtain continuous sea ice images P0 on the ship's side.
[0014] The continuous ship side sea ice image P0 collected by the camera module and the point cloud data measured by the lidar module are transmitted to the Jetson NANO processor through wires; the Jetson NANO processor processes the sea ice image P0 and the point cloud data to obtain the actual ice thickness L3 and snow thickness L4.
[0015] Furthermore, the Jetson NANO processor fits the plane model of the point cloud data using a least squares method to obtain a tilt angle θ of the plane model.
[0016] Furthermore, the Jetson NANO processor uses a pixel-level sea ice thickness model trained by a neural network algorithm to identify the flipped ice thickness section P1 and snow thickness section P2 in the sea ice image P0, and corresponds each pixel point of the ice thickness section P1 and the snow thickness section P2 to the coordinates in the world coordinate system to obtain the true size of the ice thickness and snow thickness; by making an inscribed circle for the ice thickness section P1 and the snow thickness section P2, the diameter of the maximum inscribed circle is obtained to obtain the width L1 of the ice thickness section and the width L2 of the snow thickness section; using the width L1 of the ice thickness section, the width L2 of the snow thickness section and the inclination angle θ of the sea ice, the actual ice thickness L3 and snow thickness L4 are obtained.
[0017] Furthermore, the camera module includes a camera.
[0018] Furthermore, the LED module includes an LED device.
[0019] Furthermore, the device body is also connected to a cable, through which power is supplied to the Jetson NANO processor, the laser radar module, the camera module and the LED module.
[0020] Furthermore, the Jetson NANO processor is connected to a host computer via Ethernet mode.
[0021] A method for detecting sea ice thickness based on image recognition comprises the following steps:
[0022] Step 1: The camera module is used to collect the sea ice image on the side of the icebreaker in real time during its voyage. The collected image is P0;
[0023] Step 2: Identify the flipped ice thickness section as P1 and the snow thickness section as P2 in the image P0 through a neural network algorithm;
[0024] Step 3: correspond each pixel point of the ice thickness section P1 and the snow thickness section P2 to the coordinates in the world coordinate system to obtain the real size of the ice thickness and snow thickness;
[0025] Step 4: By making an inscribed circle for the ice thickness section P1 and the snow thickness section P2, the diameter of the largest inscribed circle can be obtained to obtain the width L1 of the ice thickness section P1 and the width L2 of the snow thickness section P2;
[0026] Step 5: Use the laser radar module to emit a laser beam to the sea ice on the side of the ship to obtain point cloud data on the sea ice surface;
[0027] Step 6: Preprocess the acquired point cloud data;
[0028] Step 7: Use a fitting algorithm to fit the plane in the point cloud data;
[0029] Step8: Get the unit normal vector of the fitting plane;
[0030] Step 9: Get the plane tilt angle θ;
[0031] Step 10: Obtaining the actual ice thickness L3 and snow thickness L4: Using the width L1 of the ice thickness section P1, the width L2 of the snow thickness section P2 and the inclination angle θ, the actual ice thickness L3 and the snow thickness L4 are obtained. The specific calculation formula is as follows:
[0032]
[0033] Furthermore, the steps of the neural network algorithm in Step 2 are:
[0034] Step 2a: Collect sea ice images on the ship’s side: calibrate the ice thickness section and snow thickness section of each image to form a neural network model data set.
[0035] Step 2b: Divide the data of the neural network model data set into a training set and a test set, and the ratio of the training set to the test set is set to 4:1.
[0036] Step 2c: Based on the training set data, train the neural network model of the semantic segmentation algorithm.
[0037] Step 2d: Use the test set data to verify the neural network model. If the verification accuracy does not meet the requirements, adjust the parameters to continue training; if the verification accuracy meets the requirements, complete the training and form a pixel-level sea ice thickness model.
[0038] Furthermore, the step 7 of fitting the plane in the point cloud data using a fitting algorithm includes: fitting the plane model of the point cloud data using the least squares method so that the sum of the distances between the plane and all points is minimized, and the fitting plane equation is ax+by+cz+d=0.
[0039] The effects and benefits of the present invention are: through the ship-borne camera module and the lidar module, the thickness parameters of the sea ice on the side of the ship can be identified in real time, especially the thickness parameters of the sea ice that has not been completely turned over can be identified, which can not only improve navigation safety and efficiency, but also provide important support for environmental protection, scientific research and other ships, and has broad social and economic significance. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 A schematic diagram of the installation of a sea ice thickness detection device provided in an embodiment of the present invention;
[0041] Figure 2 A structural diagram of a sea ice thickness detection device provided in an embodiment of the present invention;
[0042] Figure 3 A block diagram of the internal connection relationship of the sea ice thickness detection device provided by an embodiment of the present invention;
[0043] Figure 4 A schematic diagram of sea ice thickness parameter identification for a sea ice thickness detection device provided in an embodiment of the present invention.
[0044] The following are marked in the figure:
[0045] 1-Jetson NANO processor, 2-cable, 3-device body, 4-LiDAR module, 5-camera module, 6-LED module, 7-host computer. DETAILED DESCRIPTION
[0046] The specific implementation of the present invention is described in detail below in conjunction with the technical solution (and drawings).
[0047] like Figure 1As shown, the sea ice thickness detection device is installed on the side of the icebreaker hull to collect and process the real-time data of sea ice on the side of the ship during the voyage of the icebreaker, and calculate the actual ice thickness and snow thickness.
[0048] like Figure 2 and Figure 3 As shown, a sea ice thickness detection device based on image recognition includes a Jetson NANO processor 1, a cable 2, a device body 3, and a laser radar module 4, a camera module 5 and an LED module 6 installed on the device body 3.
[0049] The Jetson NANO processor 1 is disposed inside the device body 3 , and the laser radar module 4 , the camera module 5 and the LED module 6 are all communicatively connected to the Jetson NANO processor 1 .
[0050] The camera module 5 includes a camera; the LED module 6 includes an LED device; the device body 3 is connected to the cable 2, and the Jetson NANO processor 1, the laser radar module 4, the camera module 5 and the LED module 6 are powered by the cable 2; the Jetson NANO processor 1 is connected to a host computer 7 via Ethernet mode; the LED module 6 is located directly below the camera module 5 and is lit when working to assist the camera module 5 in shooting.
[0051] The laser radar module 4 is used to emit a laser beam to form point cloud data and pre-process the acquired point cloud data; the point cloud data is transmitted to the Jetson NANO processor 1 through a wire, and the Jetson NANO processor 1 uses the least squares method to fit the plane model of the pre-processed point cloud data to obtain the fitted plane unit normal vector, and obtains the inclination angle θ of the plane model according to the unit normal vector and the unit horizontal direction vector.
[0052] The camera module 5 is located directly below the laser radar module 4, and is used to obtain continuous ship side sea ice images P0; the continuous ship side sea ice images P0 are transmitted to the Jetson NANO processor 1 through a wire, and the Jetson NANO processor 1 is trained using a neural network algorithm such as U-Net, DeepLab series, PSPNet, etc. If the verification accuracy meets the requirements, the training is completed to form a pixel-level sea ice thickness model, and the sea ice thickness model is used to identify the flipped ice thickness section P1 and snow thickness section P2 in the sea ice image P0, and each pixel point of the ice thickness section P1 and the snow thickness section P2 is matched with the coordinates in the world coordinate system to obtain the true size of the ice thickness and the snow thickness; by making an inscribed circle for the ice thickness section P1 and the snow thickness section P2, the diameter of the maximum inscribed circle is obtained, that is, the width L1 of the ice thickness section and the width L2 of the snow thickness section are obtained.
[0053] like Figure 4 As shown, the actual ice thickness L3 and snow thickness L4 are obtained by using the obtained width L1 of the ice thickness section P1 and the width L2 of the snow thickness section P2 and the inclination angle θ of the sea ice.
[0054] A sea ice thickness identification method suitable for polar ship navigation test, the specific implementation case is as follows:
[0055] Step 1: Collect sea ice images on the ship’s side. Calibrate the ice thickness section and snow thickness section of each sea ice image to form a neural network model data set.
[0056] Step 2: Divide the data of the neural network model dataset into a training set and a test set in proportion, such as setting the ratio of the training set to the test set to 4:1.
[0057] Step 3: Based on the data in the training set, a neural network model of a semantic segmentation algorithm is used for training, wherein the semantic segmentation algorithm may use some segmentation algorithms commonly used in the art, such as U-Net, DeepLab series, PSPNet, etc.
[0058] Step 4: Use the data in the test set to verify the neural network model. If the verification accuracy does not meet the requirements, adjust the parameters to continue training; if the verification accuracy meets the requirements, complete the training and form a pixel-level sea ice thickness model.
[0059] Step 5: The camera module is used to collect the sea ice image on the side of the icebreaker in real time during its voyage. The collected sea ice image is P0.
[0060] Step 6: Use the trained pixel-level sea ice thickness model to identify the flipped ice thickness section and snow thickness section in the sea ice image P0, where the ice thickness section is P1 and the snow thickness section is P2.
[0061] Step 7: Match each pixel point of the ice thickness section P1 and the snow thickness section P2 with the coordinates in the world coordinate system to obtain the real size of the ice thickness and snow thickness.
[0062] Step 8: By making an inscribed circle for the ice thickness section P1 and the snow thickness section P2 and obtaining the diameter of the largest inscribed circle, the width L1 of the ice thickness section P1 and the width L2 of the snow thickness section P2 can be obtained.
[0063] Step 9: Use the laser radar module to emit a laser beam toward the sea ice on the side of the ship to obtain point cloud data of the sea ice surface.
[0064] Step 10: Preprocess the acquired point cloud data: according to the installation angle of the laser radar, use the de-noising and filtering algorithm to remove abnormal points and noise in the point cloud data.
[0065] Step 11: Use the fitting algorithm to fit the plane in the point cloud data: Use the least squares method to fit the plane model of the point cloud data so that the sum of the distances between the plane and all points is minimized, and the fitting plane equation is ax+by+cz+d=0.
[0066] Step 12: Obtain the unit normal vector of the fitted plane: The unit normal vector n of the plane can be obtained from the coefficients of the fitting equation, which is (a, b, c), and the unit normal vector n is normalized, that is, the unit normal vector n is divided by its modulus to obtain the unit normal vector n to represent the tilt direction of the plane.
[0067] Step 13: Get the plane tilt angle: First calculate the inner product of the unit normal vector n and the unit horizontal direction vector r. The inner product formula is as follows:
[0068] n·r=n x ·r x +n y ·r y +n z ·r z ;
[0069] Among them, n x , n y , n z and r x , r y , r z are the components of the unit normal vector n and the unit horizontal direction vector r respectively.
[0070] Next, the cosine function of the tilt angle θ can be calculated by using the inner product and the length relationship. The specific calculation formula is shown as follows:
[0071]
[0072] Then, θ is solved by inverse trigonometric functions. The specific formula is shown below.
[0073]
[0074] Step 14: Obtain the actual ice thickness L3 and snow thickness L4: In a flat ice area, when an icebreaker is breaking ice, the angle between the upper surface of the sea ice and the section of the broken ice thickness is 90 degrees. Therefore, the actual ice thickness L3 and snow thickness L4 can be obtained by using the obtained sea ice inclination angle θ. The specific calculation formula is as follows:
[0075]
[0076]
[0077] The sea ice thickness detection device based on image recognition disclosed in the present invention acquires the thickness parameters of the sea ice on the ship's side in real time through the ship-borne camera module and the laser radar module, processes the image data through the Jetson NANO processor, and calculates the actual values of ice thickness and snow thickness. In view of the defect of the prior art that only completely flipped sea ice (flip angle of more than 90°) is detected, the present invention proposes the above-mentioned recognition of sea ice thickness parameters for incompletely flipped sea ice, which can not only improve navigation safety and efficiency, but also provide important support for environmental protection, scientific research and other ships, and has broad social and economic significance.
[0078] The above-mentioned implementation cases only express the implementation methods of the present invention, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.
Claims
1. A sea ice thickness detection device based on image recognition, It is characterized in that The device comprises a Jetson NANO processor (1), a device body (3), and a laser radar module (4), a camera module (5), and an LED module (6) installed on the device body (3), wherein the Jetson NANO processor (1) is arranged inside the device body (3), and the laser radar module (4), the camera module (5), and the LED module (6) are all connected to the Jetson NANO processor (1) for communication; The LED module (6) is located directly below the camera module (5) and is lit when in operation to assist the camera module (5) in taking photos; The laser radar module (4) is used to emit a laser beam to form point cloud data; The camera module (5) is located directly below the laser radar module (4) and is used to obtain continuous sea ice images P0 on the ship's side; The sea ice image P0 collected by the camera module (5) and the point cloud data measured by the laser radar module (4) are transmitted to the Jetson NANO processor (1) via a wire; the Jetson NANO processor (1) processes the sea ice image P0 and the point cloud data to obtain actual ice thickness L3 and snow thickness L4.
2. The device according to claim 1, It is characterized in that The Jetson NANO processor (1) fits the plane model of the point cloud data using the least squares method to obtain the tilt angle θ of the plane model.
3. The device according to claim 2, It is characterized in that The Jetson NANO processor (1) uses a pixel-level sea ice thickness model trained by a neural network algorithm to identify the flipped ice thickness section P1 and snow thickness section P2 in the sea ice image P0, and corresponds each pixel point of the ice thickness section P1 and the snow thickness section P2 to the coordinates in the world coordinate system to obtain the real size of the ice thickness and the snow thickness; by making an inscribed circle for the ice thickness section P1 and the snow thickness section P2, the diameter of the largest inscribed circle is obtained to obtain the width L1 of the ice thickness section and the width L2 of the snow thickness section; using the width L1 of the ice thickness section, the width L2 of the snow thickness section and the inclination angle θ of the sea ice, the actual ice thickness L3 and the snow thickness L4 are obtained.
4. The device according to claim 1, It is characterized in that The camera module (5) comprises a camera.
5. The device according to claim 1, It is characterized in that The LED module (6) comprises an LED device.
6. The device according to claim 1, It is characterized in that The device body (3) is also connected to a cable (2), and power is supplied to the Jetson NANO processor (1), the laser radar module (4), the camera module (5) and the LED module (6) through the cable (2).
7. The device according to claim 1, It is characterized in that The Jetson NANO processor (1) is communicatively connected to a host computer (7) via an Ethernet mode.
8. A method for detecting sea ice thickness based on image recognition. It is characterized in that The following steps are involved: Step 1: The camera module collects the sea ice image on the side of the icebreaker in real time during its voyage. The collected image is P0; Step 2: Identify the flipped ice thickness section as P1 and the snow thickness section as P2 in the image P0 through a neural network algorithm; Step 3: correspond each pixel point of the ice thickness section P1 and the snow thickness section P2 to the coordinates in the world coordinate system to obtain the real size of the ice thickness and snow thickness; Step 4: By making an inscribed circle for the ice thickness section P1 and the snow thickness section P2, the diameter of the largest inscribed circle can be obtained to obtain the width L1 of the ice thickness section P1 and the width L2 of the snow thickness section P2; Step 5: Use the laser radar module to emit a laser beam to the sea ice on the side of the ship to obtain point cloud data on the sea ice surface; Step 6: Preprocess the acquired point cloud data; Step 7: Use a fitting algorithm to fit the plane in the point cloud data; Step8: Get the unit normal vector of the fitting plane; Step 9: Get the plane tilt angle θ; Step 10: Obtaining the actual ice thickness L3 and snow thickness L4: Using the width L1 of the ice thickness section P1, the width L2 of the snow thickness section P2 and the inclination angle θ, the actual ice thickness L3 and the snow thickness L4 are obtained. The specific calculation formula is as follows:
9. The method according to claim 8, It is characterized in that The steps of the neural network algorithm in Step 2 are: Step 2a: Collect sea ice images on the ship’s side: calibrate the ice thickness section and snow thickness section of each sea ice image to form a neural network model data set; Step 2b: Divide the data of the neural network model data set into a training set and a test set, and the ratio of the training set to the test set is set to 4:1; Step 2c: Based on the data in the training set, a neural network model of a semantic segmentation algorithm is used for training; Step 2d: Use the data in the test set to verify the neural network model. If the verification accuracy does not meet the requirements, adjust the parameters to continue training; if the verification accuracy meets the requirements, complete the training and form a pixel-level sea ice thickness model.
10. The method according to claim 8, It is characterized in that The step 9 of obtaining the plane tilt angle θ includes: firstly calculating the inner product of the unit normal vector n and the unit horizontal direction vector r, and the inner product formula is as follows: n·r=n x ·r x +n y ·r y +n z ·r z ; Among them, n x , n y , n z and r x , r y , r z are the components of the unit normal vector n and the unit horizontal direction vector r respectively; Next, the cosine function of the tilt angle θ can be calculated by using the inner product and the length relationship. The specific calculation formula is shown as follows: Then, θ is solved by inverse trigonometric function. The specific formula is shown below:
