Ship water gauge measuring method and system based on unmanned aerial vehicle

Abstract

The invention relates to the technical field of water gauge measurement, in particular to a ship water gauge measurement method and system based on an unmanned aerial vehicle, and the method comprises the steps: obtaining a ship water gauge image through a camera device carried by the unmanned aerial vehicle, and carrying out the preprocessing of the ship water gauge image; inputting the preprocessed ship water gauge image into the trained water gauge measurement model to obtain a water level result measured by the water gauge; the water gauge measurement model comprises a semantic segmentation module and a water gauge measurement module, and the semantic segmentation module is used for carrying out semantic segmentation on an input ship water gauge image to form a segmented image; the water gauge measurement module performs water gauge scale reconstruction on the segmented image to obtain a water gauge measurement result; the method can effectively overcome the problems of complex background, water surface reflection, floating object interference and the like, and the water level accuracy of water gauge measurement is improved.

Description

Background Art

[0002] As a tool for measuring water surface height, the water gauge plays a key role in the shipping field. It is specifically used to indicate the depth of the ship immersed in water, that is, the draft depth. This kind of ruler or scale device has evolved into various forms according to different application scenarios and needs. Specifically, the water gauge can be classified into several types, such as the traditional and stable fixed water gauge, which is firmly fixed on the hull, moves with the ship, and directly reflects the draft changes; there are floating water gauges designed to adapt to water level fluctuations, and its scale part can rise and fall with the water surface to improve the accuracy of readings; there are also telemetry water gauges and electronic water gauges realized by modern technology, which provide more convenient and high-precision water level data through remote monitoring or electronic display technology.

[0003] However, in the daily operation of ships, the most common and basic one is the water gauge that is directly fixed on the hull. This design is simple and effective. It is not only convenient for the crew to observe and record the draft of the ship at any time, but also ensures the directness and accuracy of the data. It is an indispensable auxiliary tool for safe navigation and cargo loading management. In short, the water gauge commonly used on ships is a special ruler fixed on the hull to reflect the draft depth of the ship in real time.

[0004] The water level gauge cannot be measured under complex conditions such as local dirt and damage, low image resolution, insufficient natural light, and excessive artificial light. However, in actual field applications, the distance between the water level gauge and the shooting equipment is far, resulting in low image resolution and more interference noise. In addition, changes in the intensity of light in the field have a greater impact on the imaging effect. On sunny days, the water surface is glaring, backlit, the water body is transparent, and the reflection of the water level gauge is strong; on rainy days, the grayscale values ​​of the water level gauge and the water surface are close, and the image signal-to-noise ratio is reduced; at night, artificial light is overexposed, making it difficult for traditional image processing technology based on grayscale and edge information to accurately and effectively detect the position of the water level line. At the same time, the measurement results of occlusion conditions such as floating objects wrapped around the water level gauge sheet at the water level line cannot be effectively identified. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a ship draft gauge measurement method and system based on an unmanned aerial vehicle in view of the deficiencies in the above-mentioned prior art, which solves the problem that the draft gauge cannot accurately measure the ship's draft depth under complex conditions.

[0006] The present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for measuring a water level of a ship based on an unmanned aerial vehicle, comprising:

[0008] Acquire a ship water gauge image by using a camera device carried by the drone, and pre-process the ship water gauge image;

[0009] The preprocessed ship draft gauge image is input into the trained draft gauge measurement model to obtain the water level result of the draft gauge measurement; the draft gauge measurement model includes a semantic segmentation module and a draft gauge measurement module, the semantic segmentation module is used to perform semantic segmentation on the input ship draft gauge image to form a segmented image; the draft gauge measurement module reconstructs the draft gauge scale on the segmented image to obtain the draft gauge measurement result;

[0010] The data set used for training the water gauge measurement model includes sample images of water gauges, water surfaces, and floating objects with colored labels.

[0011] As a further improvement of the present invention, the preprocessing of the ship water gauge image specifically includes: denoising, filtering and smoothing, and image enhancement processing of the ship water gauge image.

[0012] As a further improvement of the present invention, the semantic segmentation module adopts an FCN network model or a YOLOv3-Tiny model.

[0013] As a further improvement of the present invention, the semantic segmentation module adopts an FCN network model, and the FCN network model includes a downsampling subunit, a pixel classification subunit, an upsampling subunit and an output layer;

[0014] The downsampling subunit performs feature extraction on the input preprocessed water gauge image to obtain feature maps at different levels; the upsampling subunit is used to perform feature fusion on the feature maps at different levels and adjust the size of the feature map output by the downsampling subunit to a set size; the pixel classification subunit is used to classify and predict pixels according to the feature map output by the upsampling subunit to obtain the category results of the pixels; the output layer is used to output a segmentation result map with category labels.

[0015] As a further improvement of the present invention, the pixel classification subunit adopts a Softmax classifier, which is used to perform exponential calculations on the linear prediction values ​​of all categories to obtain exponential results respectively; all exponential results are normalized to obtain the probability of each pixel corresponding to the category, and the category with the highest probability is selected as the classification result of the pixel.

[0016] As a further improvement of the present invention, the water gauge measurement module reconstructs the water gauge scale of the segmented image, specifically comprising:

[0017] A weighted directed graph is constructed based on the water ruler characters in the segmentation result graph. The water ruler characters in the segmentation result graph are used as vertices of the weighted directed graph. The edges of the graph are constructed based on the spatial relationship and semantic relationship between the characters and weights are assigned.

[0018] By finding the longest path in the weighted directed graph, the correct order and position relationship of the water gauge scales are determined to achieve water gauge reconstruction.

[0019] After the water gauge reconstruction is completed, the water surface area information in the segmentation result image is used to identify the position of the water level line on the water gauge;

[0020] The relative position of the water level line on the water gauge, combined with the scale value and proportional relationship of the water gauge, is used to calculate the actual water level height.

[0021] As a further improvement of the present invention, the edges of the graph are constructed according to the spatial relationship and semantic relationship between characters, and weights are assigned, specifically including: the weight calculation expression is:

[0022]

[0023] in,

[0024]

[0025] w is the weight from vertex i to vertex j, is the angle weight from vertex i to vertex j, is the pixel distance weight from vertex i to vertex j, θ1 is the angle between the directed line segment and the coordinate axis, d p1 is the pixel distance between the characters in the ordered character pair in the weighted directed graph, d t1 is the label distance between characters in an ordered pair of characters.

[0026] As a further improvement of the present invention, after the water level measurement model obtains the water level result measured by the water gauge, it also includes visually displaying the result.

[0027] As a further improvement of the present invention, after the water gauge measurement model obtains the water level result measured by the water gauge, it also includes real-time analysis of the measured water level data, and when the water level exceeds the water level warning line, an alarm is issued.

[0028] In a second aspect, the present invention further provides a ship water gauge measurement system based on a drone, which is used to implement the above-mentioned ship water gauge measurement method based on a drone, comprising:

[0029] The UAV image acquisition module is used to obtain an orthophotod ship water gauge image through a camera device carried by the UAV, and pre-process the ship water gauge image;

[0030] The data processing module is used to input the pre-processed ship water gauge image into the trained water gauge measurement model to obtain the water level result measured by the water gauge;

[0031] The water gauge measurement model includes a semantic segmentation submodule and a water gauge measurement submodule;

[0032] The semantic segmentation submodule is used to perform semantic segmentation on the input ship water gauge image to form a segmented image; the water gauge measurement submodule reconstructs the water gauge scale on the segmented image to obtain a water gauge measurement result;

[0033] The data set used for training the water gauge measurement model includes sample images of water gauges, water surfaces, and floating objects with colored labels.

[0034] Compared with the prior art, the present invention has at least the following beneficial effects: A ship water gauge measurement method of the present invention can perform remote monitoring through drones without interfering with the normal operation of the ship, reducing the safety risks of on-site workers. The acquired images can provide a more comprehensive perspective, which helps the model to better learn and understand the water gauge characteristics under different conditions. After a large amount of data training, the water gauge measurement model given to the deep learning model can accurately identify and measure the water gauge, reduce human errors, and improve the accuracy of water level measurement. At the same time, the present invention can effectively overcome problems such as complex background, water surface reflection and floating object interference through water gauge reconstruction, thereby improving the water level accuracy of water gauge measurement.

[0035] Furthermore, the ship water gauge reconstruction and water level intelligent monitoring method based on weighted directed graph and image semantic segmentation has achieved remarkable results in improving the accuracy and robustness of water level recognition. Future work will further optimize the semantic segmentation model and weighted directed graph construction algorithm to improve the automation and intelligence level of the system.

[0036] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of a method for measuring a ship water level based on an unmanned aerial vehicle in an embodiment of the present invention;

[0038] Figure 2 Schematic diagram of pixel distance and angle of ordered characters in an embodiment of the present invention;

[0039] Figure 3 is a schematic diagram of ordered character pairs in an embodiment of the present invention;

[0040] Figure 4 is a schematic diagram of constructing a weighted directed graph in an embodiment of the present invention;

[0041] Figure 5 It is a structural diagram of a ship water level measurement system based on an unmanned aerial vehicle in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The idea of ​​the present invention is to provide a method and system for measuring a ship water gauge based on an unmanned aerial vehicle. The measuring method specifically includes:

[0044] Acquire a ship water gauge image by using a camera device carried by the drone, and pre-process the ship water gauge image;

[0045] The preprocessed ship draft gauge image is input into the trained draft gauge measurement model to obtain the water level result of the draft gauge measurement; the draft gauge measurement model includes a semantic segmentation module and a draft gauge measurement module, the semantic segmentation module is used to perform semantic segmentation on the input ship draft gauge image to form a segmented image; the draft gauge measurement module reconstructs the draft gauge scale on the segmented image to obtain the draft gauge measurement result;

[0046] The data set used for training the water gauge measurement model includes sample images of water gauges, water surfaces, and floating objects with colored labels.

[0047] The present invention can effectively overcome the problems of complex background, water surface reflection and floating object interference through water gauge reconstruction, and improve the water level accuracy measured by the water gauge.

[0048] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments, wherein the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0049] Example 1

[0050] like Figure 1-Figure 4 As shown, this embodiment provides a ship water level measurement method based on a drone, and the specific implementation method is as follows.

[0051] The ship's water gauge image is obtained by the camera device carried by the drone, and the ship's water gauge image is preprocessed. The drone is controlled to hover near the ship, and the ship's water gauge image at various angles is obtained by the drone's built-in shooting equipment. The obtained water gauge image is uploaded to the processing center. The processing center performs denoising, filtering and smoothing, and image enhancement processing on the ship's water gauge image.

[0052] The preprocessed ship water gauge image is input into the trained water gauge measurement model to obtain the water level result measured by the water gauge.

[0053] The dataset used for training the water gauge measurement model includes sample images of water gauges, water surfaces, and floating objects with colored labels.

[0054] The data set of the draft measurement model includes several ship draft images of different historical weather conditions and different ship draft types acquired by drones. Specifically, different weather conditions include normal weather and extreme weather; normal weather includes sunny days, rainy days, etc., and extreme weather includes hurricanes, typhoons, tornadoes, thunderstorms, hail, etc. Different ship draft types include draft gauges of different shapes and sizes. For example, standard draft gauge, load line draft gauge, dynamic draft gauge, etc.

[0055] Several ship water gauge images in the data set are preprocessed images, specifically including: after obtaining the ship water gauge images through the drone, the acquired water gauge images are preprocessed, and the preprocessing includes denoising, filtering and smoothing, and image enhancement processing of the water gauge images.

[0056] Among them, the water gauge measurement model includes a semantic segmentation module and a water gauge measurement module. The semantic segmentation module is used to perform semantic segmentation on the input ship water gauge image to form a segmented image; the water gauge measurement module reconstructs the water gauge scale on the segmented image to obtain the water gauge measurement result.

[0057] The semantic segmentation module adopts an FCN network model or a YOLOv3-Tiny model. In this embodiment, the semantic segmentation module adopts an FCN network model, and the FCN network model includes a downsampling subunit, a pixel classification subunit, an upsampling subunit, and an output layer.

[0058] Specifically, the downsampling subunit performs feature extraction on the input preprocessed water level gauge image to obtain feature maps at different levels; the upsampling subunit is used to perform feature fusion on the feature maps at different levels and adjust the size of the feature map output by the downsampling subunit to a set size; the pixel classification subunit is used to classify and predict pixels according to the feature map output by the upsampling subunit to obtain pixel category results; the output layer is used to output a segmentation result map with category labels.

[0059] The pixel classification subunit uses a Softmax classifier, which is used to perform exponential calculations on the linear prediction values ​​of all categories to obtain exponential results respectively; all exponential results are normalized to obtain the probability of each pixel corresponding to the category, and the category with the highest probability is selected as the classification result of the pixel. The Softmax multi-classification layer performs exponential calculations on the probability values ​​of each input classifier. After normalization, the category represented by the maximum value is the classification result. During the training process, the loss function is calculated after determining the category to which each pixel belongs, and the output vector of the Softmax multi-classification layer and the label value of the sample data are cross-entropy processed. The model parameters are adjusted by minimizing the cross-entropy loss function so that the predicted probability distribution of the model is as close to the true label distribution as possible. This method is very effective in pixel-level classification tasks such as image segmentation, and can improve the accuracy and robustness of the model.

[0060] The water gauge measurement module reconstructs the water gauge scale of the segmented image, including:

[0061] A weighted directed graph is constructed based on the water gauge characters in the segmentation result image. The water gauge characters in the segmentation result image are used as vertices of the weighted directed graph. The edges of the graph are constructed according to the spatial relationship and semantic relationship between the characters and weights are assigned. The correct order and position relationship of the water gauge scale are determined by finding the longest path in the weighted directed graph to achieve reconstruction of the water gauge. After the reconstruction of the water gauge is completed, the position of the water level line on the water gauge is identified using the water surface area information in the segmentation result image. The relative position of the water level line on the water gauge is combined with the scale value and proportional relationship of the water gauge to calculate the actual water level height.

[0062] Specifically, there is a very strong constraint dependency between the fraction characters on the water gauge, such as Figure 2 As shown in , the adjacent water gauge quantile characters are equal in size, arranged at equal intervals, and the angles between the character lines and the vertical direction are equal. Based on this feature, a weighted directed graph is constructed. Figure 3 , Figure 4 shown.

[0063] For each detected character, define a five-tuple: z = (x, y, w, h, t), where x and y represent the coordinates of the character center point p on the image, such as Figure 4 As shown (the coordinate origin is the upper left corner of the image, horizontally to the right is the positive direction of the x-axis, and vertically downward is the positive direction of the y-axis); w, h represent the width and height of the character respectively; t∈{2,4,6,8} represents the value corresponding to the character. In this embodiment, the value corresponding to the character M is 0.

[0064] For any two characters, an ordered character pair can be formed<z1,z2> , where z1 and z2 are the five-tuples corresponding to the characters, satisfying that character z1 is above character z2.

[0065] For ordered character pairs<z1,z2> , the pixel distance d between these two characters on the image p That is, the modulus of the directed line segment p1p2 between the center points of the characters:

[0066]

[0067] Among them, θ is the angle between p1p2 and the y-axis, and the label distance between two characters is:

[0068]

[0069] Among them, t1, t2∈{0,2,4,6,8} represent the numerical values ​​corresponding to the characters.

[0070] The n detected characters with labels "2, 4, 6, 8, M" are selected as candidate characters for the water scale quantile, denoted as S, and the characters in S are paired in pairs to form Ordered character pairs As a vertex V of a weighted directed graph i .in, like Then construct a line from vertex V i Point to V j The weighted directed edge of is calculated as follows:

[0071]

[0072] in,

[0073]

[0074] w is the weight from vertex i to vertex j, is the angle weight from vertex i to vertex j, is the pixel distance weight from vertex i to vertex j, θ1 is the angle between the directed line segment and the coordinate axis, d p1 is the pixel distance between the characters in the ordered character pair in the weighted directed graph, d t1 is the label distance between characters in an ordered pair of characters.

[0075] The classic Dijkstra algorithm is used to calculate the longest path and obtain the quantile character of the water gauge.

[0076] In addition, after the water gauge measurement model obtains the water level result measured by the water gauge, it also includes a visual display of the result.

[0077] In addition, it also includes real-time analysis of the measured water level data and an alarm when the water level exceeds the water level warning line.

[0078] This method can effectively overcome problems such as complex background, water surface reflection and floating interference, and improve the accuracy of water level recognition. Within the error range of 10mm, the accuracy of water level recognition can reach more than 91.3%, which is significantly better than traditional methods and mainstream methods in the industry. At the same time, this method has a faster processing speed and can meet the needs of real-time water level monitoring.

[0079] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0080] Example 2

[0081] In another embodiment of the present invention, a ship water gauge measurement system based on a drone is provided, which can be used to implement the above-mentioned ship water gauge measurement method based on a drone, specifically as follows: Figure 5 As shown, the ship water level image recognition system of the UAV includes a UAV image acquisition module and a data processing module.

[0082] The UAV image acquisition module is used to obtain an orthophotod ship water gauge image through a camera device carried by the UAV, and pre-process the ship water gauge image;

[0083] The data processing module is used to input the pre-processed ship water gauge image into the trained water gauge measurement model to obtain the water level result measured by the water gauge;

[0084] The water gauge measurement model includes a semantic segmentation submodule and a water gauge measurement submodule;

[0085] The semantic segmentation submodule is used to perform semantic segmentation on the input ship water gauge image to form a segmented image; the water gauge measurement submodule reconstructs the water gauge scale on the segmented image to obtain a water gauge measurement result;

[0086] The data set used for training the water gauge measurement model includes sample images of water gauges, water surfaces, and floating objects with colored labels.

[0087] The drone collection module is communicatively connected with the data processing module, and the data processing module can be arranged in the drone or on the terminal device.

[0088] The terminal device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the ship water level image recognition method of the unmanned aerial vehicle.

[0089] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A method for measuring ship water level based on drone, characterized in that: include: Acquire a ship water gauge image by using a camera device carried by the drone, and pre-process the ship water gauge image; The preprocessed ship draft gauge image is input into the trained draft gauge measurement model to obtain the water level result of the draft gauge measurement; the draft gauge measurement model includes a semantic segmentation module and a draft gauge measurement module, the semantic segmentation module is used to perform semantic segmentation on the input ship draft gauge image to form a segmented image; the draft gauge measurement module reconstructs the draft gauge scale on the segmented image to obtain the draft gauge measurement result; The data set used for training the water gauge measurement model includes sample images of water gauges, water surfaces, and floating objects with colored labels.

2. The method for measuring ship water level based on unmanned aerial vehicle according to claim 1, characterized in that: The preprocessing of the ship water gauge image specifically includes: denoising, filtering and smoothing, and image enhancement processing of the ship water gauge image.

3. The method for measuring ship water level based on unmanned aerial vehicle according to claim 2 is characterized in that: The semantic segmentation module adopts the FCN network model or the YOLOv3-Tiny model.

4. The method for measuring ship water level based on unmanned aerial vehicle according to claim 3 is characterized in that: The semantic segmentation module adopts an FCN network model, which includes a downsampling subunit, a pixel classification subunit, an upsampling subunit and an output layer; The downsampling subunit performs feature extraction on the input preprocessed water gauge image to obtain feature maps at different levels; the upsampling subunit is used to perform feature fusion on the feature maps at different levels and adjust the size of the feature map output by the downsampling subunit to a set size; the pixel classification subunit is used to classify and predict pixels according to the feature map output by the upsampling subunit to obtain the category results of the pixels; the output layer is used to output a segmentation result map with category labels.

5. The method for measuring ship water level based on unmanned aerial vehicle according to claim 4 is characterized in that: The pixel classification subunit adopts a Softmax classifier, which is used to perform exponential calculations on linear prediction values ​​of all categories to obtain exponential results respectively; all exponential results are normalized to obtain the probability of each pixel corresponding to the category respectively, and the category with the highest probability is selected as the classification result of the pixel.

6. The method for measuring ship water level based on unmanned aerial vehicle according to claim 1, characterized in that: The water ruler measurement module reconstructs the water ruler scale of the segmented image, specifically including: A weighted directed graph is constructed based on the water ruler characters in the segmentation result graph. The water ruler characters in the segmentation result graph are used as vertices of the weighted directed graph. The edges of the graph are constructed based on the spatial relationship and semantic relationship between the characters and weights are assigned. By finding the longest path in the weighted directed graph, the correct order and position relationship of the water gauge scales are determined to achieve water gauge reconstruction. After the water gauge reconstruction is completed, the water surface area information in the segmentation result image is used to identify the position of the water level line on the water gauge; The relative position of the water level line on the water gauge, combined with the scale value and proportional relationship of the water gauge, is used to calculate the actual water level height.

7. The method for measuring ship water level based on unmanned aerial vehicle according to claim 6, characterized in that: The method of constructing the edges of the graph according to the spatial relationship and semantic relationship between the characters and assigning weights specifically includes: the weight calculation expression is: in, w is the weight from vertex i to vertex j, is the angle weight from vertex i to vertex j, is the pixel distance weight from vertex i to vertex j, θ1 is the angle between the directed line segment and the coordinate axis, d p1 is the pixel distance between the characters in the ordered character pair in the weighted directed graph, d t1 is the label distance between characters in an ordered pair of characters.

8. The method for measuring ship water level based on an unmanned aerial vehicle according to any one of claims 1 to 7, characterized in that: After the water level measurement model obtains the water level result measured by the water gauge, it also includes visually displaying the result.

9. The method for measuring ship water level based on an unmanned aerial vehicle according to any one of claims 1 to 7, characterized in that: After obtaining the water level result measured by the water gauge, the water gauge measurement model also includes real-time analysis of the measured water level data, and when the water level exceeds the water level warning line, an alarm is issued.

10. A ship water gauge measurement system based on an unmanned aerial vehicle, used to implement the ship water gauge measurement method based on an unmanned aerial vehicle according to any one of claims 1 to 9, characterized in that: include: The UAV image acquisition module is used to obtain an orthophotod ship water gauge image through a camera device carried by the UAV, and pre-process the ship water gauge image; The data processing module is used to input the pre-processed ship water gauge image into the trained water gauge measurement model to obtain the water level result measured by the water gauge; The water gauge measurement model includes a semantic segmentation module and a water gauge measurement module; The semantic segmentation module is used to perform semantic segmentation on the input ship water gauge image to form a segmented image; the water gauge measurement module reconstructs the water gauge scale on the segmented image to obtain a water gauge measurement result; The data set used for training the water gauge measurement model includes sample images of water gauges, water surfaces, and floating objects with colored labels.

Images