Unmanned aerial vehicle-based ship water gauge image recognition method and system

Abstract

The invention relates to the technical field of water gauge weighing, in particular to a ship water gauge image recognition method and system based on an unmanned aerial vehicle, and the method comprises the steps: obtaining a ship water gauge image through the unmanned aerial vehicle; inputting the ship water gauge image into the trained image recognition model to obtain a water gauge recognition result; the image recognition model adopts a convolutional neural network model; a data set adopted during training comprises a plurality of ship water gauge images of different weather conditions and different ship water gauge types. According to the method, the text in the water gauge image can be accurately identified.

Description

Background Art

[0002] A draft gauge is a scale or graduated device used to measure the height of the water surface. On a ship, it specifically refers to a mark used to indicate the draft of the ship. Depending on the use environment and needs, draft gauges can be divided into many types, including but not limited to fixed draft gauges, floating draft gauges, telemetered draft gauges, and electronic draft gauges. However, on ships, draft gauges that are fixed to the hull are usually used.

[0003] The traditional method of identifying the water gauge of a ship is manual observation. However, manual observation requires taking a small boat close to the ship for detection. This method not only requires too much manpower and financial resources, but is also affected by subjective and objective factors, resulting in inaccurate water gauge identification. When the water surface fluctuates greatly, not only will the detection error increase again, but it may also threaten the lives of the inspectors. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a ship water gauge image recognition method and system based on a drone in view of the deficiencies in the above-mentioned prior art, thereby solving the problem of inaccurate water gauge recognition.

[0005] The objective of the present invention is achieved by the following technical solutions: In a first aspect, the present invention provides a method for recognizing a ship water gauge image based on an unmanned aerial vehicle, comprising: Obtain ship water gauge images through drones; The ship draft gauge image is input into a trained image recognition model to obtain a draft gauge recognition result; the image recognition model adopts a convolutional neural network model; the data set used in training the image recognition model includes a number of ship draft gauge images under different weather conditions and different ship draft gauge types. As a further improvement of the present invention, after the ship water gauge image is obtained by the drone, the obtained water gauge image is preprocessed, and the preprocessing includes tilt correction, denoising, grayscale and binarization processing of the water gauge image.

[0006] As a further improvement of the present invention, after the ship water gauge image is preprocessed, it also includes an image segmentation processing step, and the image segmentation processing includes: taking the water surface, hull and water gauge mark area in the ship water gauge image in the image as the area of ​​interest, and extracting the area of ​​interest.

[0007] As a further improvement of the present invention, the convolutional neural network model adopts a YOLOv5 model, a YOLOv10 model or a FasterRCNN model.

[0008] As a further improvement of the present invention, the image recognition model is a YOLOv5 model, and the YOLOv5 model includes a backbone network module, a head network module and an image recognition module; The backbone network module is used to extract features from the region of interest to obtain a feature map; the head network module is used to use an attention or spatial attention mechanism to adaptively weight the feature map to enhance the feature representation capability; the image recognition module is used to perform character recognition based on the feature map after feature enhancement.

[0009] As a further improvement of the present invention, the backbone network module adopts the EfficientNet network architecture, and the EfficientNet network architecture is used to perform feature extraction based on deep separable convolution and residual connection.

[0010] As a further improvement of the present invention, the head network module adopts the SENet attention mechanism, which is used to perform global average pooling on the input feature map to obtain a compressed feature vector; after the feature vector passes through the fully connected layer, the weight of each channel is learned, and a weighted feature map is obtained according to the weight of each channel and the input feature map.

[0011] As a further improvement of the present invention, the image recognition module adopts a PSENet detection algorithm, which is used to generate a text area candidate frame based on a weighted feature map, and to identify and detect the text area candidate frame through non-maximum suppression to obtain a final water gauge recognition result.

[0012] As a further improvement of the present invention, the optimization algorithm used in training the image recognition model is the gradient optimization algorithm AdamW.

[0013] In a second aspect, the present invention further provides a ship water gauge image recognition system based on a drone, which is used to implement the above-mentioned ship water gauge image recognition method based on a drone, comprising: UAV acquisition module, used to obtain ship water gauge images through UAV; A data processing module is used to input the ship water gauge image into a trained image recognition model to obtain a water gauge recognition result; The image recognition model adopts a convolutional neural network model; the data set used in training the image recognition model includes a number of ship water gauge images under different weather conditions and different ship water gauge types.

[0014] The beneficial effects of the present invention are as follows: a ship water gauge image recognition method based on a drone of the present invention can hover in the air and take images of the ship water gauge, thereby avoiding the risk of direct contact between personnel and the water surface and high-altitude operations, and significantly improving the safety of the operation. The high maneuverability and flexibility of the drone enable it to quickly reach the designated location and take images, greatly improving work efficiency. At the same time, compared with the traditional method, the cost of drone operation is lower, which helps to save manpower and material resources. When training the image recognition model, several ship water gauge images containing different weather conditions (such as sunny days, rainy days, foggy days, etc.) and different ship water gauge types (such as water gauges of different sizes and shapes) are used as data sets. This diversity can ensure that the model can maintain a high recognition accuracy in different environments and improve the generalization ability of the model. The high-definition images obtained by the drone and the well-trained image recognition model can greatly improve the accuracy of ship water gauge recognition. This helps to reduce manual errors and improve the accuracy of weight measurement. Since the training data set contains a variety of weather conditions and ship water gauge types, the trained image recognition model has strong adaptability. Whether in sunny or bad weather, it can maintain high recognition accuracy and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 It is a schematic flow chart of a method for ship water gauge image recognition based on a drone in an embodiment of the present invention; Figure 2 Schematic diagram of a neural network structure based on YOLOv5 in an embodiment of the present invention; Figure 3 It is a schematic diagram of the structure of a ship water gauge image recognition system based on an unmanned aerial vehicle in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose and technical solution of the present invention clearer and easier to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0018] Terminology explanation: FasterRCNN: Fast Regional Convolutional Neural Network, a deep learning model for object detection. EfficientNet: A convolutional neural network architecture designed to improve network performance while reducing computational resource consumption; SENet: (Squeeze-and-Excitation Network) A revolutionary deep convolutional neural network architecture designed to improve the model's ability to model image features; PSENet: A text detection technology based on the progressive scale expansion algorithm, which is particularly suitable for solving complex text detection problems in OCR (Optical Character Recognition), such as the detection of arbitrary-shaped text and adjacent text instances; MBConv: (Mobile Inverted Bottleneck Convolution) is a convolutional neural network (CNN) building block that combines the Depthwise Separable Convolution and Squeeze-and-Excitation (SE) modules.

[0019] OpenCV: An open source computer vision and machine learning software library.

[0020] The idea of ​​the present invention is to provide a method and system for unmanned aerial vehicle (UAV) ship water gauge image recognition. The recognition method mainly includes: Obtain ship water gauge images through drones; The ship draft gauge image is input into a trained image recognition model to obtain a draft gauge recognition result; the image recognition model adopts a convolutional neural network model; the data set used in training the image recognition model includes a number of ship draft gauge images under different weather conditions and different ship draft gauge types.

[0021] This recognition method can identify water gauge images under different weather conditions and maintain high recognition accuracy and stability.

[0022] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments, wherein the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0023] Example 1 like Figure 1 , Figure 2 As shown, this embodiment provides a ship water level image recognition method based on a drone, and the specific implementation method is as follows.

[0024] First, the ship's water gauge image is obtained through the drone. The drone is controlled to hover near the ship, and the drone's built-in shooting equipment is used to obtain the ship's water gauge image from various angles. The obtained water gauge image is uploaded to the processing center.

[0025] A trained image recognition model is built in the processing center. By inputting the ship water gauge image into the trained image recognition model, the text recognition result in the water gauge image is obtained.

[0026] The dataset for training the image recognition model includes several images of ship draft gauges under different weather conditions and different types of ship draft gauges. Specifically, different weather conditions include normal weather and extreme weather; normal weather includes sunny days, rainy days, etc., and extreme weather includes hurricanes, typhoons, tornadoes, thunderstorms, hail, etc. Different types of ship draft gauges include draft gauges of different shapes and sizes, such as standard draft gauges, load line draft gauges, dynamic draft gauges, etc.

[0027] Several ship water gauge images in the data set are preprocessed images, specifically including: after obtaining the ship water gauge images through the UAV, the acquired water gauge images are preprocessed, and the preprocessing includes tilt correction, denoising, grayscale and binarization of the water gauge images.

[0028] Specifically, the tilt correction is implemented through algorithms such as Hough transform and rectangular approximation, detecting straight lines in the image or finding the minimum bounding box to determine the rotation angle, and then correcting the image tilt through affine transformation, rotation and translation operations. The noise in the water gauge image is eliminated through common algorithms such as median filtering, Gaussian filtering and bilateral filtering.

[0029] Use the functions in the OpenCV library to convert the water gauge image into a grayscale image, and use an adaptive function to determine the binarization threshold to improve the accuracy of binarization.

[0030] After the ship water gauge image is preprocessed, it also includes an image segmentation processing step. The image segmentation processing includes: taking the water surface, hull and water gauge marking area in the ship water gauge image as the region of interest, and extracting the region of interest.

[0031] Specifically, the water gauge image in the data set is searched to detect the water gauge area as the region of interest. The region of interest in this embodiment is composed of the water surface in the lower part of the image, the waterline in the middle part, and the hull with water gauge inscriptions above it, so it is easy to detect this area. The detection in this embodiment adopts an image segmentation method. This method adopts a fixed ratio based on the observation and experience summary of the water body in the image and the water gauge marked area, specifically a fixed area within 1 / 3 above and below the image centerline as the ROI area of ​​a certain frame image. Subsequent recognition processing is only performed within the ROI area, including the positioning detection and recognition of the decimal part and integer part of the water gauge inscriptions, and the recognition of the boundary line (waterline) between the hull and the water body.

[0032] The image recognition model adopts a convolutional neural network model, and the convolutional neural network model adopts a YOLOv5 model, a YOLOv10 model, or a FasterRCNN model.

[0033] This embodiment uses the YOLOv5 model, the network structure of which is as follows: Figure 2 As shown in Figure 2, the YOLOv5 model includes a backbone network module, a head network module, and an image recognition module.

[0034] The backbone network module is used to extract features from the region of interest and obtain feature maps. The backbone network module adopts the EfficientNet network architecture. The EfficientNet network architecture is used to extract features based on the depth-separable convolution and residual connection method.

[0035] Specifically, depthwise separable convolution is used to reduce the amount of computation and parameters. This convolution method decomposes the standard convolution into two steps: depthwise convolution (channel-by-channel convolution) and pointwise convolution (1x1 convolution), thereby improving the efficiency of the model.

[0036] The SE module learns the relationship between channels and adaptively adjusts the weights of each channel, which enhances the network's ability to focus on important features, thereby improving the performance of the network. In EfficientNet, the SE module is integrated into the MBConv module to further improve the model's feature selection ability and efficiency. The MBConv module is the basic building block in EfficientNet, which combines depth-separable convolution, residual connection, and expansion operations. Each MBConv block consists of three convolutional layers: a 1x1 convolution for expansion, a 3x3 depth convolution for spatial feature extraction, and a 1x1 convolution for dimensionality reduction. This design significantly reduces the number of parameters and computational complexity of the model while maintaining the model's expressiveness.

[0037] The head network module is used to use the attention or spatial attention mechanism to adaptively weight the feature map to enhance the feature representation capability. The head network module adopts the SENet attention mechanism, which is used to perform global average pooling on the input feature map to obtain a compressed feature vector; after the feature vector passes through the fully connected layer, the weight of each channel is learned, and the weighted feature map is obtained according to the weight of each channel and the input feature map.

[0038] Perform a global average pooling operation on the input feature map to compress the feature map of each channel into a real number. The purpose of this step is to compress the spatial dimensions (height and width) of the feature map into a point, thereby obtaining a feature vector containing only channel information.

[0039] Specifically, when the shape of the input feature map is (C, H, W), where C is the number of channels, H and W are the height and width respectively. After global average pooling, a tensor with a shape of (C, 1, 1) is obtained.

[0040] The compressed feature vector is transformed through a fully connected layer (or convolutional layer) to learn the importance weight of each channel.

[0041] This fully connected layer (or convolutional layer) is usually followed by an activation function (such as ReLU or Sigmoid) to enhance nonlinear expression capabilities. In SENet, the ReLU activation function is usually used for the first transformation, and then the Sigmoid function is used to normalize the output to between 0 and 1 to represent the weight of each channel.

[0042] Finally, the learned weight vector is multiplied by the original feature map to obtain the reweighted feature map. The purpose of this step is to weight different channels of the original feature map to different degrees according to the learned weights, thereby enhancing the representation of useful information and suppressing irrelevant information. By selecting the appropriate head network module, it can be ensured that the model can accurately identify the target object in the image and give its precise location information.

[0043] The image recognition module is used to perform text recognition based on the feature map after feature enhancement. The image recognition module adopts the PSENet detection algorithm, which is used to generate a text area candidate box based on the weighted feature map, and to identify and detect the text area candidate box through non-maximum suppression to obtain the final watermark recognition result. Specifically, the PSENet detection algorithm ResNet is used as the basic network, and feature fusion is performed through the feature pyramid network (FPN) to generate a feature map containing rich information. The feature map is mapped to multiple branches to generate text instance segmentation maps (S1, S2,..., Sn) of different scales, where S1 is the segmentation map of the smallest scale and Sn is the segmentation map of the largest scale. Through the progressive scale expansion algorithm based on breadth-first search (BFS), starting from the small-scale segmentation map S1, it is gradually expanded to the largest-scale segmentation map Sn. During the expansion process, the complete text instance is gradually constructed by merging adjacent text pixels.

[0044] At the same time, in order to avoid the influence of redundant detection frames on the recognition results, this embodiment uses non-maximum suppression to remove redundant detection frames and retain the optimal recognition results. Specifically, all rectangular frames are grouped according to different category labels and sorted according to the confidence level in each group. Then, starting from the rectangular frame with the highest confidence, the remaining rectangular frames are traversed to calculate the intersection over Union (IoU) with the current rectangular frame with the highest confidence. If the IoU of a rectangular frame and the current highest confidence rectangular frame is greater than the preset threshold, it is considered that the two rectangular frames detect the same target, and the rectangular frame is removed. This process is repeated until all rectangular frames are processed, and finally the detection frames with the highest confidence and non-overlapping in each category are obtained.

[0045] During the training process, in order to ensure that the labels can be correctly recognized by the neural network, the image annotation tool labelImg software is used to encode the category label characters 0~9 and m in the data set, a total of 11 text characters, where the label order is 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, m. Each line of the annotation result file represents a label, and each line has five numbers. The first number represents the character represented by the label order, and the remaining four numbers represent the upper left corner position P (rectX, rectY) and size (width, height) of the label rectangle in the image. These labels are used together with the image data to train the model.

[0046] During the training process, the model uses more advanced optimization methods and strategies such as the improved gradient optimization algorithm AdamW, the learning rate scheduler CosineAnnealing, and the parameter initialization strategy Xavier to better guide the model.

[0047] Among them, there is a certain mutual influence between the loss function and the average precision during the training process. Optimizing the loss function helps to improve the overall classification performance of the model, which may indirectly improve the average precision. At the same time, if the model can accurately identify and locate the target, its average precision will also increase accordingly, which is usually reflected in the reduction of classification loss. The results show that the intelligent recognition method of tide level gauge based on neural network proposed in this paper has good recognition effect, and the recognition accuracy rate reaches 99.9%.

[0048] In summary, the high maneuverability and flexibility of drones enable them to quickly reach designated locations and take images, greatly improving work efficiency. When training the image recognition model, several images of ship draft gauges with different weather conditions (such as sunny days, rainy days, foggy days, etc.) and different ship draft gauge types (such as draft gauges of different sizes and shapes) are used as data sets. This diversity can ensure that the model can maintain a high recognition accuracy in different environments and improve the generalization ability of the model. The high-definition images obtained by drones and well-trained image recognition models can greatly improve the accuracy of ship draft gauge recognition. This helps to reduce human errors and improve the accuracy of weight measurement. Since the training data set contains a variety of weather conditions and ship draft gauge types, the trained image recognition model has strong adaptability. Whether in sunny or bad weather, it can maintain a high recognition accuracy and stability.

[0049] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "platform".

[0050] Example 2 In another embodiment of the present invention, a ship water gauge image recognition system for an unmanned aerial vehicle is provided. The system can be used to implement the ship water gauge image recognition method for the unmanned aerial vehicle, specifically as follows: Figure 3 As shown, the ship water gauge image recognition system of the UAV includes a UAV acquisition module and a data processing module.

[0051] Among them, the drone acquisition module is used to obtain the ship water level image through the drone; A data processing module is used to input the ship water gauge image into a trained image recognition model to obtain a water gauge recognition result; The image recognition model adopts a convolutional neural network model; the data set used in the training of the image recognition model includes a number of ship water gauge images of different weather conditions and different types of ship water gauges.

[0052] The drone collection module is communicatively connected with the data processing module, and the data processing module can be arranged in the drone or on the terminal device.

[0053] The terminal device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the ship water level image recognition method of the unmanned aerial vehicle.

Claims

1. A ship water gauge image recognition method based on drone, characterized in that: include: Obtain ship water gauge images through drones; The ship draft gauge image is input into a trained image recognition model to obtain a draft gauge recognition result; the image recognition model adopts a convolutional neural network model; the data set used in training the image recognition model includes a number of ship draft gauge images under different weather conditions and different ship draft gauge types.

2. The method for ship water gauge image recognition based on unmanned aerial vehicle according to claim 1 is characterized in that: After the ship water gauge image is acquired by the unmanned aerial vehicle, the acquired water gauge image is preprocessed, and the preprocessing includes tilt correction, denoising, grayscale and binarization processing on the water gauge image.

3. The method for ship water gauge image recognition based on unmanned aerial vehicle according to claim 2 is characterized in that: After the ship draft gauge image is preprocessed, it also includes an image segmentation processing step, and the image segmentation processing includes: taking the water surface, hull and draft gauge mark area in the ship draft gauge image as the area of ​​interest, and extracting the area of ​​interest.

4. The method for ship water gauge image recognition based on unmanned aerial vehicle according to claim 3 is characterized in that: The convolutional neural network model adopts a YOLOv5 model, a YOLOv10 model or a FasterRCNN model.

5. The method for ship water gauge image recognition based on unmanned aerial vehicle according to claim 4 is characterized in that: The image recognition model is a YOLOv5 model, which includes a backbone network module, a head network module and an image recognition module; The backbone network module is used to extract features from the region of interest to obtain a feature map; the head network module is used to use an attention or spatial attention mechanism to adaptively weight the feature map to enhance the feature representation capability; the image recognition module is used to perform text recognition based on the feature map after feature enhancement.

6. The method for ship water gauge image recognition based on unmanned aerial vehicle according to claim 5 is characterized in that: The backbone network module adopts the EfficientNet network architecture, which is used to extract features based on deep separable convolution and residual connection.

7. The method for ship water gauge image recognition based on unmanned aerial vehicle according to claim 5 is characterized in that: The head network module adopts the SENet attention mechanism, which is used to perform global average pooling processing on the input feature map to obtain a compressed feature vector; After the feature vector passes through the fully connected layer, the weight of each channel is learned, and a weighted feature map is obtained according to the weight of each channel and the input feature map.

8. The method for ship water gauge image recognition based on unmanned aerial vehicle according to claim 5 is characterized in that: The image recognition module adopts the PSENet detection algorithm, which is used to generate a text area candidate frame according to the weighted feature map, and to identify and detect the text area candidate frame through non-maximum suppression to obtain the final water gauge recognition result.

9. The method for ship water gauge image recognition based on unmanned aerial vehicle according to claim 4 is characterized in that: The optimization algorithm used in training the image recognition model is the gradient optimization algorithm AdamW.

10. A ship water gauge image recognition system based on an unmanned aerial vehicle, used to implement the ship water gauge image recognition method based on an unmanned aerial vehicle according to any one of claims 1 to 9, characterized in that: include: UAV acquisition module, used to obtain ship water gauge images through UAV; A data processing module is used to input the ship water gauge image into a trained image recognition model to obtain a water gauge recognition result; The image recognition model adopts a convolutional neural network model; the data set used in training the image recognition model includes a number of ship water gauge images under different weather conditions and different ship water gauge types.

Images