Water gauge reading method and device based on double neural network model

Abstract

The invention relates to the technical field of computer vision, in particular to a double neural network model-based water gauge reading method and equipment. The method aims to solve the problem of inaccurate reading caused by factors such as rustiness of water gauge characters, transparency of water and hull interferents. A first neural network model is adopted to identify water gauge characters in an image, and category and position information of the water gauge characters is obtained; performing instance segmentation on the water surface area in the image by adopting a second neural network model to generate a mask representing the water surface area; based on the mask, extracting an upper boundary of the mask as a waterline; and calculating a ship water gauge reading according to the water gauge character information and the waterline. According to the method, the waterline is determined by performing instance segmentation on the whole water surface, local linear interference such as a welding seam on a ship body can be effectively ignored, waterline misrecognition is avoided, and the accuracy and robustness of reading are improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and specifically to a method and device for reading water level gauges based on a dual neural network model. Background Technology

[0002] Draft readings are a crucial step in measuring the weight of bulk cargo during ship loading and unloading, and their accuracy directly affects the fairness of trade settlements. Traditional manual reading methods are not only inefficient and labor-intensive, but also easily affected by various factors such as the observer's subjective judgment, weather, and sea conditions, leading to significant reading errors.

[0003] To improve efficiency and accuracy, automatic reading technology based on computer vision and deep learning has been widely researched and applied in recent years. In existing technologies, a common approach is to use a neural network model to process the water gauge image. This involves locating and recognizing characters on the water gauge using a recognition model, while simultaneously using image segmentation and other techniques to determine the boundary between the waterline and the hull, i.e., the waterline. Finally, the reading is calculated by combining the character information and the waterline position.

[0004] However, in practical applications, the aforementioned existing technical solutions still face severe challenges, resulting in insufficient robustness and accuracy. Firstly, ships operate in the marine environment for extended periods, and the characters on the draft gauge surface often become blurred due to corrosion, paint peeling, or contamination. Simultaneously, the shooting angles of observation equipment such as drones or shore-based cameras are often tilted, and these factors collectively cause character image distortion, severely impacting the accuracy of character recognition. Secondly, under certain lighting conditions, such as strong or weak light, the water may exhibit high clarity, leading to low contrast between the image features of the water surface and the ship's hull. This makes it difficult for draft line localization methods relying on edge detection or simple segmentation to accurately identify the true waterline boundary. More seriously, welds, water stains, or paint demarcation lines commonly found on the hull can form strong linear features similar to the true draft line under specific lighting conditions, easily misidentified as the draft line, resulting in significant deviations in the final reading. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a water level reading method and device based on a dual neural network model to overcome the problems existing in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: On the one hand, this application provides a water level reading method based on a dual neural network model, including: Acquire a target image; wherein the target image includes the ship's draft gauge and the water surface; The first neural network model is used to identify the water ruler characters in the target image to obtain the category and location information of each identified water ruler character; A second neural network model is used to perform instance segmentation on the water surface region in the target image to generate a mask representing the water surface region; Based on the mask of the water surface area, its upper boundary is extracted as the waterline; By combining the category and location information of the waterline characters with the waterline information, the ship's waterline reading is calculated.

[0007] Furthermore, in the method described above, acquiring the target image includes: Acquire video data within two wave cycles; The target image is obtained by extracting frames from the video data at fixed intervals; wherein the target image includes the ship's draft gauge and the water surface.

[0008] Furthermore, in the method described above, the first neural network model is a YOLO network model.

[0009] Furthermore, in the method described above, the second neural network model is the YOLO-seg network model.

[0010] Furthermore, the method described above, before acquiring the target image, further includes: Obtain the YOLO network model training dataset and the YOLO-seg network model training dataset; The YOLO network model training dataset was labeled using the labelImg tool, and the YOLO-seg network model training dataset was labeled with waterline using the irregular box tool under the Roboflow annotation tool. The YOLO network model training dataset and the YOLO-seg network model training dataset are divided into the YOLO network model training set, the YOLO network model validation set, the YOLO network model test set, the YOLO-seg network model training set, the YOLO-seg network model validation set, and the YOLO-seg network model test set according to a preset ratio. Obtain the YOLO network model and the YOLO-seg network model pre-trained on the large-scale dataset COCO; The YOLO network model is iteratively trained using the YOLO network model training set and the YOLO network model validation set, and the YOLO network model and the corresponding loss values ​​on the YOLO network model training set and YOLO network model validation set are saved after each round of training. After a preset number of iterations of training, the YOLO network model with the best training performance is determined based on the loss value, and the YOLO network model with the best training performance is verified through the YOLO network model test set. The YOLO-seg network model is iteratively trained using the YOLO-seg network model training set and the YOLO-seg network model validation set, and the YOLO-seg network model and the corresponding loss values ​​on the YOLO-seg network model training set and the YOLO-seg network model validation set are saved after each round of training. After a preset number of iterations of training, the YOLO-seg network model with the best training performance is determined based on the loss value, and the YOLO-seg network model with the best training performance is verified using the YOLO-seg network model test set.

[0011] Furthermore, the method described above, wherein the first neural network model is used to identify water level characters in the target image to obtain the category and location information of each identified water level character, includes: Feature extraction is performed on the target image using the backbone of the YOLO network model; By fusing the feature extraction results generated from the backbone of the YOLO network model through the neck of the YOLO network model, three feature maps representing different levels are generated. The YOLO network model uses a decoupled head structure and employs an anchorless method to perform classification and regression tasks on the feature maps at different levels. The regression task identifies predicted bounding boxes in the feature map and identifies the predicted category of the predicted bounding boxes. The loss is calculated on the output results of the regression task and the classification task using a loss function; Based on the loss calculation results, the output is a predicted image labeled with the prediction category, confidence level, prediction box boundary coordinates, and prediction box coordinate normalization value.

[0012] Furthermore, the method described above, wherein the step of employing a second neural network model to perform instance segmentation on the water surface region in the target image to generate a mask representing the water surface region, includes: Feature extraction is performed on the target image using the backbone of the YOLO-seg network model; The feature extraction results generated by the backbone of the YOLO-seg network model are fused through the neck of the YOLO-seg network model to generate a high-resolution feature map. A prototype mask based on the high-resolution feature map is generated by the prototype branch of the YOLO-seg network model head. The linear combination coefficients of the prototype mask are generated through the mask coefficient branch of the YOLO-seg network model head. The regression branch of the YOLO-seg network model head generates a prediction box based on the high-resolution feature map. The classification results based on the predicted bounding boxes are generated through the classification branch of the YOLO-seg network model head; A prediction mask is generated by linearly superimposing the prototype mask, the prediction box, and the classification result using the linear combination coefficients. The loss function of the YOLO-seg network model is used to calculate the loss of the predicted mask, thereby generating a mask that represents the water surface region.

[0013] Furthermore, in the method described above, extracting the upper boundary of the water surface region as the waterline based on the mask includes: Obtain the confidence level of the mask in the water surface area, and remove the masks in the water surface area that are below a preset threshold; Calculate the area of ​​the mask for the remaining water surface areas, and determine the mask for the water surface area with the largest area; Extract the top boundary points of the mask in the water surface region with the largest area, and perform least squares regression fitting on the top boundary points to generate the waterline.

[0014] Furthermore, the method described above, wherein calculating the ship's draft reading by combining the category and position information of the draft characters with the waterline, includes: Determine the prediction box for the water gauge character closest to the waterline; Determine the coordinates of the waterline and the bottom coordinates of the prediction box of the water gauge character closest to the waterline; The reading of the waterline is calculated based on the coordinates of the waterline and the bottom coordinates of the prediction box of the water gauge character closest to the waterline, and the waterline and the reading of the waterline are marked in the target image.

[0015] On the other hand, this application provides a water level gauge reading device based on a dual neural network model, including a processor and a memory, wherein the processor is connected to the memory: The processor is used to call and execute the program stored in the memory; The memory is used to store the program, which is at least used to execute the water level reading method based on the dual neural network model described above.

[0016] The beneficial effects of this invention are as follows: This application employs a dual-neural network model parallel processing strategy to specifically address the challenges of character recognition and waterline detection, thereby improving the accuracy and robustness of draft readings. Specifically, by segmenting the entire water surface area using a second neural network model, the water surface is recognized as a complete, connected object, and the waterline is determined based on the upper boundary of its mask. This approach effectively ignores local linear interference on the hull caused by welds, water stains, paint boundaries, etc., fundamentally avoiding waterline misidentification due to interference. This significantly improves performance under complex conditions such as character corrosion, tilted shooting, transparent water, and the presence of interference on the hull. Furthermore, the neural network model used in this application is optimized for fast computation, meeting real-time requirements. Moreover, training on a dedicated dataset containing various harsh conditions gives the model strong generalization capabilities, enabling it to adapt to draft reading tasks on different ships and in different environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart provided in one embodiment of a water level reading method based on a dual neural network model according to the present invention; Figure 2 This is a test result diagram of the YOLO network model provided in an embodiment of the water level gauge reading method based on a dual neural network model of the present invention; Figure 3 This is a test result diagram of the YOLO-seg network model provided in an embodiment of the water level gauge reading method based on a dual neural network model of the present invention; Figure 4 This is a YOLO network model diagram provided in one embodiment of the water level gauge reading method based on a dual neural network model of the present invention; Figure 5 This is a structural diagram of a convolution module provided in an embodiment of a water level reading method based on a dual neural network model according to the present invention; Figure 6 This is a C2f module structure diagram provided in an embodiment of the water level gauge reading method based on a dual neural network model of the present invention; Figure 7 This is a diagram of the Bottleneck structure provided in one embodiment of the water level gauge reading method based on a dual neural network model of the present invention. Figure 8 This is an SPPF module structure diagram provided in an embodiment of the water level gauge reading method based on a dual neural network model of the present invention; Figure 9 This is a schematic diagram of the structure of an embodiment of a water level gauge reading device based on a dual neural network model according to the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0020] Figure 1 This is a flowchart illustrating an embodiment of a water level gauge reading method based on a dual neural network model according to the present invention. Please refer to [link / reference]. Figure 1 This embodiment may include the following steps: S1. Acquire the target image; wherein, the target image includes the ship's draft and the water surface; S2. Using the first neural network model, the water ruler characters in the target image are identified to obtain the category and location information of each identified water ruler character; S3. Using a second neural network model, instance segmentation is performed on the water surface region in the target image to generate a mask representing the water surface region; S4. Based on the mask of the water surface region, extract its upper boundary as the waterline; S5. Calculate the ship's draft reading by combining the category and location information of the draft marks with the waterline.

[0021] Preferably, step S1 includes: Acquire video data within two wave cycles; The target image is obtained by extracting frames from the video data at fixed intervals; the target image includes the ship's draft and the water surface.

[0022] Preferably, the first neural network model is the YOLO network model.

[0023] Preferably, the second neural network model is the YOLO-seg network model.

[0024] Preferably, before step S1, the method further includes: Obtain the YOLO network model training dataset and the YOLO-seg network model training dataset; The YOLO network model training dataset was labeled using the labelImg tool, and the waterline of the YOLO-seg network model training dataset was labeled using the irregular box tool under the Roboflow annotation tool. The YOLO network model training dataset and the YOLO-seg network model training dataset are divided into the YOLO network model training set, the YOLO network model validation set, the YOLO network model test set, the YOLO-seg network model training set, the YOLO-seg network model validation set, and the YOLO-seg network model test set according to a preset ratio. Obtain the YOLO network model and YOLO-seg network model pre-trained on the large-scale dataset COCO; The YOLO network model is iteratively trained using the YOLO network model training set and the YOLO network model validation set. After each round of training, the YOLO network model and the corresponding loss values ​​on the YOLO network model training set and the YOLO network model validation set are saved. After a preset number of iterations of training, the YOLO network model with the best training performance is determined based on the loss value, and the YOLO network model with the best training performance is validated using the YOLO network model test set. The YOLO-seg network model is iteratively trained using the YOLO-seg network model training set and the YOLO-seg network model validation set. After each training round, the YOLO-seg network model and the corresponding loss values ​​on the YOLO-seg network model training set and the YOLO-seg network model validation set are saved. After a predetermined number of iterations of training, the YOLO-seg network model with the best training performance is determined based on the loss value. The YOLO-seg network model with the best training performance is then validated using a YOLO-seg network model test set.

[0025] Understandably, it's necessary to photograph and sample ship draft data on-site, discarding images of poor quality (such as blurry images due to out-of-focus issues, images missing large characters from the draft, images with excessive pausing or tilt angles), to obtain a YOLO network model training dataset containing 3000 images. This YOLO network model training dataset includes images under various conditions, such as good lighting and poor lighting at night, clear characters and blurred or rusted characters, and drafts that are vertical and tilted. This comprehensively reflects the real-world situation of draft observations and improves the generalization ability of the trained model.

[0026] The labelImg tool was used to label the YOLO network model training dataset, outputting labels in YOLO dataset format in XML file format. The categories to be detected are the ten digits "0-9" and the letter "M". Only characters above the waterline are labeled; characters invisible to the human eye or reflections on the water surface are not labeled. To enhance the model's robustness, rusted and tilted characters are also labeled.

[0027] The YOLO network model training dataset was divided into three parts: 70% training set, 20% validation set, and 10% test set. The random.sample function was used to randomly split the YOLO network model training dataset. The final YOLO network model training set contains 2100 images, the YOLO network model validation set contains 600 images, and the YOLO network model test set contains 300 images.

[0028] The YOLO-seg network model training dataset is relatively small, containing a total of 500 images. These images were selected from the YOLO network model training dataset and must include real-world conditions that are unfavorable for waterline detection, such as poor lighting at night, relatively transparent water, and water surface color similar to the ship's hull color.

[0029] Waterline detection requires instance segmentation of the water surface, separating the water body from the non-water body. The boundary is the waterline, which is annotated using the irregular bounding box tool in Roboflow. During annotation, multiple straight lines are used to connect and approximate the curve, outlining the entire water surface. Since the characters on the water gauge are raised, the waterline will bend significantly when passing through them; these features should be reflected in the annotation. Then, the Generate tool is used to adjust the segmentation ratio of the YOLO-seg network model training data, and image preprocessing and data augmentation are performed.

[0030] The YOLO-seg network model training dataset is divided into the same proportions as the YOLO network model training dataset: the YOLO-seg network model training set contains 350 images, the YOLO-seg network model validation set contains 100 images, and the YOLO-seg network model test set contains 50 images.

[0031] The system used was Ubuntu 20.04, with NVIDIA driver version 470.239.06. PyCharm was used as the Python integrated development environment (IDE), and PyTorch was used as the deep learning framework. To utilize GPU-accelerated training, CUDA and cuDNN were configured according to the driver version and YOLOv8 version requirements. The tools and libraries required for the model were configured through the installation manifest. The configuration environment is shown in Table 1.

[0032] Table 1 Environment Configuration

[0033] By employing pre-training, loading model files already trained on the large-scale COCO dataset before training allows the model to achieve better results in specific watermark character detection tasks, improving training efficiency. When inputting images, YOLO performs mosaic processing to create 640*640 pixels, followed by random affine transformations, image blurring, and color space enhancement (HSV) to augment the data, enriching the dataset and enhancing the model's robustness. Training parameter settings are shown in Table 2.

[0034] Table 2 Training Parameter Settings

[0035] The number of classes indicates the number of detection categories. Character detection requires recognizing 10 digits and 1 letter, while water surface detection uses only 1 category. The number of epochs is the number of model iterations. Each iteration traverses the entire dataset, and after multiple rounds of training and learning, the model's detection capability gradually improves. The number of workers is the number of work queues. Multiple threads can be used to load data in parallel during data loading, improving data reading speed. The batch size is the number of images read in each batch; due to hardware performance limitations, this parameter is set to 2. The image size is the input image size, set to 640*640. The close_mosaic option disables Mosaic enhancement in the last few epochs. This operation can enhance the model's ability to detect small targets and improve accuracy. Overlap mask determines whether masks should overlap during training. Setting it to True indicates the occlusion relationship between target instances and the accurate position and shape of the target instances. mask_ratio is the mask downsampling ratio, representing the proportion of mask downsampling during training. lr (learning rate) determines the step size of model parameter updates. optimizer specifies the type of optimizer used during training. Selecting the AdamW optimizer moves the weight decay term from gradient calculation to the final weight update step, decoupling gradient calculation and weight decay to improve optimization performance. momentum represents optimizer momentum, a technique to accelerate the gradient descent process and increase the stability of parameter updates. weight_decay is the optimizer's weight decay, used to reduce model complexity and prevent overfitting. During the training of the YOLO network model, the classification loss and bounding box regression loss (including CIoU and DFL) of the YOLO network model on the YOLO network model training set and the YOLO network model validation set were recorded after each iteration, as well as the precision, recall, mAP@0.5 and mAP@[0.5:0.95] of the YOLO network model on the YOLO network model validation set.

[0036] Then analyze the loss curve during training.

[0037] In practice, the loss value decreased rapidly after 1-20 rounds of training. In the subsequent 80 rounds, the rate of decrease slowed and converged. Finally, the classification loss on the validation set decreased to 0.559, and the bounding box regression loss decreased to 1.396 and 1.228, respectively. While precision and recall fluctuated significantly, these fluctuations decreased in the last 80-100 rounds and gradually stabilized, reaching 0.920 and 0.913, respectively. The mAP@0.5 and mAP@[0.5:0.95] curves were relatively smooth, rising rapidly in the first 1-30 rounds before slowly increasing, finally reaching 0.954 and 0.613, respectively. The processing speed on the validation set was also relatively fast, with each image taking 0.2ms for preprocessing, 10.4ms for inference, and 0.5ms for post-processing, for a total of 11.1ms.

[0038] Analyzing the normalized confusion matrix, the detection accuracy for even numbers, "1", and "M" is above 90%, indicating good detection performance. However, because even numbers are commonly used in the character representation of the ruler, odd numbers (except "1") appear less frequently in the dataset, resulting in poorer detection performance. "5" has a 0.17% chance of being predicted as "3", and "9" has a 23% chance of being predicted as "8", which may lead to confusion during detection due to the similar shapes of the two numbers.

[0039] After 100 rounds of training, the model with the best training performance was used to predict the test set, and the results are as follows. Figure 2 For each image, preprocessing takes 1.7ms, inference takes 5.5ms, and post-processing takes 2.8ms, indicating a relatively fast inference speed. The model's character detection and recognition capabilities meet expectations, and it can still recognize characters well under adverse conditions such as tilted water level gauges, rusted characters, and poor lighting, with a high confidence rate.

[0040] During the training of the YOLO-seg network model, the changes in the loss value were analyzed.

[0041] In practice, the instance segmentation loss (seg_loss) decreases rapidly in the first 10 rounds, then decreases slowly in subsequent training, eventually reaching a loss of 0.197 on the validation set. The classification loss (cls_loss) and regression loss (box_loss and dfl_loss) show similar decreasing trends. Although the loss values ​​decrease more slowly than instance segmentation in the first 10 rounds, they maintain a relatively fast decreasing rate in the last 80 rounds. After 100 rounds of training, the three losses on the validation set decrease to 0.229, 0.260, and 0.525, respectively.

[0042] Because of the pre-training approach, precision and recall were already at high levels at the start of training, gradually increasing with each training epoch, stabilizing around 50 epochs, and ultimately reaching 0.997 and 0.994, respectively. The average precision with a threshold of 0.5 was also relatively high initially, stabilizing around 30 epochs and reaching 0.995 after training. The average precision required a longer time to stabilize when the threshold varied between 0.5 and 0.95, ultimately reaching 0.967.

[0043] After 100 rounds of training, the best-performing model was used to predict the test set, and the results are as follows: Figure 3 As shown in the figure, the model preprocesses images in 1.7ms, inference in 6.7ms, and postprocessing in 4.2ms. The red area represents the detected targets during instance segmentation, and the number in the upper left corner represents the confidence rate. The prediction results are good even with raised characters, transparent water, and minimal color difference between the water surface and the ship's hull.

[0044] Preferably, step S2 includes: Feature extraction of the target image is performed using the backbone of the YOLO network model; By fusing the feature extraction results generated by the backbone of the YOLO network model through the neck of the YOLO network model, three feature maps representing different levels are generated. By using a decoupled head structure in the YOLO network model, and employing an anchorless method to perform classification and regression tasks on feature maps at different levels; The predicted bounding boxes are identified in the feature map through a regression task, and the predicted category of the predicted bounding boxes is also identified through a regression task. Loss is calculated on the output results of regression and classification tasks using a loss function; Based on the loss calculation results, the output is a predicted image labeled with the prediction category, confidence level, prediction box boundary coordinates, and prediction box coordinate normalization value.

[0045] Understandably, the YOLO network model mainly consists of three parts: the backbone, the neck, and the head. Figure 4 As shown. The backbone mainly uses convolution and deconvolution to extract features, and introduces residual connections and bottleneck structures to enhance feature extraction capabilities; the neck uses multi-scale feature fusion technology to fuse feature maps from different stages output by the backbone structure, thereby enhancing feature representation capabilities; the head is responsible for target detection and classification, and includes a detection head and a classification head.

[0046] The backbone consists of four C2f modules, five convolutional modules, and one SPPF module. The C2f modules use multiple overlapping identical structures to enhance feature extraction capabilities.

[0047] The convolution module uses a 3x3 convolution kernel with a stride of 2, such as... Figure 5 As shown, the convolutional module downsamples the image, reducing its size and increasing the number of channels. Then, it performs batch normalization (BatchNorm2d) to standardize each mini-batch of data, making training more stable and reducing the dependence on initial weights. Finally, it uses the SiLU activation function for processing. SiLU (Sigmoid Gated Linear Unit) is an improvement on ReLU. By introducing the sigmoid function, it still outputs when the input value is negative, and it is differentiable throughout its domain, which is beneficial for the flow of information in the neural network and avoids the gradient vanishing problem.

[0048] The SiLU activation function is:

[0049] The C2f module first performs a convolution on the input with a 1x1 kernel and a stride of 1, such as... Figure 6 As shown, increasing the network depth increases the number of non-linear mappings, enabling dimensionality scaling of the feature maps, reducing the parameters of the convolutional kernels, and improving model efficiency. Then, after a splitting operation, half of the input is fed into Bottleneck, where multiple 3x3 convolutional kernels are used for convolution operations. The Bottleneck structure is as follows: Figure 7 As shown, after n Bottleneck layer feature extractions, the outputs of all Bottleneck layers are concatenated with the module input. Finally, the output is passed through a 1x1 convolution kernel with a stride of 1. C2f uses the idea of ​​residual connections to reduce gradient explosion or vanishing, allowing gradients to propagate more easily.

[0050] The SPPF module, also known as the Fast Spatial Pyramid Pooling layer, can convert feature maps of arbitrary size into feature vectors of fixed size, improving generalization ability and applicability. For example... Figure 8SPPF's initial and final structures are similar to C2f's. The middle structure uses three layers of max pooling, and the output is obtained by concatenating the results of the first convolution and the three pooling layers.

[0051] The Neck section employs the PAN-FAN approach, fusing features extracted from the Backbone. The FAN structure uses the outputs P3, P4, and P5 from the Backbone after different convolution iterations as input, performing concatenation and upsampling operations from the top layer down to preserve strong semantic information from higher layers and enhance lower-layer features. PAN is an improvement on FAN, addressing FAN's inability to convey localization information. PAN performs concatenation and convolution from the bottom layer up, aggregating shallow feature maps with high resolution and weak semantic information with deep feature maps with low resolution and strong semantic information, further enhancing the expressive power of multi-scale features. Finally, the Neck section outputs three feature maps representing different levels.

[0052] The Head section processes the feature maps analyzed by the model and outputs the results. A decoupled-head structure is used, replacing the anchor-based method with an anchor-free method for both classification and regression tasks. The regression head outputs the coordinates of the predicted bounding boxes and calculates the offset between the predicted and ground truth bounding boxes for loss calculation. The classification head performs convolution operations on candidate boxes and outputs the probability that each candidate box belongs to each category, obtaining the final detection result through maximum suppression.

[0053] The loss calculation for object detection tasks consists of classification loss and bounding box regression loss. The classification loss uses the Binary Cross Entropy Loss (BCE) function, which analyzes the difference between the probability distribution predicted by the model and the probability distribution of the true class for each category. The following is the Binary Cross Entropy Loss function, LOSS. BCE The result is the binary cross-entropy loss function, where N is the number of targets. The true class of the i-th sample Let be the predicted probability of the i-th target.

[0054]

[0055] The regression loss uses two loss calculation methods: CIoU and DFL. IoU can reflect the degree of overlap between two boxes, but it cannot reflect the relative position of the two boxes, and different detection results may occur even with the same degree of overlap. CIoU is an improvement on IoU. In addition to considering the overlapping area, it also adds center point distance and aspect ratio similarity, and comprehensively reflects the difference between the target box and the ground truth box, thereby improving the model's ability to predict bounding boxes.

[0056] IoU calculation method:

[0057] CIoU penalty function:

[0058] in, , These are the center points of the predicted bounding box and the ground truth bounding box (gt), respectively. To find the Euclidean distance between the two center points, The length of the diagonal of the area covered by the predicted bounding box and the ground truth bounding box. As weight, The similarity is based on aspect ratio.

[0059]

[0060] CIoU loss function:

[0061] DFL, or Distributed Focal Loss, replaces the Dirac distribution of bounding box regression values ​​with an arbitrary probability distribution within a certain range. It optimizes the probability of locations near the true label (the two adjacent left and right locations) through cross-entropy, increasing the probability of regression values ​​that are closer to the true label, so that the network can focus on the distribution of the area near the target location more quickly.

[0062] DFL calculation method:

[0063] Global minimum solution:

[0064] Guarantee the estimation of the regression target Infinitely close to the corresponding label y:

[0065] The regression loss calculation first uses DFL to calculate the probability distribution of bounding boxes and labels and optimizes each edge. Then, the probability distribution of bounding boxes is restored to the predicted boxes, and CIoU is used to calculate the loss of the predicted boxes and the ground truth boxes, thus completing the overall optimization of the predicted boxes.

[0066] Finally, a weighted average of CIoU and DFL is calculated for the classification loss and regression loss to complete the loss calculation.

[0067]

[0068] When the YOLO network model performs object detection inference, it outputs predicted images labeled with the predicted class, corresponding confidence score, and predicted bounding boxes. The specific results include four parameters: boxes, masks, keypoints, and probs. Boxes contain cls (predicted class), conf (confidence score), data (coordinates of the top-left and bottom-right corners of the bounding box, class, and confidence score), and normalized bounding box coordinates (xywh: bounding box center coordinates, xywhn: bounding box width and height, xyxy: top-left corner coordinates, xyxyn: bottom-right corner coordinates). Masks, keypoints, and probs represent instance segmentation prediction masks, pose estimation prediction results, and image classification results, respectively. When the task is object detection, all three elements are None. All output data are data types of tensors. To facilitate subsequent computation, the data is converted to arrays, and detection results with low confidence scores are removed.

[0069] Preferably, step S3 includes: Feature extraction of the target image is performed using the backbone of the YOLO-seg network model; By fusing the feature extraction results generated by the backbone of the YOLO-seg network model through the neck of the YOLO-seg network model, a high-resolution feature map is generated. A prototype mask based on a high-resolution feature map is generated by using the prototype branch of the YOLO-seg network model head. Linear combination coefficients of the prototype mask are generated through the mask coefficient branch in the head of the YOLO-seg network model. Predicted bounding boxes based on high-resolution feature maps are generated by the regression branch of the YOLO-seg network model head. Classification results based on predicted boxes are generated by the classification branch of the YOLO-seg network model head; A prediction mask is generated by linearly superimposing the prototype mask, prediction box, and classification result using linear combination coefficients. The loss function of the YOLO-seg network model is used to calculate the loss of the predicted mask, generating a mask that represents the water surface area.

[0070] Understandably, the YOLO-seg network model is structurally similar to the YOLO network model, with the same Backbone and Neck parts. In the Head part, YOLO-seg adds a prototype branch (Proto) and a mask coefficient branch (Mask_coef) to the original classification and regression heads. After extracting high-resolution feature maps from the image using FPN (Feature Pyramid Network), the Protonet network generates multiple prototype masks based on the image. Then, a fully connected layer generates multiple anchor boxes, predicts the linear combination coefficients of each prototype mask, and generates instance masks after non-maximum suppression (NMS) and linear combination.

[0071] The Head section's processing logic is similar to YOLACT. The prototype branch outputs a set of 32 prototype masks. Different masks in the set correspond to different masking information learned by the model. Furthermore, a single mask contains more than one target instance; as shown in the Prototypes section of the YOLACT structure, a person and a racket may appear simultaneously or separately in the same mask. A set of masks needs to be linearly combined and superimposed to obtain the final instance segmentation result. The coefficients of the linear combination can be obtained from the mask coefficient branch. After adding branches, the Head section can output three types of information: classification, predicted bounding boxes, and linear combination coefficients. Based on the classification and predicted bounding boxes, a 32-dimensional linear combination coefficient can be obtained, corresponding to the number of prototype masks in the set.

[0072] The final linear superposition method is shown below:

[0073] The YOLO-seg classification and regression branches use the same loss functions as the object detection model: the classification branch uses BCE, and the regression branch uses CIoU and DFL. During training, the prototype branch does not supervise the prototype mask and linear combination coefficients; instead, it performs pixel-by-pixel loss calculations on the predicted mask and the ground truth labeled regions, using a binary cross-entropy loss function that combines BCE and sigmoid (BCEWithLogitsLoss).

[0074] The calculation method for the cross-entropy loss in binary classification is as follows:

[0075] in, It is the target's true label, with only two possible values: 0 and 1. This indicates that the pixel belongs to the target. This indicates that the pixel belongs to the background; is the logistic value (logits) output by the model, representing the probability that the target is predicted as 1; N is the number of targets. The function first applies a sigmoid activation to the input, and then calculates the binary cross-entropy loss.

[0076] Compared to object detection, the evaluation metrics for instance segmentation include an assessment of the accuracy of the prediction mask, such as segloss, mAP@0.5 (Mask), and mAP@[0.5:0.95] (Mask), which correspond to the loss of instance segmentation, the average accuracy of the prediction mask when the IoU threshold is 0.5, and the average accuracy of the prediction mask when the IoU threshold varies between 0.5 and 0.95, respectively.

[0077] Preferably, step S4 includes: Obtain the confidence level of the mask in the water surface area and remove the mask in the water surface area that is below the preset threshold; Calculate the area of ​​the mask for the remaining water surface region and determine the mask for the water surface region with the largest area; Extract the top boundary points of the mask in the largest water surface area, and perform least squares regression fitting on the top boundary points to generate the waterline.

[0078] Understandably, the output of the water surface instance segmentation task includes masks containing the waterline instance masks predicted by the model, specifically a four-dimensional tensor containing the x and y coordinates of the water surface mask boundary points. Confidence scores are obtained from the boxes, and the mask and confidence scores are packaged using the `zip` function. Objects with confidence scores below 0.5 in the mask are removed. Since water level detection video recordings may encounter obstructions such as ropes, causing the water surface to be segmented into different parts, the dataset after removing low-confidence masks is traversed, and the `contourArea` function from the OpenCV library is used to calculate the area. The mask with the largest area is selected as the processing result. The mask obtained through inference represents the entire water surface. To obtain the waterline data, boundary points unrelated to the waterline must be removed from the mask. In images, the water surface is mostly curved inverted triangles or curved trapezoids; deleting the bottom boundary points and retaining the top boundary points yields the waterline pixels.

[0079] After obtaining the boundary points related to the waterline, the polyfit and poly1d functions from the NumPy basic package for scientific computing in Python are used to perform least squares regression fitting on the boundary points.

[0080] Preferably, step S5 includes: Determine the prediction box for the water gauge character closest to the waterline; Determine the coordinates of the waterline and the bottom coordinates of the prediction box of the water gauge character closest to the waterline; Calculate the waterline reading based on the coordinates of the waterline and the bottom coordinates of the prediction box of the water gauge character closest to the waterline, and mark the waterline and the waterline reading in the target image.

[0081] Understandably, the formula for calculating the reading is as follows:

[0082] Where poly represents the waterline fitting function.

[0083] The present invention also provides a water level gauge reading device based on a dual neural network model for implementing the above method embodiments. Figure 9 This is a schematic diagram of the structure of an embodiment of a water level gauge reading device based on a dual neural network model according to the present invention. Figure 3 As shown, the water level reading device based on a dual neural network model in this embodiment includes a processor 21 and a memory 22, with the processor 21 connected to the memory 22. The processor 21 is used to call and execute the program stored in the memory 22; the memory 22 is used to store the program, which is at least used to execute the water level reading method based on the dual neural network model in the above embodiments.

[0084] The specific implementation scheme of the water level reading device based on the dual neural network model provided in this application can refer to the implementation scheme of the water level reading method based on the dual neural network model in any of the above embodiments, and will not be repeated here.

[0085] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0086] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0087] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0088] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0089] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0090] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0091] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0092] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0093] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A water level reading method based on a dual neural network model, characterized in that, include: Acquire a target image; wherein the target image includes the ship's draft gauge and the water surface; The first neural network model is used to identify the water ruler characters in the target image to obtain the category and location information of each identified water ruler character; A second neural network model is used to perform instance segmentation on the water surface region in the target image to generate a mask representing the water surface region; Based on the mask of the water surface area, its upper boundary is extracted as the waterline; By combining the category and location information of the waterline characters with the waterline information, the ship's waterline reading is calculated.

2. The method according to claim 1, characterized in that, The acquisition of the target image includes: Acquire video data within two wave cycles; The target image is obtained by extracting frames from the video data at fixed intervals; wherein the target image includes the ship's draft gauge and the water surface.

3. The method according to claim 2, characterized in that, The first neural network model is the YOLO network model.

4. The method according to claim 3, characterized in that, The second neural network model is the YOLO-seg network model.

5. The method according to claim 4, characterized in that, Before acquiring the target image, the following is also included: Obtain the YOLO network model training dataset and the YOLO-seg network model training dataset; The YOLO network model training dataset was labeled using the labelImg tool, and the YOLO-seg network model training dataset was labeled with waterline using the irregular box tool under the Roboflow annotation tool. The YOLO network model training dataset and the YOLO-seg network model training dataset are divided into the YOLO network model training set, the YOLO network model validation set, the YOLO network model test set, the YOLO-seg network model training set, the YOLO-seg network model validation set, and the YOLO-seg network model test set according to a preset ratio. Obtain the YOLO network model and the YOLO-seg network model pre-trained on the large-scale dataset COCO; The YOLO network model is iteratively trained using the YOLO network model training set and the YOLO network model validation set, and the YOLO network model and the corresponding loss values ​​on the YOLO network model training set and YOLO network model validation set are saved after each round of training. After a preset number of iterations of training, the YOLO network model with the best training performance is determined based on the loss value, and the YOLO network model with the best training performance is verified through the YOLO network model test set. The YOLO-seg network model is iteratively trained using the YOLO-seg network model training set and the YOLO-seg network model validation set, and the YOLO-seg network model and the corresponding loss values ​​on the YOLO-seg network model training set and the YOLO-seg network model validation set are saved after each round of training. After a preset number of iterations of training, the YOLO-seg network model with the best training performance is determined based on the loss value, and the YOLO-seg network model with the best training performance is verified using the YOLO-seg network model test set.

6. The method according to claim 5, characterized in that, The first neural network model is used to identify water ruler characters in the target image to obtain the category and location information of each identified water ruler character, including: Feature extraction is performed on the target image using the backbone of the YOLO network model; By fusing the feature extraction results generated from the backbone of the YOLO network model through the neck of the YOLO network model, three feature maps representing different levels are generated. The YOLO network model uses a decoupled head structure and employs an anchorless method to perform classification and regression tasks on the feature maps at different levels. The regression task identifies predicted bounding boxes in the feature map and identifies the predicted category of the predicted bounding boxes. The loss is calculated on the output results of the regression task and the classification task using a loss function; Based on the loss calculation results, the output is a predicted image labeled with the prediction category, confidence level, prediction box boundary coordinates, and prediction box coordinate normalization value.

7. The method according to claim 6, characterized in that, The step of employing a second neural network model to perform instance segmentation on the water surface region in the target image to generate a mask representing the water surface region includes: Feature extraction is performed on the target image using the backbone of the YOLO-seg network model; The feature extraction results generated by the backbone of the YOLO-seg network model are fused through the neck of the YOLO-seg network model to generate a high-resolution feature map. A prototype mask based on the high-resolution feature map is generated by the prototype branch of the YOLO-seg network model head. The linear combination coefficients of the prototype mask are generated through the mask coefficient branch of the YOLO-seg network model head. The regression branch of the YOLO-seg network model head generates a prediction box based on the high-resolution feature map. The classification results based on the predicted bounding boxes are generated through the classification branch of the YOLO-seg network model head; A prediction mask is generated by linearly superimposing the prototype mask, the prediction box, and the classification result using the linear combination coefficients. The loss function of the YOLO-seg network model is used to calculate the loss of the predicted mask, thereby generating a mask that represents the water surface region.

8. The method according to claim 7, characterized in that, The method of extracting the upper boundary of the mask based on the water surface region as the waterline includes: Obtain the confidence level of the mask in the water surface area, and remove the masks in the water surface area that are below a preset threshold; Calculate the area of ​​the mask for the remaining water surface areas, and determine the mask for the water surface area with the largest area; Extract the top boundary points of the mask in the water surface region with the largest area, and perform least squares regression fitting on the top boundary points to generate the waterline.

9. The method according to claim 8, characterized in that, The process of calculating the ship's draft reading by combining the category and location information of the draft characters with the waterline includes: Determine the prediction box for the water gauge character closest to the waterline; Determine the coordinates of the waterline and the bottom coordinates of the prediction box of the water gauge character closest to the waterline; The reading of the waterline is calculated based on the coordinates of the waterline and the bottom coordinates of the prediction box of the water gauge character closest to the waterline, and the waterline and the reading of the waterline are marked in the target image.

10. A water level reading device based on a dual neural network model, characterized in that, It includes a processor and a memory, wherein the processor is connected to the memory: The processor is used to call and execute the program stored in the memory; The memory is used to store the program, which is at least used to execute the water level reading method based on a dual neural network model as described in any one of claims 1-9.

Images