An intelligent recognition algorithm for river water levels based on deep learning

Abstract

The present invention relates to an intelligent recognition algorithm for river water levels based on deep learning, comprising the following steps: setting N marking lines of different colors on the river bank. Obtaining images of the marking lines and the water area near the marking lines, establishing an image sample library of the marking lines and the water area images near the marking lines, and annotating the contours of the marking lines, the reflections of the marking lines, and the water surface. Constructing a Mask RCNN model and training the Mask RCNN model using the image sample library. Obtaining real-time images of the marking lines and the water area near the marking lines, and using the trained Mask RCNN model to segment the marking lines and the water surface in the currently obtained images of the marking lines and the water area near the marking lines. Judging the current water level situation according to the segmentation result. The present invention can improve the real-time performance and accuracy of river water level recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of water conservancy intelligent monitoring, and particularly relates to an intelligent recognition algorithm for river water levels based on deep learning. Background Art

[0002] Water conservancy is the infrastructure of the national economy. The identification of water levels in waters such as rivers, reservoirs, ponds, dams, levees, and sluice gates is an important part of flood control and drought relief. Most of the water level monitoring devices on the market use signal sensing technologies such as lasers and radars, which are expensive, often costing tens of thousands of yuan per unit, and are not suitable for large-scale promotion. In many waters in our country, there is a lack of water level detection devices, or only a surveillance camera is installed near the water area. The operation and maintenance personnel regularly check the water area images uploaded to the background by the camera, and the human eye identifies the water level, which is time-consuming and laborious.

[0003] In recent years, with the rapid development of deep learning technology, computer vision technology based on deep learning has begun to be widely used in all walks of life.

[0004] The master's thesis of Zhao Na from Hebei GEO University discloses a water level detection method based on the fusion of Faster R-CNN and GrabCut. This method first uses the Faster R-CNN network to detect the water gauge part above the water surface, then uses the GrabCut algorithm to accurately segment this part, and finally calculates the water level value using the mapping relationship between pixel coordinates and world coordinates. The disadvantages of this method are as follows: ① During the positioning process of the water gauge by Faster R-CNN, the processing of the reflection of the water gauge in the water surface completely depends on the performance of the Faster R-CNN network. The reflection of the water gauge in the calm and clear water surface is very similar to the characteristics of the water gauge. At this time, Faster R-CNN is very likely to identify the reflection of the water gauge as the water gauge, resulting in a large deviation in the positioning of the water gauge, and then leading to a poor segmentation effect of the GrabCut on the water gauge, and further leading to a large error in water level identification. ② This algorithm needs to establish and fix the mapping relationship between pixel coordinates and world coordinates in advance. However, in actual applications, there is a problem that the relative position between the water gauge and the camera shifts, such as the camera shifting downward due to gravity, and the reset accuracy of the preset position of the spherical camera has a deviation, etc. At this time, the mapping relationship between pixel coordinates and world coordinates will change, and using the original mapping relationship to calculate the water level will result in a deviation.

[0005] Chinese Patent Application 202110134842.6 discloses a method and system for automatically reading water level from water gauge images based on the Mask RCNN algorithm, which has the following deficiencies: ① The influence of the water gauge reflection is not considered. On a calm and clear water surface, the reflection of the water gauge is very similar to the features of the water gauge, and Mask RCNN is very likely to recognize the water gauge reflection as the water gauge as well, resulting in a large error in the detection and segmentation of the water gauge, and further causing a large error in the reading of the water gauge. ② It is necessary to record the coordinates of the four corner points of the water gauge in the image. In actual applications, there are situations where the camera is offset downward due to gravity and the reset accuracy of the PTZ preset position is deviated. These will cause the coordinates of the four corner points of the water gauge in the image to shift. At this time, if the original set parameters are still used, it will cause errors in the water gauge reading.

[0006] Chinese Patent Application 201910536834.7 discloses a method for identifying water gauges based on deep learning. Although this method considers the influence of the water gauge reflection on the water gauge positioning accuracy on the water surface, the universality of the method for dealing with the water gauge reflection is not high. Its processing method is "to identify the water surface by the method of binaryzation through color, and use the Lab color space to judge the color". In actual applications, affected by light, weather, and complex background conditions, it is very difficult to identify the water surface by color, and the accuracy of such methods for water surface identification is not high.

[0007] Therefore, it is necessary to design a river water level identification algorithm that can improve the water level identification accuracy. Summary of the Invention

[0008] In view of the current situation of water level detection in the water conservancy field, the present invention discloses an intelligent identification algorithm for river water level based on deep learning, which realizes high-precision identification of water levels in waters such as rivers, reservoirs, ponds, dams, levees, and sluice gates.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] An intelligent identification algorithm for river water level based on deep learning, the method comprising the following steps:

[0011] S1. Set N marking lines of different colors from top to bottom on the river bank; where N is a positive integer.

[0012] S2. Obtain images of the marking lines and the water area near the marking lines through a monitoring camera installed on the river section, establish an image sample library of the marking lines and the water area near the marking lines, and annotate the outlines of the marking lines, the reflections of the marking lines, and the water surface in each image in the image sample library of the marking lines and the water area near the marking lines to obtain an annotated image sample library of the marking lines and the water area near the marking lines.

[0013] S3. Build a Mask RCNN model and train the Mask RCNN model using the labeled marking lines and the image sample library of the water area near the marking lines to obtain a trained Mask RCNN model.

[0014] S4. Obtain real-time images of the marking lines and the water area near the marking lines through the monitoring cameras installed on the river section, and use the trained Mask RCNN model to segment the marking lines and the water surface in the currently obtained images of the marking lines and the water area near the marking lines.

[0015] S5. According to the segmentation result of step S4, judge the current water level situation. If not all color types of marking lines are segmented, issue an alarm of the corresponding level according to the color of the unsegmented marking lines; if all color types of marking lines are segmented, use graphics algorithms and mathematical formulas to calculate the current water level of the water surface.

[0016] Further, the data in the labeled marking lines and the image sample library of the water area near the marking lines are randomly divided into a training set, a validation set, and a test set according to a ratio of 8:1:1. Use the training set to train the Mask RCNN model, use the validation set to screen the optimal weights of the Mask RCNN model, and use the test set to test the trained Mask RCNN model. When the recognition rate of the trained Mask RCNN model on the test set reaches 95%, it is considered that the algorithm of the Mask RCNN model is qualified; otherwise, adjust the training hyperparameters and retrain the algorithm.

[0017] Further, the value of N is 3. Assuming that the three colors are color A, color B, and color C from top to bottom in sequence, then the statement "if not all color types of marking lines are segmented, issue an alarm of the corresponding level according to the color of the unsegmented marking lines" specifically includes the following steps:

[0018] If the color C marking line is not segmented, it is determined that the water level is above the color C marking line, and a color C alarm is executed; if the color C and color B marking lines are not segmented, it is determined that the water level is above the color B marking line, and a color B alarm is executed; if the color C, color B, and color A marking lines are not segmented, it is determined that the water level is above the color A marking line, and a color A alarm is executed. The color A alarm, color B alarm, and color C alarm represent different levels of alarms.

[0019] Further, the statement "use graphics algorithms and mathematical formulas to calculate the current water level of the water surface" specifically includes the following steps:

[0020] S51. Eliminate the reflection of the marking line in the water surface

[0021] Obtain the areas of the regions enclosed by the contour lines of N color - marked lines and the area of the region enclosed by the water surface contour line. Calculate the areas of the intersections between the regions enclosed by the contour lines of each color - marked line and the region enclosed by the water surface contour line respectively, and obtain the ratio of this area to the area of the region enclosed by the contour line of the marked line. Eliminate the marked lines with an area ratio greater than 0.5.

[0022] Considering the execution speed of the algorithm, a sampling method is used to roughly calculate the area ratio, as follows:

[0023] ① Uniformly extract 9 points on the contour line of the marked line.

[0024] ② Use the ray - casting method to determine whether each point is within the region enclosed by the water surface contour line respectively.

[0025] ③ Count the number of points among the 9 points that are within the region enclosed by the water surface contour line, denoted as n. Then the required area ratio can be roughly calculated as n / 9.

[0026] S52. According to the segmentation result, obtain the minimum - enclosing rotated rectangles of the contour lines of the three color - marked lines A, B, and C respectively. The minimum - enclosing rotated rectangle is obtained by calling the minAreaRect function in the computer vision and machine learning software library opencv.

[0027] S53. Obtain one long side of the minimum - enclosing rotated rectangles of the three color - marked lines A, B, and C respectively, and a total of three sides are obtained.

[0028] S54. Calculate the unit direction vectors of the three sides obtained in step S53, make the signs of the x - axis components of the three unit direction vectors the same, and calculate the mean vector of the three unit vectors. Take this mean vector as the horizontal vector, and take the unit vector orthogonal to this horizontal vector as the vertical vector, denoted as v:(x_v,y_v).

[0029] The calculation method of the unit direction vector is as follows:

[0030] Let (x1,y1) and (x2,y2) be the two endpoints of a side, then the unit direction vector of this side is:

[0031]

[0032] The calculation method of the mean vector is as follows:

[0033] Let e1:(x 1 ,y 1 ),e2:(x 2 ,y 2 ),e3:(x 3 ,y 3 ) be the three unit vectors, then the mean vector of the three unit vectors is:

[0034] ((x 1 +x 2 +x 3 ) / 3,(y 1 +y 2 +y 3 ) / 3)

[0035] The calculation method of the vertical vector is as follows:

[0036] The unit vector orthogonal to the vector (x, y) is:

[0037]

[0038] S55. Obtain the center point Ca of the minimum circumscribed rotation rectangle of the A-color marked line, which is the center point of the A-color marked line.

[0039] Let (x1’, y1’), (x2’, y2’), (x3’, y3’), (x4’, y4’) be the four vertex coordinates of the rotation rectangle, then the coordinates of the center point Ca of the rotation rectangle are (x_c, y_c), where,

[0040] x_c = (x1’ + x2’ + x3’ + x4’) / 4

[0041] y_c = (y1’ + y2’ + y3’ + y4’) / 4.

[0042] S56. Obtain the straight line L passing through the center point Ca of the A-color marked line and parallel to the vertical vector v, and find the intersection points of the straight line L and the water surface contour line, and obtain the point P with the minimum ordinate among all the intersection points.

[0043] Let the coordinates of the center point Ca be (x_c, y_c), and the vertical vector v be (x_v, y_v), then the equation of the straight line L passing through the center point Ca and parallel to the vertical vector v is:

[0044] A*x + B*y + C = 0

[0045] Among them,

[0046] A = y_v

[0047] B = -x_v

[0048] C = x_v*y_c - y_v*x_c

[0049] Let the equation of the straight line L be A*x + B*y + C = 0, and the water surface contour line be a set of points arranged clockwise or counterclockwise, denoted as [pt1, pt2,..., ptn], where the first point is exactly the same as the last point. Then the calculation method of the intersection points of the straight line L and the water surface contour line is as follows:

[0050] Traverse the line segment [pti, pti+1] on the water surface contour line, where i = 1, 2,..., n-1, calculate the intersection points of the line segment and the straight line L, and the point with the smallest ordinate among all the intersection points is the required intersection point P.

[0051] The calculation method of the intersection point of the straight line L: A*x + B*y + C = 0 and the line segment [(xa, ya), (xb, yb)] is as follows:

[0052] Step 1: Judge whether the two endpoints (xa, ya) and (xb, yb) of the line segment are on the straight line L, that is, check whether the following two equalities hold.

[0053] A*xa + B*ya + C = 0

[0054] A*xb + B*yb + C = 0

[0055] If the equality holds, the corresponding point is the intersection point of the straight line and the line segment; otherwise, go to Step 2.

[0056] Step 2: Judge whether the two endpoints (xa, ya) and (xb, yb) of the line segment are on the same side of the straight line L, that is, judge whether the following inequality holds.

[0057] (A*xa + B*ya + C)*(A*xb + B*yb + C) > 0

[0058] If the inequality holds, the two endpoints of the line segment are on the same side of the straight line, and there is no intersection point between the straight line and the line segment; if the inequality does not hold, the two endpoints are on different sides of the straight line, and there is an intersection point between the straight line and the line segment, go to Step 3.

[0059] Step 3: Calculate the functional relationship of the straight line A2*x + B2*y + C2 = 0 where the line segment is located, where

[0060] A2 = yb - ya

[0061] B2 = xa - xb

[0062] C2 = xb*ya - xa*yb

[0063] Use the following formula to obtain the intersection point (x0, y0) of the straight line L and the straight line where the line segment is located, where

[0064] x0 = (C2*B - C*B2) / (A*B2 - A2*B)

[0065] y0 = (C*A2 - C2*A) / (A*B2 - A2*B)

[0066] S57. Calculate the distance d(Ca, P) from point Ca to point P, which is the pixel distance from the water surface to the center point of the A-color marking line;

[0067] If the coordinates of point Ca are (x_c, y_c) and the coordinates of point P are (x0, y0), then the distance from point Ca to point P is

[0068]

[0069] S58. Calculate the physical distance corresponding to a single pixel of the real-time marking line and the image of the water area near the marking line using the following formula:

[0070] d_per_pixel = A_w_cm / A_w_pixel

[0071] where d_per_pixel is the physical distance corresponding to the unit pixel distance, with the unit of cm; A_w_cm is the actual width of the A-color marking scale, with the unit of cm, which is obtained through actual measurement when setting the marking line on the river bank; A_w_pixel is the length of the shorter side of the minimum circumscribed rectangle of the A-color marking line, with the unit of pixel;

[0072] S59. Calculate the physical distance d(water, A) from the center point Ca of the A-color marking line to the water surface in the vertical direction using the following formula:

[0073] d(water, A) = d(Ca, P) × d_per_pixel

[0074] S510. Calculate the water level at the current water surface using the following formula:

[0075] water_level = A_level - d(water, A) - A_w_cm / 2

[0076] where water_level is the water level at the current water surface to be obtained, A_level is the water level at the upper edge of the A-color marking line, d(water, A) is the physical distance from the center point Ca of the A-color marking line to the water surface in the vertical direction, and A_w_cm is the actual width of the A-color marking line measured.

[0077] As can be seen from the above technical solution, an intelligent recognition algorithm for river water level based on deep learning proposed by the present invention uses a Mask RCNN model obtained through training with a large number of samples, which has a high segmentation accuracy for the water level marking lines of three colors A, B, and C and the water surface. Moreover, the segmentation of the water surface can eliminate the reflection of the water level marking lines on the water surface and exclude the influence of the reflection on the algorithm. In addition, the algorithm allows for some offset of the camera during use, and can perform recognition as long as the water level marking lines and the water surface are within the camera's field of view, without affecting the accuracy. Moreover, while giving the water level where the water surface is located, the algorithm also realizes three-level warnings of water levels A, B, and C through the recognition of the three water level marking lines of A, B, and C, and the level warning of the water level is not affected by the recognition accuracy of the water surface. Compared with the prior art, an additional guarantee is added, increasing the practicality of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 is the flowchart of the method of the intelligent recognition algorithm in the present invention;

[0079] Figure 2 is the image of the water area near the marking line collected in step S2 in Embodiment 1 of the present invention;

[0080] Figure 3 is the marked contour line in step S3 in Embodiment 1 of the present invention;

[0081] Figure 4 are the center point Ca of the A-color marking line, the straight line L, and the intersection point P of the straight line L and the water surface contour calculated in step S5 in Embodiment 1 of the present invention;

[0082] Figure 5 is the image of the water area near the marking line collected in step S2 in Embodiment 2 of the present invention;

[0083] Figure 6 are the contour lines of the A, B, and C color marking lines and the water surface segmented in step S3 in Embodiment 2 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0084] The present invention will be further described below with reference to the accompanying drawings:

[0085] As Figure 1 shown, an intelligent recognition algorithm for river water level based on deep learning includes the following steps:

[0086] S1. Set N different color marking lines on the river bank from top to bottom in sequence; where N is a positive integer.

[0087] S2. Obtain images of the marking line and the water area near the marking line through surveillance cameras installed on the river section, establish an image sample library of the marking line and the water area near the marking line, and annotate the marking line, the reflection of the marking line, and the contour of the water surface in each image in the image sample library of the marking line and the water area near the marking line to obtain the annotated image sample library of the marking line and the water area near the marking line.

[0088] S3. Construct a Mask RCNN model, and use the annotated image sample library of the marking line and the water area near the marking line to train the Mask RCNN model to obtain a trained Mask RCNN model.

[0089] The network structure of the conventional Mask RCNN algorithm includes five modules, namely, a feature extraction module, a region proposal network (RPN), an ROIAlign module, a classification module, and a mask module.

[0090] The feature extraction module is used to generate feature maps and includes two parts: a convolutional neural network (CNN) and a feature pyramid network (FPN). The input of this module is an image, and the output is feature maps of various sizes of the image. The CNN extracts high-dimensional features of the image through a series of convolution and pooling operations. The CNN used in the present invention here is Resnet101.

[0091] The FPN is a network structure similar to a pyramid. The FPN further processes the deep feature maps of the CNN into feature maps of the same size as the shallow feature maps of the CNN through operations such as deconvolution or upsampling, and then fuses the two feature maps of the same size but different depths. The fused feature maps contain both the highly abstract features and the detailed information of the image and have stronger feature expression capabilities. Finally, the fused feature maps of different sizes are output to the following service modules for corresponding operations.

[0092] The region proposal network (RPN) is used to extract foreground boxes and includes a convolutional layer, a class regression sub-module, a box regression layer, and finally a non-maximum suppression (nms) operation. The convolutional layer of the RPN performs a 3×3 convolution operation on the feature maps output by the feature extraction module to obtain feature maps with 256 channels.

[0093] Each point (a 256-dimensional vector) on the feature map is called an anchor point, and 9 anchor boxes are generated at each anchor point. The 9 anchor boxes are obtained by combining 3 box sizes and 3 aspect ratios. If the size of the feature map is H×W×256, the RPN will pre-set H×W×9 anchor boxes. The class regression sub-module of the RPN is used to classify the anchor boxes, and here it is only divided into two categories: foreground boxes and background boxes. The specific method is to perform a 1×1 convolution operation on the feature map output by the RPN convolutional layer to obtain a feature map with a size of H×W×18. Here, 18 = 9×2, 9 represents the 9 anchor boxes corresponding to the anchor point, and 2 represents the two categories to which the anchor box belongs. Then, after reshape and softmax operations, the scores of each category are obtained. The box regression layer of the RPN is used to fine-tune the coordinates of the anchor boxes to make them fit the target better. The specific method is to perform a 1×1 convolution operation on the feature map output by the RPN convolutional layer to obtain a feature map with a size of H×W×36. Here, 36 = 9×4, 9 represents the 9 anchor boxes corresponding to the anchor point, and 4 represents the offsets of the 4 parameters (x, y, w, h) of the anchor box coordinates. Here, (x, y) is the center point of the anchor box, and w and h are the width and height of the anchor box. The final non-maximum suppression (nms) operation of the RPN is used to filter the foreground boxes generated by the RPN and eliminate the overlapping boxes. The specific method is to sort the foreground boxes generated by the RPN from large to small in terms of score, obtain the top 12000 foreground boxes, calculate the intersection over union (IoU) of these foreground boxes pairwise, and if the IoU is greater than 0.7, eliminate the foreground box with a lower score. Then, sort the remaining foreground boxes from large to small in terms of score and obtain the top 1200 foreground boxes. The formula for calculating the intersection over union of the two foreground boxes is as follows:

[0094]

[0095] Here, box1 and box2 are two foreground boxes. The box1∩box2 on the right side of the above equation represents the area of the intersection of the two foreground boxes. The box1 on the right side of the above equation represents the area of box1, and the box2 on the right side of the above equation represents the area of box2.

[0096] The ROIAlign module is used to replace the ROIPooling operation to solve the problem of mis-alignment caused by the two quantizations in the POIPooling operation. ROIPooling maps the candidate boxes generated by the RPN layer to a feature map of a fixed size. The specific steps are as follows:

[0097] a. Map the candidate boxes to the feature map to obtain the candidate box feature region, and quantize the boundaries of the feature region to integer point coordinates.

[0098] b. Divide the candidate box feature region into k×k cells, and quantize the boundaries of the cells to integer point coordinates.

[0099] c. Perform max pooling within each unit to obtain a feature map of a fixed size.

[0100] Quantization is performed once in each of the above steps a and b.

[0101] In ROIAlign, the bilinear interpolation method is used to obtain the pixels at floating-point coordinates, replacing the quantization operation in ROIPooling. The specific steps are as follows:

[0102] a. Map the candidate box to the feature map to obtain the candidate box feature region, and keep the floating-point boundaries of the feature region without quantization.

[0103] b. Divide the candidate box feature region into k×k units, and keep the floating-point boundaries of the units without quantization.

[0104] c. Calculate four fixed coordinate positions in each unit, calculate the values at these four positions using the bilinear interpolation method, and then perform the max pooling operation.

[0105] The classification module has two functions: classifying the candidate boxes and regressing the boundaries of the candidate boxes. The specific steps are described as follows: After the candidate boxes pass through ROIAlign to obtain a feature map of a fixed size, they first pass through a fully connected layer of this module, and then are divided into two branches. One branch predicts the category of the candidate boxes, and the other branch regresses the boundaries of the candidate boxes.

[0106] The mask module is used to segment the objects in the candidate boxes. The specific steps are described as follows: After the candidate boxes pass through ROIAlign to obtain a feature map of a fixed size, they first pass through several convolutional operations of this module, and then perform a deconvolution operation to obtain a mask of 28×28×4. Finally, in combination with the candidate box category predicted by the classification module, the mask of the corresponding channel of this category is obtained to get the final segmentation result of the candidate boxes.

[0107] S4. Obtain real-time images of the marking lines and the water area near the marking lines through the monitoring cameras installed on the river sections, and use the trained Mask RCNN model to segment the marking lines and the water surface in the currently obtained images of the marking lines and the water area near the marking lines.

[0108] S5. According to the segmentation results in step S4, judge the current water level situation. If not all color types of marking lines are segmented, issue an alarm of the corresponding level according to the color of the unsegmented marking lines; if all color types of marking lines are segmented, use graphics algorithms and mathematical formulas to calculate the current water level of the water surface.

[0109] The present invention uses the instance segmentation algorithm of Mask RCNN to segment the water level marking line and the water surface, improving the accuracy of the segmentation of the water level marking line and the water surface, maximizing the advantages of computer vision based on deep learning, and improving the accuracy of river water level recognition. Moreover, the present invention can calculate the water level where the water surface is located according to the segmentation results of the river water level marking line and the water surface. The present invention also proposes an alarm strategy for the river water level, that is, when the river water level reaches the alarm height, the alarm level is returned, and when it does not reach the alarm height, the height difference between the water level and the water levels of each alarm level is returned.

[0110] Embodiment 1

[0111] Select a section of the river in a high-tech zone of a certain city as a pilot for water level detection. The specific steps are as follows:

[0112] S1: Paste the marking lines of three colors, A, B, and C, from top to bottom on the river bank. Record the water level at the upper edge of the A-color marking line as 300 cm, denoted as A_level, and measure the actual width of the A-color marking line obtained as 15 cm, denoted as A_w_cm. In this embodiment, A represents red, B represents yellow, and C represents blue.

[0113] S2: Set up a camera near the river bank to monitor the water area near the marking line, and obtain video information in real time to the server. Obtain the image of the water area near the marking line by extracting frames from the video stream, as Figure 2 shown.

[0114] S3: Based on the segmentation algorithm of Mask RCNN, segment the three marking lines of A, B, and C and the water surface. The specific steps are as follows:

[0115] S3.1: Obtain 5000 images of the water area near the marking line with a resolution of 1920×1080 through cameras installed in different river sections, and expand the data set by rotating, adding Gaussian noise, etc. to obtain a sample data set.

[0116] S3.2: Annotate the sample data set. The specific steps are as follows:

[0117] a. Download, install and open the image annotation tool labelme.

[0118] b. Click the "File->Open Dir" button in the upper right corner of labelme in turn, select the folder where the images are located in the pop-up dialog box, and at this time the first image in the image folder will be displayed in the working area of labelme.

[0119] c. Click the "Create Polygons" button in the left toolbar of labelme, create a polygon along the edge of the A-color marked line in the image, and name the polygon red. Similarly, create polygons along the B, C marked lines and the edge of the water surface in the image, and name them yellow, blue, and water in sequence. It should be noted that the reflection of the marked line in the water surface is not distinguished from the marked line, and polygons also need to be created along the edge of the reflection of the marked line, and the names of the polygons are the same as those of the marked lines.

[0120] d. After creating polygons for all the A, B, C-color marked lines and the water surface, click the "Save" button in the left toolbar of labelme to save the created polygon information as a json file. The json file name is the same as the image name. Thus, the annotation of the current image is completed.

[0121] e. After completing the annotation of the current image, click the "NextImage" button in the left toolbar of label to annotate the next image.

[0122] S3.3: Build a Mask RCNN model and train the Mask RCNN model. Specifically as follows: Randomly divide the data annotated in step S3.2 into a training set, a validation set, and a test set in a ratio of 8:1:1. Use the training set to train the segmentation algorithm based on Mask RCNN, use the validation set to screen the optimal weights of the algorithm, and use the test set to test the algorithm obtained by training. When the recognition rate of the algorithm obtained by training reaches 95% on the test set, the algorithm is considered qualified; otherwise, adjust the training hyperparameters and retrain the algorithm.

[0123] S3.4: Use the trained segmentation algorithm based on Mask RCNN in step S3.3 to segment the image of the water area near the marked line to be detected, and obtain the contour lines of the water level marked line and the water surface as Figure 3 shown.

[0124] S4: Since three contour lines are segmented, directly go to the next step.

[0125] S5: Calculate the water level at which the current water surface is located through a series of graphics algorithms and mathematical formulas. The specific steps are as follows:

[0126] S5.1: Eliminate the reflection of the marked line in the water surface. The specific steps are: Calculate the ratio of the area of the intersection of the area enclosed by the contour line of each marked line and the area enclosed by the water surface contour line to the area enclosed by the marked line respectively, and eliminate the marked line with an area ratio greater than 0.5.

[0127] Through calculation, the area of the intersection of the regions enclosed by the contour lines of the three marker lines segmented in step S4 and the region enclosed by the water surface contour line is 0. Therefore, the segmented marker lines are not removed here.

[0128] S5.2: According to the segmentation results in step S3, respectively obtain the minimum circumscribed rotated rectangles of the contour lines of the A-color marker lines, which are the minimum circumscribed rotated rectangles of the A, B, and C marker lines. The results are as follows:

[0129] The four vertices of the minimum circumscribed rotated rectangle of the A-color marker line are: (1012, 453), (1115, 453), (1115, 480), (1012, 480).

[0130] The four vertices of the minimum circumscribed rotated rectangle of the B-color marker line are: (1012, 524), (1115, 524), (1115, 551), (1012, 551).

[0131] The four vertices of the minimum circumscribed rotated rectangle of the C-color marker line are: (1012, 588), (1115, 588), (1115, 615), (1012, 615).

[0132] S5.3: Respectively obtain one long side of the minimum circumscribed rotated rectangles of the A, B, and C marker lines, and a total of three sides are obtained.

[0133] S5.4: Calculate the unit direction vectors of the three sides obtained in S5.3, ensure that the positive and negative signs of the components of the three unit direction vectors on the x-axis are consistent, calculate the mean vector of the three vectors, and consider this mean vector as the horizontal vector. Take the unit vector orthogonal to this horizontal vector as the vertical vector. The calculated vertical vector v is as follows: Denote it as v: (0, 1).

[0134] S5.5: Obtain the center point Ca of the minimum circumscribed rotated rectangle of the A-color marker line, which is the center point of the A-color marker line. The result is as follows:

[0135] Ca: (1063.5, 466.5).

[0136] S5.6: Calculate the straight line L passing through the center point Ca of the A-color marker line and parallel to the vertical vector v. The result is as follows:

[0137] L: x = 1063.5.

[0138] Calculate the intersection points of the straight line L and the water surface contour line, and obtain the point with the minimum ordinate among all the intersection points. The result is as follows:

[0139] P: (1063.5, 926.5).

[0140] S5.7: Calculate the distance d(Ca, P) from point Ca to point P, which is the pixel distance from the water surface to the center point of the A-color marking line. The result is as follows:

[0141] d(Ca, P) = 460.

[0142] S5.8: Calculate the physical distance corresponding to a single pixel of the image according to the following formula:

[0143] d_per_pixel = A_w_cm / A_w_pixel

[0144] = 15 / d((1012, 480), d(1012, 453))

[0145] = 15 / (480 - 453)

[0146] = 0.555 (cm / pixel)

[0147] Wherein, d_per_pixel is the physical distance corresponding to the unit pixel distance, with the unit of cm; A_w_cm is the actual width of the A-color marking scale, with the unit of cm, which is obtained through actual measurement when pasting the marking line on the river bank; A_w_pixel is the length of the short side of the minimum circumscribed rectangle of the A-color marking line, with the unit of pixel.

[0148] S5.9: Calculate the physical distance from the center point of the A-color marking line to the water surface in the vertical direction according to the following formula:

[0149] d(water, A) = d(Ca, P) × d_per_pixel

[0150] = 460 × 0.555

[0151] = 255.56

[0152] Wherein, d(water, A) is respectively the physical distance from the water surface to the A-color marking line in the vertical direction.

[0153] S5.10: Calculate the water level at which the current water surface is located. The calculation formula is as follows:

[0154] water_level = A_level - d(water, A) - A_w_cm / 2

[0155] = 300 - 255.56 - 15 / 2

[0156] = 36.94 (cm)

[0157] Among them, water_level is the water level where the current water surface is located, A_level is the water level at the upper edge of the A-color marking line, d(water, A) is the physical distance in the vertical direction from the center point of the A-color marking line to the water surface calculated in S5.9, and A_w_cm is the actual width of the A-color marking line measured.

[0158] Embodiment 2

[0159] Select a section of river in a high-tech zone of a certain city as a pilot for water level detection.

[0160] S1: Paste the three marking lines A, B, and C on the river bank from top to bottom in sequence, record the water level at the upper edge of the A-color marking line as 100 cm, denoted as A_level, and measure the actual width of the A-color marking line obtained as 15 cm, denoted as A_w_cm.

[0161] S2: Set up a camera near the river bank to monitor the water area near the marking lines, and obtain video information in real time to the server. Obtain images of the water area near the marking lines by extracting frames from the video stream, as Figure 5 shown.

[0162] S3: Use the segmentation algorithm based on Mask RCNN to segment the three marking lines A, B, and C and the water surface. The specific steps are as follows:

[0163] S3.1: Through cameras installed in different river sections, obtain 5000 images of the water area near the marking lines with a resolution of 1920×1080, and expand the dataset by rotating, adding Gaussian noise, etc. to obtain a sample dataset.

[0164] S3.2: Annotate the sample dataset. The specific steps are as follows:

[0165] a. Download, install and open the image annotation tool labelme.

[0166] b. Click the "File->Open Dir" button in the upper right corner of labelme in sequence, select the folder where the images are located in the pop-up dialog box. At this time, the first image in the image folder will be displayed in the working area of labelme.

[0167] c. Click the "Create Polygons" button in the left toolbar of labelme, create a polygon along the edge of the A-color marked line in the image, and name the polygon red. Similarly, create polygons along the B, C marked lines and the edge of the water surface in the image, and name them yellow, blue, and water in sequence. It should be noted that the reflection of the marked line in the water surface is not distinguished from the marked line, and polygons also need to be created along the edge of the reflection of the marked line, and the names of the polygons are the same as those of the marked lines.

[0168] d. After creating polygons for all the A, B, C marked lines and the water surface, click the "Save" button in the left toolbar of labelme to save the created polygon information as a json file. The json file name is the same as the image name. Thus, the annotation of the current image is completed.

[0169] e. After completing the annotation of the current image, click the "NextImage" button in the left toolbar of label to annotate the next image.

[0170] S3.3: Build a Mask RCNN model and train the Mask RCNN model. Specifically as follows: Randomly divide the data annotated in step S3.2 into a training set, a validation set, and a test set according to the ratio of 8:1:1. Use the training set to train the Mask RCNN model, use the validation set to screen the optimal weights of the algorithm, and use the test set to test the trained algorithm. When the recognition rate of the trained algorithm on the test set reaches 95%, the algorithm is considered qualified; otherwise, adjust the training hyperparameters and retrain the algorithm.

[0171] S3.4: Adopt the segmentation algorithm based on Mask RCNN trained in step S3.3 to segment the image of the water area near the marked line to be detected, and obtain the contour lines of the water level marked line and the water surface, as Figure 6 shown.

[0172] S4: Since only the A-color and B-color marked lines are segmented, and the C-color marked line is not segmented, execute the C-color alarm.

[0173] The embodiments described above are only used to describe the preferred embodiments of the present invention, and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. An intelligent recognition algorithm for river water level based on deep learning, characterized in that, the algorithm comprises the following steps: S1. Sequentially set N kinds of marking lines with different colors from top to bottom on the river bank; where N is a positive integer; S2. Obtain images of the marking lines and the water area near the marking lines through a monitoring camera installed on the river section, establish an image sample library of the marking lines and the water area near the marking lines, and label the marking lines, the reflections of the marking lines, and the contours of the water surface in each image in the image sample library of the marking lines and the water area near the marking lines to obtain a labeled image sample library of the marking lines and the water area near the marking lines; The reflection of the marking line refers to: obtaining the areas enclosed by the contour lines of the N kinds of color marking lines and the area enclosed by the water surface contour line, respectively calculating the area of the intersection of the area enclosed by the contour line of each color marking line and the area enclosed by the water surface contour line, and obtaining the ratio of this area to the area enclosed by the contour line of the marking line. The marking line with an area ratio greater than 0.5 is the reflection of the marking line; S3. Construct a Mask RCNN model, and use the labeled image sample library of the marking lines and the water area near the marking lines to train the Mask RCNN model to obtain a trained Mask RCNN model; S4. Obtain real-time images of the marking lines and the water area near the marking lines through a monitoring camera installed on the river section, and use the trained Mask RCNN model to segment the marking lines and the water surface in the currently obtained images of the marking lines and the water area near the marking lines; S5. According to the segmentation result, judge the current water level situation. If not all color types of marking lines are segmented, issue an alarm of the corresponding level according to the color of the unsegmented marking lines; if all color types of marking lines are segmented, use a graphics algorithm and a mathematical formula to calculate the current water level of the water surface.

2. The intelligent recognition algorithm for river water level based on deep learning according to claim 1, characterized in that: The data in the labeled image sample library of the marking lines and the water area near the marking lines are randomly divided into a training set, a validation set, and a test set according to a ratio of 8:1:

1. The Mask RCNN model is trained using the training set, the optimal weights of the Mask RCNN model are selected using the validation set, and the trained Mask RCNN model is tested using the test set.

3. The intelligent recognition algorithm for river water level based on deep learning according to claim 1, characterized in that: The value of N is 3. Let the three colors be color A, color B, and color C from top to bottom in sequence. Then, if not all color types of marking lines are segmented, issue an alarm of the corresponding level according to the color of the unsegmented marking lines, which specifically comprises the following steps: If the C-color marking line is not segmented, it is determined that the water level is above the C-color marking line, and a C-color alarm is executed; If the C-color and B-color marking lines are not segmented, it is determined that the water level is above the B-color marking line, and a B-color alarm is executed; If the C-color, B-color, and A-color marking lines are not segmented, it is determined that the water level is above the A-color marking line, and an A-color alarm is executed.

4. An intelligent recognition algorithm for river water level based on deep learning according to claim 3, characterized in that: The water level of the current water surface is calculated by using graphics algorithms and mathematical formulas, and the specific steps are as follows: S51. Eliminate the reflection of the marking line on the water surface Obtain the areas of the regions enclosed by the contour lines of N color marking lines and the area of the region enclosed by the water surface contour line, calculate the area of the intersection of the region enclosed by the contour line of each color marking line and the region enclosed by the water surface contour line respectively, and obtain the ratio of this area to the area of the region enclosed by the contour line of the marking line. Eliminate the marking lines with an area ratio greater than 0.5; S52. According to the segmentation result, obtain the minimum circumscribed rotated rectangles of the contour lines of the marking lines of three colors A, B, and C respectively; S53. Obtain one long side of the minimum circumscribed rotated rectangles of the marking lines of three colors A, B, and C respectively, and a total of three sides are obtained; S54. Calculate the unit direction vectors of the three sides obtained in step S53. The signs of the components of the three unit direction vectors on the x-axis are the same, and calculate the mean vector of the three unit vectors. Take the unit vector orthogonal to this mean vector as the vertical vector, denoted as v: (x_v, y_v); S55. Obtain the center point Ca of the minimum circumscribed rotated rectangle of the A-color marking line, which is the center point of the A-color marking line; S56. Obtain the straight line L passing through the center point Ca of the A-color marking line and parallel to the vertical vector v, and obtain the intersection point of the straight line L and the water surface contour line. Obtain the point P with the minimum ordinate among all the intersection points; S57. Calculate the distance d(Ca, P) from the point Ca to the point P by the following formula, which is the pixel distance d(Ca, P) from the water surface to the center point of the A-color marking line; where, (x_c, y_c) is the coordinate of the point Ca, and (x0, y0) is the coordinate of the point P; S58. Calculate the physical distance corresponding to a single pixel in the image of the real-time marking line and the water area near the marking line by the following formula; d_per_pixel = A_w_cm / A_w_pixel where, d_per_pixel is the physical distance corresponding to a single pixel in the image, with the unit of cm; A_w_cm is the actual width of the A-color marking scale, with the unit of cm, which is obtained by actual measurement when setting the marking line on the river bank; A_w_pixel is the length of the short side of the minimum circumscribed rectangle of the A-color marking line, with the unit of pixel; S59. Calculate the physical distance d(water, A) from the center point Ca of the A-color marking line to the water surface in the vertical direction by the following formula: d(water, A) = d(Ca, P) × d_per_pixel S510. Calculate the water level at which the current water surface is located by the following formula: water_level = A_level - d(water, A) - A_w_cm / 2 Among them, water_level is the water level where the current water surface is located, A_level is the water level where the upper edge of the A-color marking line is located, d(water, A) is the physical distance from the center point Ca of the A-color marking line to the water surface in the vertical direction, and A_w_cm is the actual width of the A-color marking line measured.

5. An intelligent recognition algorithm for river water level based on deep learning according to claim 4, characterized in that: The minimum circumscribed rotated rectangle is obtained by calling the minAreaRect function in the computer vision and machine learning software library opencv.

6. An intelligent recognition algorithm for river water level based on deep learning according to claim 4, characterized in that: The calculation method of the unit direction vector is as follows: Let (x1, y1) and (x2, y2) be the two end points of a side, then the unit direction vector of this side is: The calculation method of the mean vector is as follows: Let \(e_1:(x 1 ,y 1 ), e_2:(x 2 ,y 2 ), e_3:(x 3 ,y 3 ) be three unit vectors, then the mean vector of the three unit vectors is: ((x 1 +x 2 +x 3 ) / 3,(y 1 +y 2 +y 3 ) / 3) The calculation method of the vertical vector is as follows: The unit vector orthogonal to the vector (x, y) is:

7. An intelligent recognition algorithm for river water level based on deep learning according to claim 4, characterized in that: The center point Ca of the minimum circumscribed rotated rectangle of the A-color marking line is obtained in the following way: Let (x1’, y1’), (x2’, y2’), (x3’, y3’), (x4’, y4’) be the four vertex coordinates of the rotated rectangle, and the coordinates of the center point Ca of the rotated rectangle are (x_c, y_c), then x_c = (x1’ + x2’ + x3’ + x4’) / 4; y_c = (y1’ + y2’ + y3’ + y4’) / 4.

8. An intelligent recognition algorithm for river water level based on deep learning according to claim 4, characterized in that: Obtain the straight line L passing through the center point Ca of the A-color marking line and parallel to the vertical vector v, and find the intersection point of the straight line L and the water surface contour line, and obtain the point P with the smallest ordinate among all the intersection points; the specific steps are as follows: S561. Let the coordinates of the point Ca be (x_c, y_c), and the vertical vector v be (x_v, y_v), then the equation of the straight line L passing through the center point Ca and parallel to the vertical vector v is: A*x + B*y + C = 0 where, A = y_v B = -x_v C = x_v*y_c - y_v*x_c S562. The water surface contour line is a set of points arranged in a clockwise or counterclockwise direction, denoted as [pt1, pt2,..., ptn], where the first point is exactly the same as the last point; traverse the line segment [pti, pti+1] on the water surface contour line, where i = 1, 2,..., n - 1; S563. Calculate the intersection point of the line segment on the water surface contour line and the straight line L, and the point with the smallest ordinate among all the intersection points is the required intersection point P; The calculation method of the intersection point of the straight line L: A*x + B*y + C = 0 and the line segment [(xa, ya), (xb, yb)] is as follows: Step 1: Judge whether the two end points (xa, ya) and (xb, yb) of the line segment are on the straight line L, that is, check whether the following two equalities hold: A*xa + B*ya + C = 0 A*xb + B*yb + C = 0 If the equation holds, the corresponding point is the intersection of the line and the line segment; otherwise, go to step 2; Step 2: Determine whether the two endpoints (xa, ya) and (xb, yb) of the line segment are on the same side of the line L, that is, determine whether the following inequality holds: (A*xa + B*ya + C)*(A*xb + B*yb + C) > 0 If the inequality holds, the two endpoints of the line segment are on the same side of the line L, and there is no intersection between the line and the line segment; if the inequality does not hold, the two endpoints are on different sides of the line, and there is an intersection between the line and the line segment. Go to step 3; Step 3: Calculate the functional relationship of the line where the line segment is located: A2*x + B2*y + C2 = 0; where A2 = yb - ya, B2 = xa - xb, C2 = xb*ya - xa*yb; The intersection point (x0, y0) of the line L and the line where the line segment is located is obtained by the following formula, where: x0 = (C2*B - C*B2) / (A*B2 - A2*B) y0 = (C*A2 - C2*A) / (A*B2 - A2*B).

Images