Multi-level water gauge water level visual measurement method based on deep learning target detection and regression model

Abstract

The invention discloses a multi-level water gauge water level visual measurement method based on deep learning target detection and a regression model, and the method comprises the steps: expanding a sample data set through the water level range extended data enhancement based on image stretching and the image data enhancement based on brightness, hue and saturation in a model training stage, and enabling a model to have higher generalization; in the measuring stage, the water gauge areas above the water surface are recognized through the target detection model, the water gauge series and the bottom edge coordinates of the target frame are determined, the ROI image corresponding to the first water gauge area recognized through target detection is input into the regression model, the water line coordinates are further predicted, and the coordinates, obtained through the two methods, of the water line in the original image are compared. Judging whether the detection is valid; finally, the regressed water line coordinates are converted into water gauge readings through an N-point linear fitting method, and the water level value is calculated in combination with the zero point elevation. And in combination with target detection and regression prediction, the number of times of step-by-step regression is reduced, and the efficiency and precision of multi-stage water gauge water level visual measurement are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent water conservancy video monitoring, and in particular to a multi-level water gauge water level visual measurement method based on deep learning target detection and regression model. Background Art

[0002] Water level is one of the basic hydrological elements of rivers, lakes and reservoirs, and is an important indicator of water flow changes. Water level monitoring of rivers, lakes and reservoirs plays an important role in water conservancy and social and economic development, and has become an important means of disaster prevention and mitigation. The water gauge records the height of the water level by reading, and is the most intuitive and simple measurement tool; however, traditional water gauge measurement requires manual regular observation, has a low degree of automation, and is labor-intensive. Existing automatic water level gauges mainly include float type, pressure type, ultrasonic type and radar type, but they generally have the disadvantages of high equipment and installation costs, measurement accuracy is easily affected by ambient temperature, sediment content and on-site control structure, and requires regular maintenance by staff.

[0003] At present, important hydrological monitoring sites in China are equipped with video surveillance equipment and standard water gauges to ensure daily monitoring of the water conditions of rivers, lakes and reservoirs. This provides a basis for water level measurement based on computer vision to replace traditional physical measurement equipment. Standard water gauge types usually include upright single-stage water gauges, inclined water gauges and short-pile multi-stage water gauges. Image-based water gauge water level measurement technology uses image acquisition equipment to obtain the image of the water gauge to be measured, and then uses digital image processing technology to complete the detection of the water level line and convert it into the actual water level value, thereby realizing automatic acquisition of water level information. With the continuous improvement of video surveillance technology, image-based water gauge water level measurement technology has gradually been widely used in the field of water level measurement due to its advantages such as non-contact, no temperature drift, low system cost and maintenance cost.

[0004] Traditional image-based water gauge water level measurement methods include the method of identifying water gauge surface characters and scales, Hough transform straight line detection and horizontal projection methods. 1) The method of identifying water gauge surface characters and scales detects the characters and scales on the water gauge surface through image processing, and obtains the actual water level from the character category and the total number of scales. There is no need to convert the actual water level value. It is specifically divided into two types: detecting water gauge scale lines and identifying water gauge characters. When the image resolution is low and the water gauge scale lines and characters are not clear, it is difficult to ensure the accuracy of detection. 2) The method of Hough transform straight line detection detects the boundary between the water gauge and the water body in the image through image preprocessing and Hough transform. The straight line detection method is mostly used in static water conditions, when the water gauge sheet is close to the wall, but the water gauge of the hydrological station is mostly located in the river channel. The water level line in the image may be accompanied by wave impact and floating objects entangled, making the water level line in the image not an ideal straight line, and it is difficult to accurately detect a straight line representing the true position of the water level line. 3) The general idea of ​​the horizontal projection method is to divide the water gauge and water body into two binary images, use the change of the horizontal projection curve to reflect the grayscale features in the image, and then determine the water level by detecting the peak. The horizontal projection method will be affected by ambient light, and the waves and floating objects on the water surface will cause noise in the horizontal projection curve of the edge image, which will interfere with the detection of the water level.

[0005] With the advancement of big data technology and high-performance computing equipment, deep learning technology has been widely used in computer vision tasks. Compared with the limitations of traditional image processing methods that only use low-level features, deep learning methods can effectively deal with water level measurement problems under complex water flow conditions (such as wave breaking, floating objects entanglement) and lighting conditions (including strong light, shadows, reflections).

[0006] At present, there are three methods for measuring water gauges and water levels based on deep learning: target detection, image semantic segmentation, and regression prediction. 1) Model: Input the water gauge image to be measured into the trained target detection model to detect the water gauge object and identify the water gauge characters. According to the number of water gauge characters and the total length of the water gauge, the current water level is calculated. 2) The water gauge and water level measurement method based on image semantic segmentation establishes an orthogonal water gauge image dataset, trains a semantic segmentation network model, inputs the water gauge image to be measured into the trained semantic segmentation model, completes the pixel-by-pixel classification prediction of the water gauge image, and determines the position of the water level line through the pixel horizontal projection method. 3) The water gauge and water level measurement method based on the regression model establishes a water gauge image dataset, trains a deep neural network regression model, and directly inputs the water gauge image to be measured into the trained regression model during measurement to predict the coordinate value of the water level line.

[0007] Since the measurement range of a single-stage water gauge is limited, if the range is expanded by increasing the length of the water gauge, it will make the installation of the water gauge more difficult, and it is easy to be shaken, tilted, or even damaged by the impact of floods and floating objects. Therefore, most of the hydrological stations in China currently measure water levels in stages by setting up multi-stage water gauges. If target detection or semantic segmentation methods are used to identify the coordinates of the water level line, it is necessary to first orthorectify the ROI images of each level of water gauges, and the correction process of multi-stage water gauges is relatively cumbersome. If the water level line regression method is used to identify the coordinates of the water level line, it is necessary to perform water level regression prediction of the water level gauge ROI image step by step in a certain order according to the position of each level of water gauge in the image, and the detection efficiency is low. In addition, due to the interference of various complex scenes such as lighting, weather, and water flow, as well as changes under various water level conditions (low, medium, and high water levels), these methods have problems with scene adaptability and generalization ability in achieving accurate water level detection. Summary of the invention

[0008] Purpose of the invention: The present invention provides a multi-level water gauge water level visual measurement method based on deep learning target detection and regression model, which combines target detection and regression prediction, reduces the number of step-by-step regression, and improves the efficiency and accuracy of multi-level water gauge water level visual measurement.

[0009] Technical solution: The multi-level water gauge water level visual measurement method based on deep learning target detection and regression model described in the present invention comprises the following steps:

[0010] Step 1: Use the image measurement system to obtain the original low water level image, and calibrate the ROI according to the position of each level of water gauge in the image;

[0011] Step 2: Select original images under different scenes to perform appearance-based image data enhancement, manually mark the water level areas above the water surface to construct a multi-scene sample data set, and use the sample data set to train the YOLOv5 model;

[0012] Step 3, select the original images under different scenes, crop the water gauge ROI images at various levels according to the ROI selected in the water gauge calibration stage, scale the water gauge ROI images at various levels to the same size, mark the water level line coordinates of the water gauge ROI images at various levels, perform water level range expansion data enhancement based on image stretching and ROI image data enhancement based on appearance, and use the sample data set to train the SE-HRNet regression model;

[0013] Step 4: Use the image method measurement system to obtain the original image, and input the original image into the YOLOv5 model for water gauge detection;

[0014] Step 5: According to the output information of the target detection network, the number of water level levels and the bottom edge coordinates of the water level detection frame can be determined;

[0015] Step 6: After preprocessing, the ROI image corresponding to the first water gauge area detected by the target detection network is input into the regression model to obtain the water level line ROI image coordinates, and the water level line ROI image coordinates are converted into coordinates in the original image. The coordinates of the bottom edge of the target detection box and the coordinates of the water level line obtained by the regression method in the original image are compared to determine whether the detection is reliable;

[0016] Step 7: Convert the regressed water level line coordinates into water gauge readings through the N-point linear fitting method and calculate the water level value in combination with the zero point elevation.

[0017] Furthermore, in step 1, the image method flow measurement system is used to obtain the original water gauge image of the low water level. First, it is necessary to connect the camera, set the measurement time, download the video file, and then capture the image sequence to obtain an original low water level image. The ROI of each level of water gauge is calibrated according to the position of each level of water gauge in the original image, wherein the ROI image of each level of water gauge represents the complete water gauge image above and below the water surface.

[0018] Furthermore, in step 2, original images under different scenes are selected and image data enhancement is performed based on brightness, hue, and saturation.

[0019] Furthermore, in step 3, data enhancement for water level range expansion based on image stretching is performed, and the ordinate value of the water level line generated by a random algorithm (i.e., a random label) is compared with the ordinate value of the original water level line (i.e., the original true value). If the random label is greater than the original true value, the top of the water ruler ROI image is cropped, and the cropping height is the difference between the random label and the original true value. The image of the area below the original water level position and 10 pixels away from the original water level line to the bottom of the image is stretched, and the height increased by the stretching is equal to the cropping height; otherwise, the bottom of the water ruler ROI image is cropped, and the cropping height is the difference between the original true value and the random label. The image of the area above the original water level position and 10 pixels away from the original water level line to the top of the image is stretched, and the height increased by the stretching is equal to the cropping height.

[0020] Furthermore, the linear difference method is used to stretch the image in the vertical direction. First, the corresponding coordinates in the original image are calculated by vertical coordinate mapping. The calculation formula is as follows:

[0021]

[0022] Among them, H represents the height of the original image, H′ represents the height of the target image, y′ represents the vertical coordinate of the target image, and y orig Indicates the corresponding vertical coordinate in the original image;

[0023] According to y orig Perform linear interpolation, and the calculation formula of pixel value is as follows:

[0024]

[0025] Where x represents the horizontal coordinate of the image, y′ represents the vertical coordinate of the stretched image, y1 and y2 represent y orig The nearest integer pixel positions above and below, I(x,y1) represents the pixel value with coordinates (x,y1) in the original image, I(x,y2) represents the pixel value with coordinates (x,y2) in the original image, and I(x,y′) represents the interpolated pixel value with coordinates (x,y′) in the target image.

[0026] Furthermore, the SE-HRNet regression model was trained using the enhanced sample dataset, where SE-HRNet represents a high-resolution regression network based on the attention mechanism; the high-resolution regression network can maintain high-resolution representation throughout the entire process, starting with a high-resolution subnet as the first stage, and adding subnets from high to low resolution one by one to form more stages, and connecting multi-resolution subnets in parallel. During the whole process, the information in the parallel multi-resolution subnets is repeatedly exchanged to perform repeated multi-scale fusion. The attention mechanism module can make the model pay more attention to useful channel information by learning adaptive channel weights.

[0027] Furthermore, in step 5, the level of the water ruler is determined by comparing the center coordinates of the water ruler output by the target detection network with the pre-calibrated ROI parameters of each level, and the pixel vertical coordinates of the bottom frame of the target frame are calculated according to the center coordinates of the water ruler target frame output by the target detection network and its height and width information; assuming that the coordinates of the upper left corner and the lower right corner of the first-level water ruler ROI image are (X1, Y1) and (X2, Y2) respectively, if the center coordinates of the detected water ruler target frame are (x, y), and satisfy X1<x< X2 and Y1<y<Y2, it indicates that this level of water ruler belongs to the first level of water ruler. If the center coordinates of the target frame are not within the ROI image range of the first level of water ruler, then compare whether it belongs to the ROI image of the next level of water ruler in turn until the corresponding water ruler level is found; if no matching water ruler level is found, the target detection frame is judged to be a false detection, and this method is repeated for subsequent target frames. According to the detected center coordinates of the water ruler target frame (x, y), the height of the target frame is h, and the width is w, the vertical coordinate of its bottom border pixel is

[0028] Furthermore, in step 6, according to the information output by the target detection network, the water gauge ROI image corresponding to the first target frame area is selected and input into the regression model after preprocessing to obtain the water level line ROI image coordinates, wherein the preprocessing is to scale the cropped water gauge ROI images of each level to the same size in proportion, and the output information of the target detection network is sorted from small to large according to the center coordinate x of the detected target frame area. Since the water level values ​​in the data set are all higher than the zero elevation of the first-level water gauge, the area where the first target frame is located must contain water, and its bottom edge image coordinates are the original image coordinates of the water level line.

[0029] Furthermore, the water level line ROI image coordinates are converted into the original image coordinates by the following method:

[0030] According to the scaling ratio of the image preprocessing, the ordinate of the water level ROI image before image preprocessing is calculated. The ordinate of the water level in the original image is calculated by the water ruler ROI image coordinate parameters. The image scaling ratio is calculated using the following formula:

[0031]

[0032] Wherein, the target ROI width or height is the width or height of the ROI image after preprocessing, and the original ROI width or height is the width or height of the ROI image before preprocessing;

[0033] The original water level line ROI image ordinate is calculated using the following formula:

[0034]

[0035] Among them, y represents the ordinate of the water level line ROI image before preprocessing, and Y represents the ordinate of the water level line ROI image after preprocessing;

[0036] The vertical coordinate of the water level line in the original image is calculated using the following formula:

[0037] y′=y+y0

[0038] Where y′ represents the ordinate of the water level in the original image, y represents the ordinate of the water level ROI image before preprocessing, and y0 represents the ordinate of the upper edge of the water level ROI image in the original image;

[0039] By comparing the pixel coordinates of the water level lines obtained by the two methods in the original image, if the difference is less than 10 pixels, it means that the measurement result is reliable, and the regressed water level line ROI image coordinate value is converted into a water level value; if the difference is greater than 10 pixels, it means that the measurement result is unreliable, and the average of the results of the two methods is output and an error is reported.

[0040] Further, in step 7, based on the coordinates of the multiple calibration points in the ROI image and the coordinates in the physical space, the conversion relationship between the image coordinate system and the physical space coordinate system is determined, and the conversion relationship between the image coordinate system and the physical coordinate system is determined using the following formula:

[0041]

[0042] Among them, P yi Indicates the ordinate (vertical coordinate) of the i-th calibration point in the water gauge image, L yi represents the ordinate (actual height) of the i-th calibration point in the physical space, a and b are conversion parameters, and the values ​​of a and b are calculated according to the least squares method, and n represents the total number of calibration points;

[0043] Use the following formula to calculate the physical ordinate of the water level:

[0044] L=a+b*P

[0045] Among them, L represents the physical ordinate of the water level, and P represents the image ordinate of the water level;

[0046] Use the following formula to calculate the water level:

[0047] L′=L+L0

[0048] Among them, L′ represents the water level value, L0 represents the zero point elevation of the water gauge, and the zero point elevation of the water gauge refers to the height difference between the zero scale line (starting line) of the water gauge and a certain base surface.

[0049] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: in the model training stage, the present invention expands the sample data set through water level range expansion data enhancement based on image stretching and image data enhancement based on brightness, hue and saturation, so that the model is more generalized; in the measurement stage, the water gauge area above the water surface is identified through the target detection model, the water gauge level and the bottom edge coordinates of the target frame are determined, the ROI image corresponding to the first water gauge area identified by the target detection is input into the regression model, and the water level line coordinates are further predicted. The coordinates of the water level lines obtained by the two methods in the original image are compared to determine whether the detection is effective; finally, the regressed water level line coordinates are converted into water gauge readings through the N-point linear fitting method and the water level value is calculated in combination with the zero point elevation; combining target detection and regression prediction, the number of step-by-step regressions is reduced, and the efficiency and accuracy of visual measurement of water levels of multi-level water gauges are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0051] Figure 2It is a schematic diagram of ROI of each level of water gauge in the original image of the present invention.

[0052] Figure 3 Schematic diagram of appearance-based data enhancement for the target detection method of the present invention.

[0053] Figure 4 Schematic diagram of appearance-based data enhancement for the regression method of the present invention.

[0054] Figure 5 Schematic diagram of data enhancement based on range expansion of the regression method of the present invention.

[0055] Figure 6 Schematic diagram of the target detection network structure of the present invention.

[0056] FIG. 7( a ) is a schematic diagram of the regression network structure of the present invention.

[0057] FIG7( b ) is a schematic diagram of the regression network structure of the present invention.

[0058] Figure 8 Schematic diagram of target detection network input according to the present invention.

[0059] Fig. 9 This is a schematic diagram of the target detection network output of the present invention.

[0060] Fig.10 Schematic diagram of the input of the regression model of the present invention.

[0061] Fig.11 This is a schematic diagram of the regression model output of the present invention.

[0062] Fig.12 It is a schematic diagram of the water gauge ROI image calibration of the present invention. DETAILED DESCRIPTION

[0063] like Figure 1 As shown, a multi-level water gauge water level visual measurement method based on deep learning target detection and regression model includes the following steps:

[0064] S1: Use the image measurement system to obtain the original image of the low water level, and calibrate the ROI according to the position of each level of water gauge in the image;

[0065] S2: Select original images under different scenes and perform image data enhancement based on brightness, hue, and saturation, manually mark the water level areas above the water surface to construct a multi-scene sample data set, and use the sample data set to train the YOLOv5 model;

[0066] S3: Select the original images under different scenes, crop the ROI images of each level of water gauge according to the ROI selected in the water gauge calibration stage, scale the ROI images of each level of water gauge to the same size, and mark the water level coordinates of the ROI images of each level of water gauge. Perform water level range expansion data enhancement based on image stretching and ROI image data enhancement based on brightness, hue, and saturation, and use the sample data set to train the SE-HRNet regression model;

[0067] S4: Use the image method measurement system to obtain the original image, and input the original image into the YOLOv5 model for water gauge detection;

[0068] S5: According to the output information of the target detection network, the number of water level levels and the bottom edge coordinates of the water level detection frame can be determined;

[0069] S6: After preprocessing, the ROI image corresponding to the first water gauge area detected by the target detection network is input into the regression model to obtain the water level line ROI image coordinates, and the water level line ROI image coordinates are converted into coordinates in the original image. Compare the bottom edge coordinates of the target detection box with the coordinates of the water level line obtained by the regression method in the original image to determine whether the detection is reliable;

[0070] S7: The regressed water level coordinates are converted into water gauge readings by the N-point linear fitting method and the water level value is calculated in combination with the zero point elevation.

[0071] In step S1, the ROI of each level of water scale is calibrated according to the position of each level of water scale in the original image, such as Figure 2 The image coordinate parameters of the water level ROI at each level in the original image are shown in Table 1.

[0072] Table 1 Water ruler ROI image coordinate parameters

[0073]

[0074] In step S2, Figure 3 As shown in the figure, the original images under different scenes are selected and image data enhancement based on brightness, hue and saturation is performed. The detailed parameters of data enhancement are shown in Table 2. p is the key factor to control the range of image variation. Each enhancement parameter (brightness, hue, saturation) varies randomly within a certain range. The parameter p determines the amplitude of the change, and its range is [0,1]. By randomly adjusting the brightness of the image to [1-p, 1+p] times, the change from dark light to strong light (such as cloudy to sunny) is simulated; the hue of the image is randomly adjusted to [1-p, 1+p] times to simulate the color shift caused by lighting changes (such as sunrise and sunset); the saturation of the image is randomly adjusted to [1-p, 1+p] times to simulate different scenes where the color changes from dim to bright (such as cloudy and sunny). The specific pixel calculation formula is as follows:

[0075] I′(x,y)=I(x,y)*(1+Δp)

[0076] Where I(x,y) represents the original pixel value of the image at position (x,y), I′(x,y) represents the pixel value of the image at position (x,y) after data enhancement, and Δp represents the adjustment factor, which ranges from [-1,1].

[0077] Table 2 Appearance-based data augmentation parameters

[0078]

[0079] The sample data set is used to train the YOLOv5 model. The network structure of the YOLOv5 model is as follows: Figure 6 As shown. The structure is mainly divided into four parts: input, Backbone (backbone network), Neck network and Prediction (output). The backbone network is mainly composed of Focus structure and CSP structure. Focus structure is a convolutional neural network layer used for feature extraction, which is used to compress and combine the information in the input feature map to extract higher-level feature representation. The core idea of ​​​​the CSP structure is to divide the input feature map into two parts, one part is processed by a small convolutional network (called a subnetwork), and the other part is directly processed by the next layer. Then the two parts of the feature map are spliced ​​together as the input of the next layer, so that low-level detail features and high-level abstract features can be combined to improve the efficiency of feature extraction. The output end can output information such as the position, size and category of the target in the image, so that subsequent tasks such as target recognition and tracking can be processed.

[0080] In step S3, the original images in different scenes are selected, and the ROI images of each level of the water gauge are cropped according to the ROI selected in the water gauge calibration stage. The ROI images of each level of the water gauge are proportionally scaled to a uniform size, and the water level line coordinates of the ROI images of each level of the water gauge are marked. The height and width of the input image of the regression model used in the present invention are 256*64. Since the regression network needs to be downsampled 32 times, the height and width of the input image must be an integer multiple of 32. Since the regression model has a strong dependence on the distribution of water level data, we perform water level range expansion data enhancement based on image stretching and ROI image data enhancement based on brightness, hue, and saturation, such as Figure 4 The sample data set is used to train the SE-HRNet regression model. The ROI image data enhancement parameters based on brightness, hue, and saturation are shown in Table 2, and the water level range expansion data enhancement based on image stretching is shown in Table 2. Figure 5 shown. Figure 5The expansion process of the water level from low to high and from high to low is shown. Where I represents the random label value and L represents the original true value. If the random label is greater than the original true value, the top of the water gauge ROI image is cropped, and the cropping height is the difference between the random label and the original true value. The image is stretched from the position below the original water level line and 10 pixels away from it to the bottom of the image. The height increased by stretching is equal to the cropping height. Otherwise, the bottom of the water gauge ROI image is cropped, and the cropping height is the difference between the original true value and the random label. The image is stretched from the position above the original water level line and 10 pixels away from it to the top of the image. The height increased by stretching is equal to the cropping height. In order to avoid distortion caused by the water level line not being completely horizontal, we choose to start the stretching process at a certain distance above or below the water level line. This method can effectively prevent the water level line from being unnaturally deformed in the image and maintain its visual correctness. The SE-HRNet regression model was trained using the enhanced sample dataset.

[0081] As shown in Figure 7, the high-resolution regression network based on the attention mechanism mainly consists of four parts: input, backbone network, attention mechanism module, and output. The SE-HRNet regression network can maintain high-resolution representation throughout the process, starting with a high-resolution subnet as the first stage, adding high-to-low resolution subnets one by one to form more stages, and connecting multi-resolution subnets in parallel. Throughout the process, the information in the parallel multi-resolution subnets is repeatedly exchanged to perform repeated multi-scale fusion. The attention mechanism module can make the model pay more attention to useful channel information by learning adaptive channel weights.

[0082] In step S4, the input of the YOLOv5 model is an original image, such as Figure 8 As shown. The output is a txt document containing the information of all the water gauges detected in the current original image. The document content includes the category, the center x coordinate of the detection frame, the center y coordinate of the detection frame, the width of the detection frame, the height of the detection frame, and the confidence level. The water gauge information at each level output by the target detection network adopted in the present invention is sorted from small to large according to the center coordinates of the detected water gauge areas at each level. The center coordinate of the leftmost water gauge is the smallest, and the zero point elevation corresponding to the water gauge is the lowest. The zero point elevation of the water gauge refers to the height difference between the zero scale line (starting line) of the water gauge and a certain base surface. The zero point elevations of each level of water gauge are shown in Table 3.

[0083] Table 3 Parameters of zero elevation of water gauge at each level

[0084]

[0085] In step S5, by comparing the center coordinates of the water ruler output by the target detection network with the pre-calibrated ROI parameters of each level, the level of the water ruler can be determined. According to the center coordinates of the water ruler target frame output by the target detection network and its height and width information, the pixel vertical coordinates of the bottom border of the target frame can be calculated. For example, assume that the coordinates of the upper left corner and the lower right corner of the first-level water ruler ROI image are (X1, Y1) and (X2, Y2), respectively. If the center coordinates of the detected water ruler target frame are (x, y), and X1<x<X2 and Y1<y<Y2 are satisfied, it indicates that the level of water ruler belongs to the first-level water ruler. If the center coordinates of the target frame are not within the ROI image range of the first-level water ruler, it is compared in turn whether it belongs to the ROI image of the next level of water ruler until the corresponding water ruler level is found; otherwise, it is determined that there is an error in the target detection. According to the height of the detected water ruler target frame h and the width w, the pixel vertical coordinates of its bottom border are In this embodiment, Figure 8 As shown in the figure, an original image at 8:30 am on May 10, 2022 is selected and input into the target detection network. According to the information output by the target detection, Fig. 9 As shown in the figure, the center coordinates (x, y) of the first water ruler target frame are (1482, 926), the width and height are 26 and 62 respectively, and it is calculated that this is the second level water ruler and the vertical coordinate of the bottom border pixel is 957. The coordinate parameters of the ROI images of each level of water ruler in the original image are shown in Table 1.

[0086] In step S6, Fig.10 As shown in the figure, the ROI image corresponding to the second-level water gauge is preprocessed and input into the regression model. The vertical coordinate of the water level line image output by the regression model is 126, as shown in the figure. Fig.11 As shown. According to the coordinate parameters of the water level gauge ROI at each level, it can be calculated that the width and height of the original ROI of the second level water level gauge are 39 and 158 respectively, and the width and height of the pre-processed water level gauge ROI are 64 and 256 respectively. The vertical coordinate of the water level line ROI image is converted into the vertical coordinate of the original image by the following method:

[0087] According to the image preprocessing scaling ratio, the ordinate of the water level ROI image before image preprocessing is calculated, and the ordinate of the water level in the original image is calculated by the water ruler ROI image coordinate parameters. The image scaling ratio is calculated using the following formula:

[0088]

[0089] The target ROI width or height is the width or height of the ROI image after scaling, and the original ROI width or height is the width or height of the ROI image before scaling.

[0090] The original water level line ROI image ordinate is calculated using the following formula:

[0091]

[0092] Wherein, y represents the ordinate of the water level line ROI image before preprocessing, and Y represents the ordinate of the water level line ROI image after preprocessing.

[0093] The vertical coordinate of the water level line in the original image is calculated using the following formula:

[0094] y′=y+y0

[0095] Wherein, y′ represents the ordinate of the water level in the original image, y represents the ordinate of the water level ROI image before preprocessing, and y0 represents the ordinate position of the upper edge of the original ROI image of the water level in the original image.

[0096] According to the above formula, the vertical coordinate y′ of the water level line in the original image can be calculated to be 959. By comparing the pixel coordinates of the water level line detected by the two methods in the original image, it can be determined whether the detection result is reliable. In this example, the pixel coordinates of the water level line output by the target detection in the original image are 957, and the pixel coordinates of the water level line output by the regression prediction in the original image are 959. The error is less than the threshold set by the present invention, so the result is reliable.

[0097] In step 7, the water level line ROI image coordinates are converted into the original image coordinates by the following method:

[0098] The conversion relationship between the image coordinate system and the physical coordinate system is obtained by fitting the ordinates in the image coordinates of multiple calibration points and the ordinates in the physical coordinates (i.e., the actual heights of the calibration points in the physical space). On an image, the ordinate in the image coordinates of each point decreases as the ordinate in the physical coordinates of the point increases, so the conversion relationship can be a linear relationship. Fig.12 As shown, assuming that the vertical coordinates of each calibration point in the image are P1, P2... and the actual heights corresponding to each calibration point in the physical space are L1, L2..., then the linear relationship can be expressed as the following formula:

[0099] L i =a+b*P i

[0100] Among them, P i represents the vertical coordinate of the i-th calibration point in the image, L i Indicates P i The actual height in the physical space, a and b are parameters in the conversion relationship between the image coordinate system and the physical space coordinate system. According to this linear relationship, the vertical coordinates of the pixel point in any image can be converted to the actual height in the physical space.

[0101] In an example, the values ​​of a and b can be calculated using the least squares method. The specific formula is as follows:

[0102]

[0103] Among them, P i Indicates the coordinates (vertical coordinates) of the ith calibration point in the water gauge image, L i represents the coordinate (actual height) of the i-th calibration point in the physical space, and n represents the total number of calibration points.

[0104] Use the following formula to calculate the water level:

[0105] L′=L+L0

[0106] Among them, L' represents the water level value, and L0 represents the zero point elevation of the water gauge. The zero point elevation of the water gauge refers to the height difference between the zero scale line (starting line) of the water gauge and a certain base surface.

[0107] In this example, three calibration points are selected from top to bottom in the water gauge ROI image. The pixel coordinates and physical coordinate parameters of each calibration point are shown in Table 4. The values ​​of a and b are calculated to be 111.712 and -0.404 respectively. The vertical coordinate of the water level line output by the regression method in this example in the ROI image is 126, and the water gauge reading (physical coordinate) is calculated to be 60.808cm. From Table 3, it can be seen that the zero point elevation of the second-level water gauge is 42.291m, and the water level value at this time is calculated to be 42.899m. The manual reading at this moment is 42.911m, and the error is 0.012m.

[0108] Table 4 Water gauge ROI image calibration point parameters

[0109]

Claims

1. A multi-level water gauge water level visual measurement method based on deep learning target detection and regression model, characterized in that: The steps include: Step 1: Use the image measurement system to obtain the original low water level image, and calibrate the ROI according to the position of each level of water gauge in the image; Step 2: Select original images under different scenes to perform appearance-based image data enhancement, manually mark the water level areas above the water surface to construct a multi-scene sample data set, and use the sample data set to train the YOLOv5 model; Step 3, select the original images under different scenes, crop the water gauge ROI images at various levels according to the ROI selected in the water gauge calibration stage, scale the water gauge ROI images at various levels to the same size, mark the water level line coordinates of the water gauge ROI images at various levels, perform water level range expansion data enhancement based on image stretching and ROI image data enhancement based on appearance, and use the sample data set to train the SE-HRNet regression model; Step 4: Use the image method measurement system to obtain the original image, and input the original image into the YOLOv5 model for water gauge detection; Step 5: According to the output information of the target detection network, the number of water level levels and the bottom edge coordinates of the water level detection frame can be determined; Step 6: After preprocessing, the ROI image corresponding to the first water gauge area detected by the target detection network is input into the regression model to obtain the water level line ROI image coordinates, and the water level line ROI image coordinates are converted into coordinates in the original image. The coordinates of the bottom edge of the target detection box and the coordinates of the water level line obtained by the regression method in the original image are compared to determine whether the detection is reliable; Step 7: Convert the regressed water level line coordinates into water gauge readings through the N-point linear fitting method and calculate the water level value in combination with the zero point elevation.

2. The multi-level water gauge water level visual measurement method based on deep learning target detection and regression model as claimed in claim 1 is characterized in that: In step 1, the image method flow measurement system is used to obtain the original water gauge image of the low water level. First, it is necessary to connect the camera, set the measurement time, download the video file, and then capture the image sequence to obtain a low water level original image. The ROI of each level of water gauge is calibrated according to the position of each level of water gauge in the original image. Among them, the ROI image of each level of water gauge represents the complete water gauge image above and below the water surface.

3. The multi-level water gauge water level visual measurement method based on deep learning target detection and regression model as claimed in claim 1 is characterized in that: In step 2, original images under different scenes are selected and image data enhancement is performed based on brightness, hue, and saturation.

4. The multi-level water gauge water level visual measurement method based on deep learning target detection and regression model as claimed in claim 1, characterized in that: In step 3, the water level range expansion data enhancement based on image stretching is performed. The ordinate value of the water level line generated by the random algorithm is compared with the ordinate value of the original water level line. If the random label is greater than the original true value, the top of the water gauge ROI image is cropped. The cropping height is the difference between the random label and the original true value. The image of the area below the original water level line and 10 pixels away from it to the bottom of the image is selected for stretching. The height increased by stretching is equal to the cropping height. Otherwise, the bottom of the water ruler ROI image is cropped, and the cropping height is the difference between the original true value and the random label. The image of the area above the original water level line and 10 pixels away from it to the top of the image is stretched, and the height increased by stretching is equal to the cropping height.

5. The multi-level water gauge water level visual measurement method based on deep learning target detection and regression model as claimed in claim 4 is characterized in that: The linear difference method is used to stretch the image vertically. First, the corresponding coordinates in the original image are calculated through vertical coordinate mapping. The calculation formula is as follows: Among them, H represents the height of the original image, H′ represents the height of the target image, y′ represents the vertical coordinate of the target image, and y orig Indicates the corresponding vertical coordinate in the original image; According to y orig Perform linear interpolation, and the calculation formula of pixel value is as follows: Where x represents the horizontal coordinate of the image, y′ represents the vertical coordinate of the stretched image, and y1 and y2 represent y orig The nearest integer pixel positions above and below, I(x,y1) represents the pixel value with coordinates (x,y1) in the original image, I(x,y2) represents the pixel value with coordinates (x,y2) in the original image, and I(x,y′) represents the interpolated pixel value with coordinates (x,y′) in the target image.

6. The multi-level water gauge water level visual measurement method based on deep learning target detection and regression model as claimed in claim 1, characterized in that: In step 3, the SE-HRNet regression model is trained using the enhanced sample dataset, where SE-HRNet represents a high-resolution regression network based on an attention mechanism; the high-resolution regression network can maintain high-resolution representation throughout the entire process, starting with a high-resolution subnet as the first stage, adding high to low resolution subnets one by one to form more stages, and connecting multi-resolution subnets in parallel. During the entire process, the information in the parallel multi-resolution subnets is repeatedly exchanged to perform repeated multi-scale fusion. The attention mechanism module can learn adaptive channel weights to make the model pay more attention to useful channel information.

7. The multi-level water gauge water level visual measurement method based on deep learning target detection and regression model as claimed in claim 1, characterized in that: In step 5, the level of the water ruler is determined by comparing the center coordinates of the water ruler output by the target detection network with the pre-calibrated ROI parameters of each level, and the pixel ordinate of the bottom border of the target frame is calculated according to the center coordinates of the water ruler target frame output by the target detection network and its height and width information; assuming that the coordinates of the upper left corner and the lower right corner of the first-level water ruler ROI image are (X1, Y1) and (X2, Y2) respectively, if the center coordinates of the detected water ruler target frame are (x, y), and X1<x<X2 and Y1<y<Y2 are satisfied, it indicates that the level of water ruler belongs to the first-level water ruler. If the center coordinates of the target frame are not within the ROI image range of the first-level water ruler, then the ROI image of the next level of water ruler is compared in turn until the corresponding water ruler level is found; if the matching water ruler level is not found, the target detection frame is determined to be a false detection, and this method is repeated for subsequent target frames. According to the detected center coordinates of the water ruler target frame (x, y), the height of the target frame is h, and the width is w, then the pixel ordinate of its bottom border is 8. The multi-level water gauge water level visual measurement method based on deep learning target detection and regression model as claimed in claim 1, characterized in that: In step 6, according to the information output by the target detection network, the water gauge ROI image corresponding to the first target frame area is selected and input into the regression model after preprocessing to obtain the water level line ROI image coordinates, wherein the preprocessing is to scale the cropped water gauge ROI images of each level to the same size in proportion, and the output information of the target detection network is sorted from small to large according to the center coordinate x of the detected target frame area. Since the water level values ​​in the data set are all higher than the zero elevation of the first-level water gauge, the area where the first target frame is located must contain water, and its bottom edge image coordinates are the original image coordinates of the water level line.

9. The multi-level water gauge water level visual measurement method based on deep learning target detection and regression model as claimed in claim 8, characterized in that: The water level line ROI image coordinates are converted into the original image coordinates by the following method: According to the scaling ratio of the image preprocessing, the ordinate of the water level ROI image before image preprocessing is calculated. The ordinate of the water level in the original image is calculated by the water ruler ROI image coordinate parameters. The image scaling ratio is calculated using the following formula: Wherein, the target ROI width or height is the width or height of the ROI image after preprocessing, and the original ROI width or height is the width or height of the ROI image before preprocessing; The original water level line ROI image ordinate is calculated using the following formula: Among them, y represents the ordinate of the water level line ROI image before preprocessing, and Y represents the ordinate of the water level line ROI image after preprocessing; The vertical coordinate of the water level line in the original image is calculated using the following formula: y′=y+y0 Where y′ represents the ordinate of the water level in the original image, y represents the ordinate of the water level ROI image before preprocessing, and y0 represents the ordinate of the upper edge of the water level ROI image in the original image; By comparing the pixel coordinates of the water level lines obtained by the two methods in the original image, if the difference is less than 10 pixels, it means that the measurement result is reliable, and the regressed water level line ROI image coordinate value is converted into a water level value; if the difference is greater than 10 pixels, it means that the measurement result is unreliable, and the average of the results of the two methods is output and an error is reported.

10. The multi-level water gauge water level visual measurement method based on deep learning target detection and regression model as claimed in claim 1, characterized in that: In step 7, based on the coordinates of the multiple calibration points in the ROI image and the coordinates in the physical space, the conversion relationship between the image coordinate system and the physical space coordinate system is determined, and the conversion relationship between the image coordinate system and the physical coordinate system is determined using the following formula: Among them, P yi Indicates the ordinate of the i-th calibration point in the water gauge image, L yi represents the ordinate of the i-th calibration point in the physical space, a and b are conversion parameters, and the values ​​of a and b are calculated according to the least squares method, and n represents the total number of calibration points; Use the following formula to calculate the physical ordinate of the water level: L=a+b*P Among them, L represents the physical ordinate of the water level, and P represents the image ordinate of the water level; Use the following formula to calculate the water level: L′=L+L0 Among them, L′ represents the water level value, L0 represents the zero point elevation of the water gauge, and the zero point elevation of the water gauge refers to the height difference between the zero scale line of the water gauge and a certain base surface.

Images