Complex environment water level monitoring system and method based on heterogeneous light field cooperation

Abstract

The application discloses a complex environment water level monitoring system and method based on heterogeneous light field cooperation, which is characterized in that: a water gauge image is segmented from a water gauge point cloud and a joint external parameter; the water gauge image is further segmented after image enhancement, and the tilted water gauge is corrected; the intensity of the water gauge point cloud is accumulated, and the water level line is obtained through the maximum gradient method; the corrected water gauge image is detected by using a target detection algorithm to detect digital characters, the character recognition algorithm is used to perform OCR recognition on the detected digital characters, and the water level value is calculated according to the digital characters, the water level line and the water gauge image. Through the combination of laser radar and visible light field, the reflection characteristics of different materials are utilized, so that the water level in the complex environment of a river channel, a reservoir and a coast can be monitored in real time. The interference of floating objects and weather factors on water level detection is effectively avoided, and the accuracy and robustness of water level monitoring are significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of water level detection technology, and further relates to a complex environment water level monitoring system and method based on heterogeneous optical field synergy. This invention can be used for real-time water level monitoring of rivers, reservoirs, or coastlines, providing accurate data for formulating drought and flood control strategies. Background Technology

[0002] Real-time water level monitoring provides crucial information for developing drought and flood control strategies, river management plans, reservoir scheduling, and water resource management. Existing water level monitoring devices fall into two categories: contact and non-contact. Both have limitations. For example, contact-type float level gauges require a static water device; non-contact ultrasonic and radar level gauges require regular calibration and have stringent installation requirements. Image processing methods have lower installation requirements but perform poorly in complex environments. The general process for water level detection using image recognition methods is: water level gauge segmentation, tilt correction, water level line positioning, character segmentation, OCR detection, and water level calculation. Under ideal conditions, this method performs well. However, images collected from natural water bodies are complex and diverse, and many solutions ignore issues such as abnormal exposure, rain, snow, fog, and occlusion. These are unavoidable in practical applications and lead to varying degrees of image degradation, further hindering the accuracy of these solutions in harsh environments and rendering them unsuitable for practical use. To address the aforementioned issues, this invention proposes a complex environment water level monitoring system and method based on heterogeneous light field synergy. This water level monitoring system constructs a synergistic processing mechanism based on radar laser field and visible light field according to the reflection characteristics of different materials such as metal and water by lidar, which can make up for the shortcomings of using image processing alone for water level detection.

[0003] Chongqing Zhixing Shulian Intelligent Technology Co., Ltd. disclosed a deep learning-based method for detecting river water levels without a water gauge in its patent application document "A Deep Learning-Based Method for Detecting River Water Levels Without a Water Gauge" (patent application number CN202311379848.5, publication number CN117974962A). The implementation steps of this method are as follows: (1) Set a calibration object at the water level detection point and obtain an image containing the calibration object from the camera. Obtain the mapping relationship between the pixel coordinate system and the world coordinate system through calibration; (2) Use the SAM model to segment the water body area and non-water body area of ​​the target water area and obtain a binary image; (3) Crop the neighborhood of the water level calibration point on the binary image and obtain the water level segmentation line on the image through image processing; (4) Binarize the single frame image obtained by the pull flow, obtain the region of interest near the water level, divide the region of interest into multiple sub-regions of interest, and calculate the Laplace value; (5) Compare the sub-regions of interest to determine the region where the Laplace value changes abruptly. This region is to be the water level line region; (6) By comparing steps three and five, determine the position of the water level line and calculate the water level height through the mapping relationship. This method can use a calibration object instead of a water gauge. After calibration, the mapping relationship between the pixel coordinate system and the world coordinate system is obtained, which solves the dependence on the physical water gauge to a certain extent and can realize water level detection when the calibration object is blocked. However, this method still has three shortcomings: First, the size of the calibration object needs to be known, and the camera cannot be moved; recalibration is required after changes in camera position or angle. Second, when there are large floating or attached objects, the identified water level will shift downward, causing the detected value to be lower than the true value. Third, the influence of weather on the detection process is not considered, and the impact of image degradation caused by abnormal exposure, rain, snow, fog, etc., on the detection accuracy is ignored.

[0004] In their paper "Application of Deep Learning Techniques in Water Level Measurement: Combining Improved SegFormer-UNet Model with Virtual Water Gauge" (Applied Sciences 2023.5.02), Zhifeng Xie et al. proposed a water level detection method using a virtual water gauge. The implementation steps are as follows: (1) Select the matching projection points required for perspective transformation and calculate the perspective matrix of the virtual water gauge; (2) Use the improved SegFormer-UNe segmentation network to accurately segment the water body and background in the image; (3) Determine the water level line based on their boundaries using the Canny edge detection algorithm; (4) Use the perspective transformation matrix to transform the water level line from the pixel coordinate system to the virtual water gauge coordinate system, thereby obtaining the final water level value. This method improves the accuracy of water level detection by mapping the real water gauge coordinate system to the virtual water gauge coordinate system and can adapt to some harsh scenarios, such as nighttime, dirty water gauges, or even lost water gauges. However, this method still has the following three shortcomings: First, each time a virtual water level gauge is installed, the matching projection point needs to be manually selected for calculating the perspective matrix, and the camera cannot be moved. After the camera position and angle change, recalibration is required. Second, large floating objects will affect the segmentation of the water level line, causing the water level line segmentation to deviate significantly from the true value. Third, the influence of weather on the detection process is not fully considered, and the impact of rain, fog and other conditions on the detection effect is ignored. When rain, snow and fog blur the image information, the accuracy of this method in detecting the water level is low. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the existing technology by proposing a water level monitoring system and method based on heterogeneous light field collaboration. This invention solves the problems of existing technologies, such as the need for recalibration after changes in camera position and angle; failure to consider the impact of abnormal exposure, rain, snow, and fog on the detection effect; and the fact that large floating objects can affect the segmentation of water level lines, causing the segmentation of water level lines to deviate significantly from the true value.

[0006] The technical approach to achieving the objectives of this invention is as follows: The point cloud ROI localization module in this invention replaces the manual selection and image target detection methods used in existing technologies for water level image segmentation by using a method of clustering to extract water level point clouds and segmenting water level images based on the water level point clouds and joint extrinsic parameters. This solves the problems of existing technologies where manual selection of water level images requires recalibration after changes in camera position and angle, and where image target detection has poor robustness. The method of this invention adds image enhancement operations such as exposure correction and rain, snow, and fog removal to the water level image, supplementing the lack of handling for abnormal exposure, rain, snow, and fog conditions in existing technologies. Furthermore, the method of this invention utilizes the differences in LiDAR point intensity on water, metal, and obstructions to accumulate the intensity of the water level point cloud and then obtain the water level line using the maximum gradient method. This replaces the image segmentation methods used in existing technologies for locating the water level line, solving the problem that existing technologies are easily affected by reflections and floating objects, causing the water level line to deviate from the true value.

[0007] The system of this invention includes a calibration module, an image preprocessing module, a point cloud ROI localization module, a water level localization module, and a ruler detection module. Among them,

[0008] The calibration module is used to calibrate the camera and lidar;

[0009] The point cloud ROI localization module is used to extract water level gauge point clouds through clustering and segment water level gauge images based on water level gauge point clouds and joint extrinsic parameters.

[0010] The image preprocessing module is used to perform image enhancement and then use an image segmentation algorithm to further segment the water gauge image to correct the tilted water gauge.

[0011] The water level positioning module is used to accumulate the intensity of water level point clouds and obtain the water level line through the maximum gradient method.

[0012] The ruler detection module is used to detect digital characters on the corrected water level gauge image using a target detection algorithm, then use a character recognition algorithm to perform OCR recognition on the detected digital characters, and finally calculate the water level value based on the digital characters, water level line, and water level gauge image.

[0013] The water level monitoring method of the present invention includes the following steps:

[0014] Step 1: The calibration module calibrates the camera's intrinsic parameters and performs joint calibration of the camera and LiDAR's extrinsic parameters;

[0015] Step 2: The point cloud ROI localization module extracts the water level gauge point cloud through clustering, and segments the water level gauge image based on the water level gauge point cloud and joint extrinsic parameters.

[0016] Step 3: After image enhancement by the image preprocessing module, the water level gauge image is further segmented using an image segmentation algorithm to correct the tilted water level gauge.

[0017] Step 4: The water level positioning module accumulates the intensity of the water level point cloud and obtains the water level line using the maximum gradient method.

[0018] Step 5: The ruler detection module uses a target detection algorithm to detect digital characters in the corrected water level gauge image, then uses a character recognition algorithm to perform OCR recognition on the detected digital characters, and finally calculates the water level value based on the digital characters, water level line, and water level gauge image.

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] First, the point cloud ROI positioning module in the system of the present invention is used to extract water level gauge point clouds through clustering, and segment water level gauge images based on water level gauge point clouds and joint extrinsic parameters. This overcomes the shortcomings of existing technologies that require manual selection of water level gauge images, need recalibration after changes in camera position and angle, and have poor robustness in image target detection. This invention eliminates the need for calibration during installation, and the position and pitch angle of the device can be adjusted after installation, thus improving the robustness of water level gauge image segmentation.

[0021] Secondly, the method of the present invention adds image enhancement by performing exposure correction and rain, snow and fog removal operations on the water level gauge image, which supplements the existing technology that does not consider the impact of abnormal weather conditions such as abnormal exposure, rain, snow and fog on the detection effect. This allows the present invention to process water level gauge images under complex environmental conditions, improves the accuracy and robustness of water level gauge images, and expands the versatility of water level monitoring systems and methods.

[0022] Third, the method of the present invention utilizes the differences in the intensity of lidar points on water, metal, and obstructions to accumulate the intensity of the water level point cloud and then obtain the water level line according to the maximum gradient method. This overcomes the problem that the existing technology is easily affected by reflections and floating objects, causing the water level line to deviate from the true value. This invention can avoid the influence of reflections and effectively reduce the degree of deviation of the water level line when there are floating objects. Attached Figure Description

[0023] Figure 1 This is a system block diagram of the present invention;

[0024] Figure 2 This is a flowchart of the method of the present invention;

[0025] Figure 3 This is a schematic diagram showing the placement of the calibration plate in an embodiment of the method of the present invention;

[0026] Figure 4 This is a fusion effect diagram of camera images and lidar point cloud data under different top-down angles after calibration in the simulation experiment of this invention;

[0027] Figure 5This is a simulation experiment of the point cloud ROI localization module of the present invention, showing the implementation effect of prior filtering, discrete point filtering, KDTree point cloud clustering, and water level image segmentation.

[0028] Figure 6 This is a simulation experiment of the image preprocessing module of the present invention, showing the implementation effect of the SAM image segmentation algorithm and the tilt gauge correction.

[0029] Figure 7 This is a sequence curve of the cumulative intensity values ​​of the water level positioning module in the simulation experiment of this invention;

[0030] Figure 8 This is a simulation experiment diagram of the YOLO v7 digital ruler detection module's detection effect. Detailed Implementation

[0031] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0032] Reference Figure 1 The system structure of the present invention will be further described below.

[0033] The system of the present invention includes a calibration module, a point cloud ROI positioning module, an image preprocessing module, a water level positioning module, and a ruler detection module.

[0034] The calibration module is used to calibrate the camera and lidar.

[0035] The point cloud ROI localization module is used to extract water level gauge point clouds through clustering and segment water level gauge images based on water level gauge point clouds and joint extrinsic parameters.

[0036] The image preprocessing module is used to perform image enhancement and then use an image segmentation algorithm to further segment the water gauge image to correct the tilted water gauge.

[0037] The water level positioning module is used to accumulate the intensity of water level cloud data and obtain the water level line through the maximum gradient method.

[0038] The ruler detection module is used to detect digital characters on the corrected water level gauge image using a target detection algorithm, then use a character recognition algorithm to perform OCR recognition on the detected digital characters, and finally calculate the water level value based on the digital characters, water level line, and water level gauge image.

[0039] Reference Figure 2 The implementation steps of embodiments of the method of the present invention will be further described below.

[0040] Step 1: The calibration module calibrates the camera's intrinsic parameters and performs joint calibration of the camera's and LiDAR's extrinsic parameters.

[0041] The first step is to calibrate the camera intrinsic parameters using the Zhang Zhengyou calibration method. Select 10-20 calibration images taken at different positions, angles, and postures, detect the corner points of the calibration board in each image, output the average pixel error and the pose of the calibration board relative to the camera, reduce the average pixel error by filtering images with excessive errors, and output the calibrated camera intrinsic parameter parameters.

[0042] The second step involves joint calibration of the camera and LiDAR extrinsic parameters. Using the ACSC calibration algorithm, a relatively open calibration scene is selected, and the distance between the LiDAR and the calibration board is set to more than 3 meters. The calibration board is placed at at least 10 different angles and distances, and corresponding point cloud and image data are collected. After spatiotemporal integration and feature refinement, 3D corner estimation and 2D corner detection are performed. The extrinsic parameters of the camera and LiDAR are solved by combining the 3D-2D corner data.

[0043] Reference Figure 3 The placement of the calibration plate in the method of the present invention will be further described.

[0044] In embodiments of the method of the present invention, during the calibration of camera intrinsic parameters and the joint calibration of camera and LiDAR extrinsic parameters, the placement of the calibration plate when acquiring calibration images and LiDAR point clouds is as follows: Figure 3 As shown. Figure 3 The camera and LiDAR, which need to be calibrated at the top, are mounted on the base. The calibration plates are placed starting about 3 meters away from the camera and LiDAR, with each row spaced 2 meters apart. That is, the calibration plates are placed in three positions about 3 meters away from the camera and LiDAR, five positions about 5 meters away, five positions about 7 meters away, and five positions about 9 meters away, with the plates facing the camera and LiDAR at all times.

[0045] Step 2: The point cloud ROI localization module extracts the water level gauge point cloud through clustering, and segments the water level gauge image based on the water level gauge point cloud and joint extrinsic parameters.

[0046] The steps for extracting water level gauge point clouds through clustering and then segmenting the water level gauge image based on the water level gauge point clouds and joint extrinsic parameters are as follows:

[0047] The first step is to perform prior filtering on the lidar point cloud based on the prior range of the water surface in the lidar point cloud and the reflection intensity of water and metal to obtain the point cloud near the water surface.

[0048] The second step is to perform discrete point filtering on the LiDAR point cloud. A neighbor number and threshold parameter are specified. For each point in the LiDAR point cloud, a set of neighbor points is determined by K-nearest neighbor search. For each point and its neighbors, the average distance and standard deviation between these points are calculated. According to the set threshold parameter, if the average distance between a point and its neighbors exceeds a certain multiple (determined by the threshold parameter) of the standard deviation, the point is considered a discrete point and is filtered out. The neighbor number and threshold parameter are set according to the requirements of discrete point filtering.

[0049] The third step is to perform point cloud clustering on the LiDAR point cloud after the discrete point filtering process: use any one of the KDTree, K-means and DBSCAN clustering algorithms to classify the nearest neighbor points of the point cloud, and group the points with similar spatial positional relationships into one class to obtain the water scale point cloud.

[0050] The fourth step involves using the calibrated extrinsic parameters of the camera and LiDAR to project the water level point cloud onto the image to obtain water level pixels. The convex hull of each pixel is then calculated, dilated, and the water level image is segmented.

[0051] In the embodiments of the method of the present invention, the number of neighbor points used for discrete point filtering of the lidar point cloud is 10,000 and the threshold parameter is 1;

[0052] The clustering algorithm used in the embodiments of the method of the present invention is KDTree. The characteristics of this algorithm are that it can efficiently process large-scale point cloud data and can quickly search for nearest neighbor points, thereby improving the execution efficiency of the algorithm. The KDTree clustering algorithm is used to cluster the LiDAR point cloud after filtering out discrete points. The clustering tolerance is set to 0.02, the minimum cluster size is 100, and the maximum cluster size is 25000 to extract the water level point cloud.

[0053] Step 3: After image enhancement by the image preprocessing module, the water level gauge image is further segmented using an image segmentation algorithm to correct the tilted water level gauge.

[0054] The steps for further segmenting the water gauge image using an image segmentation algorithm and correcting the tilt of the water gauge in the segmented image are as follows:

[0055] The first step is to further segment the water level indicator image using an image segmentation algorithm to obtain the tilt angle of the water level indicator in the image. The image segmentation algorithm can be any one of SAM, SAM 2, or SEEM image segmentation algorithms.

[0056] The second step is to rotate the water gauge image in the opposite direction of the tilt angle, and the rotation angle is equal to the tilt angle to obtain the corrected water gauge image.

[0057] Enhance the water level indicator image using either VECNet or DIDNet exposure correction algorithms to correct the image's exposure and reduce the impact of abnormal lighting on the detection accuracy of subsequent steps. Enhance the water level indicator image using either WGWS-Net or TransWeather severe weather image restoration algorithms to remove rain, snow, and fog from the image, further reducing the impact of weather on the detection accuracy of subsequent steps.

[0058] The image segmentation algorithm used in the embodiments of the present invention is the VECNet exposure correction algorithm. This algorithm is characterized by the accurate spatial alignment of the DIME dataset used for training and evaluation; the model is based on Retinex theory and employs a dual-stream illumination mechanism to handle overexposure and underexposure problems separately, enabling it to handle mixed exposure issues well; the end-to-end solution simplifies the exposure correction process for overexposed and underexposed images; and VECNet performs excellently in both quantitative and qualitative evaluations. Since VECNet is an end-to-end exposure correction algorithm, when using it on water level gauge images in the embodiments of the present invention, the algorithm can detect the exposure effect of the water level gauge image and then perform exposure correction for overexposed or underexposed images based on the detected exposure effect.

[0059] The image segmentation algorithm used in the embodiments of the present invention is the WGWS-Net severe weather image restoration algorithm. This algorithm is characterized by a two-stage training strategy: the first stage learns general weather features, and the second stage adaptively expands weather-specific parameters to improve performance under specific weather conditions. This adaptive expansion method avoids redundant parameter design, making the model more flexible and efficient. The end-to-end solution simplifies the severe weather image restoration process and demonstrates excellent performance on multiple benchmark datasets. Since WGWS-Net is an end-to-end severe weather image restoration algorithm, when using this algorithm on water level images in the embodiments of the present invention, the algorithm can detect the types of severe weather in the water level images and then remove rain, snow, and fog based on the detected types.

[0060] The image segmentation algorithm used in the embodiments of the present invention is the SAM image segmentation algorithm. This algorithm is characterized by its powerful image encoder, cue encoder, and lightweight mask decoder; it provides an interactive segmentation mode, allowing users to simply click or annotate regions of interest, and the algorithm automatically performs high-precision segmentation; it can finely handle complex boundaries and detailed regions, providing accurate segmentation results, especially maintaining good performance even with complex object shapes and cluttered backgrounds; SAM exhibits superior computational efficiency, completing complex image segmentation tasks in a short time. In the embodiments of the present invention, the algorithm is used to further segment the water level gauge image through the interactive segmentation mode. The center pixel of the water level gauge image and the water level gauge image itself are input into the algorithm to obtain the further segmented water level gauge image.

[0061] Step 4: The water level positioning module accumulates the intensity of the water level point cloud and obtains the water level line through the maximum gradient method.

[0062] The steps for accumulating the intensity of water level point clouds and obtaining the water level line using the maximum gradient method are as follows:

[0063] The first step is to accumulate the intensity values ​​of the water level point cloud in each plane to obtain the cumulative intensity value of the water level point cloud in that plane. The cumulative intensity values ​​of the water level point cloud in all planes form a sequence of cumulative intensity values ​​of water level point clouds. The cumulative intensity value of the water level point cloud in a plane is calculated by the following formula:

[0064]

[0065] Where SUM represents the cumulative intensity of the water level cloud, w represents the maximum value of the horizontal coordinate of the water level cloud, l represents the maximum value of the vertical coordinate of the water level cloud, I(x, y) represents the intensity value of the water level cloud at the (x, y) coordinate, x represents the horizontal coordinate of the plane, and y represents the vertical coordinate of the plane.

[0066] The second step is to calculate the gradient sequence corresponding to the cumulative intensity sequence of water level point clouds, and take the point with the maximum gradient as the water level line.

[0067] Step 5: The ruler detection module uses a target detection algorithm to detect digital characters in the corrected water level gauge image, then uses a character recognition algorithm to perform OCR recognition on the detected digital characters, and finally calculates the water level value based on the digital characters, water level line, and water level gauge image.

[0068] The use of object detection algorithms to detect numeric characters refers to using object detection algorithms to detect numeric characters on the water level gauge image and extracting the numeric character image. The object detection algorithm can be any one of the YOLO series, EfficientDet, or RetinaNet object detection algorithms.

[0069] The character recognition algorithm mentioned refers to any one of the DB, CRNN, and SRN character recognition algorithms.

[0070] The water level value calculated based on the numerical characters, water level lines, and water gauge images is calculated using the following formula:

[0071]

[0072] In this coordinate system, the top left corner of the water level gauge image is taken as the origin (0, 0), the top right corner is the positive x-axis (horizontal axis), and the bottom left corner is the positive y-axis (vertical axis); wl represents the calculated water level value, num represents the OCR recognition result of the first digit character along the opposite direction of the y-axis starting from the water level line position, and h represents the actual height of the digit character. num The pixel height of a numeric character, y wl The vertical coordinate of the water level line is y. num The vertical coordinate of the lowest point of the numeric character.

[0073] The target detection algorithm used in the embodiments of the present invention is the YOLO v7 target detection algorithm. A water level gauge image dataset was established for retraining the algorithm. Using the algorithm on water level gauge images, the numerical character images of the water level gauge scale can be detected and extracted. The YOLOv7 algorithm is characterized by its ability to identify targets of different sizes in the same image, demonstrating strong multi-scale generalization ability; it has high-speed inference ability, enabling target detection with low latency; and it maintains high accuracy while keeping the model size small and the computational complexity low. The water level gauge image dataset is characterized by being collected through internet download and on-site shooting, totaling 411 images. Internet download uses web crawling technology to download water level gauge-related images to the local machine, and higher quality sample images are selected through manual observation. On-site shooting uses video surveillance equipment to collect real scene images of multiple hydrological monitoring points, capturing high-definition images of 1920×1080 pixels. Each video image contains one or more water level gauge objects. The data comes from different scenes as much as possible, with water level gauges from different perspectives, lighting conditions, and weather conditions.

[0074] The character recognition algorithm used in the embodiments of the present invention is the DB character recognition algorithm. This algorithm is characterized by its ability to detect text of different shapes, orientations, and sizes; it generates precise character boundaries by learning a differentiable binarization threshold. Compared to traditional binarization methods, the DB algorithm automatically adjusts the threshold during training, making it easier for the model to extract character boundaries in complex backgrounds; the DB algorithm can extract fine character boundaries, especially when characters are complex in shape, tightly connected, or partially occluded, still achieving relatively accurate detection results; and its end-to-end solution simplifies the character detection process. Since DB is an end-to-end character recognition algorithm, when using this algorithm on digital character images in the embodiments of the present invention, the algorithm can perform binarization processing on the digital character image and then perform character detection based on the binarization result.

[0075] The technical effects of the present invention will be further described below in conjunction with experiments.

[0076] 1. Simulation experimental conditions:

[0077] The hardware platform for the simulation experiment of this invention is as follows: the processor is an Intel(R) Core(TM) i7-9750H CPU with a main frequency of 2.90GHz, the memory is 8GB, the camera is MV-CA050-10GM / GC, the lens is MVL-MF1228M-8MP, and the lidar is Livox Avia.

[0078] The simulation experiment software platform for this invention is: Ubuntu 18.04 operating system, MATLAB 2023b and Python 3.7.

[0079] This invention collected the lidar point cloud and image datasets required for the simulation experiment. The data collection method was to install the hardware platform on the shore, with the camera and lidar having a downward viewing angle of 15°, and the water gauge placed within the field of view of the camera and lidar. A total of 121 pairs of data were collected, which included data under 7 different conditions, as shown in Table 1, hereinafter referred to as the water gauge dataset.

[0080] Table 1. Composition of the water level gauge dataset

[0081]

[0082] 2. Simulation content and result analysis:

[0083] The present invention includes six simulation experiments.

[0084] Simulation Experiment 1 simulates the calibration effect of the calibration module.

[0085] The simulation experiment 1 of this invention is calibrated according to step 1 of the method of this invention. The average reprojection error of the calibration is 0.502 pixels. Then, the camera and the lidar are used to take data at the top angles of 0°, 15° and 30° respectively. The obtained extrinsic parameters of the camera and lidar are used to fuse the data. The fusion effect is as follows. Figure 4 As shown.

[0086] Figure 4 (a) is the fused image of the image and the lidar point cloud when the top view is 0°. Figure 4 (b) is the fused image of the image and the lidar point cloud at a top angle of 15°. Figure 4 (c) is the fused image of the image and the lidar point cloud at a top-down angle of 30°. From Figure 4 (a) Figure 4 (b) Figure 4 As can be seen from the overlap between the calibration board and the edges of the surrounding scenery in (c), the data fusion effect of the camera image and the lidar point cloud can maintain good consistency under different overhead views.

[0087] Simulation Experiment 2 simulates the effects of prior filtering, discrete point removal, KDTree point cloud clustering, and water level image segmentation on the point cloud ROI localization module. The simulation results are as follows: Figure 5 As shown.

[0088] Figure 5 (a) is a diagram showing the effect of prior filtering on the lidar point cloud of the water gauge dataset. Figure 5 (b) is the effect of discrete point filtering, where the number of neighbor points is set to 10000 and the threshold parameter is set to 1. Figure 5 (c) is the result of KDTree point cloud clustering, where the clustering tolerance is set to 0.02, the minimum cluster size is 100, and the maximum cluster size is 25000. Figure 5 (d) uses the calibrated extrinsic parameters of the camera and lidar to project the water level point cloud onto the corresponding image in the water level dataset to obtain water level pixels. The convex hull of the pixels is calculated and dilated to segment the water level image.

[0089] Simulation Experiment 3 simulates the SAM image segmentation algorithm and tilt gauge correction in the image preprocessing module. The results are as follows: Figure 6 As shown.

[0090] Figure 6 (a) is the tilted water gauge image segmented from simulation experiment 2. Figure 6 (b) is the water level image obtained after further segmentation using the SAM image segmentation algorithm. Figure 6(c) is a diagram showing the effect after the tilted water gauge has been corrected.

[0091] Simulation Experiment 4 simulates the water level positioning effect of the water level positioning module.

[0092] In simulation experiment 4 of this invention, the intensity values ​​of the water level cloud points extracted in simulation experiment 2 are accumulated for each plane, resulting in a cumulative intensity value for that plane. The cumulative intensity values ​​of all planes are then combined to form a sequence of cumulative intensity values. The relationship between the obtained cumulative intensity values ​​and the pixel coordinates of the water level cloud image is then plotted. Figure 7 .

[0093] Simulation Experiment 5 simulates the target detection, OCR recognition, and water level calculation performance of the ruler detection module. The results are as follows: Figure 8 As shown.

[0094] In simulation experiment 5 of this invention, the YOLO v7 object detection algorithm is used to detect digital characters in the corrected water level indicator image from simulation experiment 3. The detection results are as follows: Figure 8 (a) Figure 8 (b) The detected digit characters were OCR-recognized using the DB character recognition algorithm, and the water level value was calculated based on the digit characters, water level lines, and water gauge images. The detection error of the water level values ​​in the water gauge dataset is shown in Table 2.

[0095] Table 2. Overview of water level detection errors in the water gauge dataset.

[0096]

[0097] The effects of the present invention will be further described below with reference to simulation diagrams.

[0098] Figure 4 (a) Figure 4 (b) Figure 4 (c) shows the colored discrete points as the effect of projecting the lidar point cloud onto the camera image. Figure 4 (a) Figure 4 (b) Figure 4 (c) It can be seen that the shape of the calibration board checkerboard is neat and there is no misalignment. It can be judged that the calibration effect is good and changing the top angle of the camera and lidar does not affect the calibration effect.

[0099] Figure 5 (a) Figure 5 (b) Figure 5 (c) Figure 5 (d) is a graph showing the result of processing a pair of normal water gauge data according to step 2 of the method of the present invention. Figure 5(a) is the result of prior filtering of the lidar point cloud. It can be seen that there are some discrete points in the lower left. Figure 5 (b) is the result after discrete point filtering, where it can be observed that... Figure 5 (a) The discrete points in the lower left corner are filtered out; Figure 5 (c) This is the result of KDTree point cloud clustering. It can be observed that the lidar point cloud is divided into two categories. The blue part in the lower right corner is the water level point cloud. Figure 5 (d) shows the result of water level gauge image segmentation, where the water level gauge image is completely segmented.

[0100] Figure 6 (a) A water gauge image obtained by processing a pair of inclined water gauge data according to step 2 of the method of the present invention; Figure 6 (b) is the result of SAM re-segmenting the water gauge image. It can be seen that the water gauge part in the image is covered with blue as a hint. Figure 6 (c) is the result of tilt correction of the water gauge image. It can be observed that the original tilted water gauge has been corrected and cut out.

[0101] Figure 7 The horizontal axis represents the pixel coordinates of the cumulative intensity sequence of water level point clouds projected onto the water level image, in pixels. The vertical axis represents the cumulative intensity of the water level point clouds, a dimensionless scalar value. It can be observed that a significant change occurs in the cumulative intensity of the water level point clouds at 465 pixels; above 465 pixels, the cumulative intensity remains stably low.

[0102] Figure 8 (a) shows the results of using YOLO v7 to detect digit characters on the water gauge image. It can be seen that the digit characters on the water gauge were successfully detected with high confidence levels of 0.89 and 0.84, respectively. Figure 8 (b) are the numeric characters extracted after detection.

Claims

1. A complex environment water level monitoring system based on heterogeneous light field cooperation, comprising a calibration module, an image preprocessing module, a water level positioning module, and a scale detection module, characterized in that, Also include point cloud ROI positioning module; wherein, The calibration module is used for calibrating the camera and the laser radar; The point cloud ROI positioning module is used for extracting the water gauge point cloud through clustering, and the water gauge image is segmented according to the water gauge point cloud and the joint external parameter, and the steps are as follows: First, according to the prior range of the water surface in the laser radar point cloud and the reflection intensity of water and metal, the laser radar point cloud is subjected to prior filtering to obtain the point cloud near the water surface; Second, the laser radar point cloud is subjected to discrete point filtering treatment, a neighbor point number and a threshold parameter are specified, a group of neighbor points of each laser radar point cloud is determined through K nearest neighbor search, the average distance and the standard deviation between each point and its neighbor points are calculated, and according to the set threshold parameter, if the average distance of a point and its neighbor points is greater than or equal to the threshold parameter, the point is determined as a discrete point and is filtered out; the neighbor point number and the threshold parameter are set according to the requirement of discrete point filtering; Third, any one of KDTree, K-means and DBSCAN clustering algorithm is used to cluster the laser radar point cloud after the discrete point filtering treatment, points with similar spatial position relationship are classified into a class, and the water gauge point cloud is obtained; Fourth, the external parameter of the calibrated camera and laser radar is used to project the water gauge point cloud to obtain the water gauge pixel point, the convex hull of the pixel point is calculated, the water gauge image is segmented after inflation; The image preprocessing module is used for image enhancement, and the water gauge image is further segmented using an image segmentation algorithm to correct the inclined water gauge; The water level positioning module is used for intensity accumulation of the water gauge point cloud, and the water level line is obtained by the maximum gradient method, and the steps are as follows: First, the water gauge point cloud intensity value in each plane is accumulated to obtain the water gauge point cloud intensity accumulation value of the plane, and the water gauge point cloud intensity accumulation values of all planes form a water gauge point cloud intensity accumulation value sequence; the water gauge point cloud intensity accumulation value in the plane is calculated by the following formula: ; in, This represents the cumulative intensity value of water level point clouds. This represents the maximum x-coordinate of the water level point cloud. This represents the maximum value of the ordinate of the water level point cloud. Indicates in The intensity value of the water level point cloud at the coordinates. Represents the x-coordinate of the plane. Represents the ordinate of the plane; Second, the gradient sequence corresponding to the water gauge point cloud intensity accumulation value sequence is calculated, and the maximum gradient is taken as the water level line; The scale detection module is used for detecting digital characters in the corrected water gauge image using a target detection algorithm, performing OCR recognition on the detected digital characters using a character recognition algorithm, and calculating the water level value according to the digital characters, the water level line and the water gauge image.

2. A complex environment water level monitoring method based on heterogeneous light field cooperation, using the water level monitoring system of claim 1, characterized in that, The water gauge image is segmented from the water gauge point cloud and the joint external parameter; the water gauge image is further segmented using an image segmentation algorithm to correct the inclined water gauge; the steps of the water level monitoring method include the following: Step 1, the calibration module calibrates the camera internal parameter, and jointly calibrates the camera and the laser radar external parameter; Step 2, the point cloud ROI positioning module extracts the water gauge point cloud through clustering, and segments the water gauge image according to the water gauge point cloud and the joint external parameter; Step 3, the image preprocessing module performs image enhancement, and further segments the water gauge image using an image segmentation algorithm to correct the inclined water gauge; Step 4, the water level positioning module accumulates the intensity of the water gauge point cloud, and obtains the water level line by the maximum gradient method; Step 5, the ruler detection module uses a digital character target detection algorithm on the corrected water gauge image, uses a character recognition algorithm to recognize the digital characters, and then calculates the water level value according to the digital characters, water level lines and water gauge image; Step 6, the ruler detection module uses a target detection algorithm on the corrected water gauge image to detect digital characters, uses a character recognition algorithm to perform OCR recognition on the detected digital characters, and calculates the water level value according to the digital characters, water level lines and water gauge image.

3. The method according to claim 2, wherein, The steps of calibrating the camera intrinsic parameters and jointly calibrating the camera and laser radar extrinsic parameters in step 1 are as follows: First, calibrate the camera intrinsic parameters, use Zhang Zhengyou calibration method, select 10-20 calibration pictures taken at different positions, angles and postures, detect the corner points of the calibration board in each picture, output the average pixel error of calibration and the pose of the calibration board relative to the camera, reduce the average pixel error by screening pictures with too large error, and output the calibrated camera intrinsic parameters; Second, jointly calibrate the camera and laser radar extrinsic parameters, use ACSC calibration algorithm, select a relatively open calibration scene, set the laser radar to reach the calibration board at a distance of more than 3 meters, select at least 10 different angles and distances to place the calibration board, collect the corresponding point cloud and image data, perform time-space integration and feature refinement, and then perform 3D corner point estimation and 2D corner point detection, and solve the extrinsic parameters of the camera and laser radar according to the 3D-2D corner point data.

4. The method of claim 2, wherein, The steps of segmenting the water gauge image according to the water gauge point cloud and joint extrinsic parameters in step 2 are as follows: First, according to the prior range of water surface in laser radar point cloud and the reflection intensity of water and metal, the laser radar point cloud is pre-filtered to obtain the point cloud near the water surface; Second, the laser radar point cloud is processed by discrete point filtering, a neighbor point number and a threshold parameter are specified, the neighbor points of each laser radar point cloud are determined through K nearest neighbor search, the average distance and standard deviation between each point and its neighbor points are calculated, and according to the set threshold parameter, if the average distance between a point and its neighbor points is greater than or equal to the threshold parameter, the point is determined as a discrete point and is filtered out; the neighbor point number and the threshold parameter are set according to the requirement of discrete point filtering; Third, use any one of KDTree, K-means and DBSCAN clustering algorithm to cluster the laser radar point cloud after discrete point filtering, and classify points with similar spatial position relationship into a class to obtain the water gauge point cloud; Fourth, use the extrinsic parameters of the calibrated camera and laser radar to project the water gauge point cloud and obtain the water gauge pixel points, calculate the convex hull of the pixel points, dilate the convex hull, and segment the water gauge image.

5. The method of claim 2, wherein, The image enhancement in step 3 refers to using any one of VECNet and DIDNet exposure correction algorithms to enhance the water gauge image, correct the exposure effect of the image, and reduce the influence of abnormal light on the detection accuracy of the subsequent steps.

6. The method of claim 2, wherein, The steps of further segmenting the water gauge image using image segmentation algorithm and correcting the inclined water gauge after segmentation in step 3 are as follows: In the first step, the image segmentation algorithm is used to further segment the water gauge image to obtain the inclination angle of the water gauge in the water gauge image, and the image segmentation algorithm is any one of SAM, SAM 2, and SEEM image segmentation algorithm. In the second step, the water gauge image is rotated in the reverse direction along the inclination angle to obtain the corrected water gauge image, wherein the rotation angle is equal to the inclination angle.

7. The method of claim 2, wherein, In step 4, the intensity accumulation of the water gauge point cloud is performed, and the water level line is obtained by the maximum gradient method. The steps are as follows: In the first step, the intensity values of the water gauge point cloud in each plane are accumulated to obtain the intensity accumulation value of the water gauge point cloud in the plane, and the intensity accumulation values of the water gauge point cloud in all planes form a sequence of intensity accumulation values of the water gauge point cloud. The intensity accumulation value of the water gauge point cloud in the plane is calculated by the following formula: ; in, This represents the cumulative intensity value of water level point clouds. This represents the maximum x-coordinate of the water level point cloud. This represents the maximum value of the ordinate of the water level point cloud. Indicates in The intensity value of the water level point cloud at the coordinates. Represents the x-coordinate of the plane. Represents the ordinate of the plane; In the second step, the gradient sequence corresponding to the sequence of intensity accumulation values of the water gauge point cloud is calculated, and the maximum gradient is taken as the water level line.

8. The method of claim 2, wherein, In step 5, the target detection algorithm is used to detect the digital character, which means that the target detection algorithm is used to detect the water gauge scale digital character on the water gauge image, and the digital character image is extracted, wherein the target detection algorithm can use any one of YOLO series, EfficientDet, and RetinaNet target detection algorithm.

9. The method of claim 2, wherein, The character recognition algorithm in step 5 refers to any one of DB, CRNN, and SRN character recognition algorithm.

10. The method of claim 2, wherein, In step 5, the water level value is calculated according to the digital character, the water level line, and the water gauge image, which is calculated by the following formula: ; wherein, represents the upper left corner of the water gauge image as the origin coordinate (0, 0), and the right upper corner direction as the horizontal axis represents the positive direction of the axis, and the lower left corner as the vertical axis represents the positive direction of the axis, and the calculated water level value, represents the vertical coordinate of the water level line from the water level line position, represents the OCR recognition result of the first numerical character in the negative direction of the axis, represents the actual height of the numerical character, represents the pixel height of the numerical character, represents the vertical coordinate of the water level line, represents the vertical coordinate of the lowest point of the numerical character.

Images