Complex environment water level monitoring system and method based on heterogeneous light field cooperation

Abstract

The invention discloses a complex environment water level monitoring system and method based on heterogeneous light field cooperation. A water gauge image is segmented through a water gauge point cloud and combined external parameters; after image enhancement, the water gauge image is further segmented, and an inclined water gauge is corrected; performing intensity accumulation on the water gauge point cloud, and obtaining a water line through a maximum gradient method; and detecting numeric characters of the corrected water gauge image by using a target detection algorithm, performing OCR (Optical Character Recognition) on the detected numeric characters by using a character recognition algorithm, and calculating a water level value according to the numeric characters, the water level line and the water gauge image. The laser radar and the visible light field are combined, and the reflection characteristics of different materials are utilized, so that water level real-time monitoring in complex environments of rivers, reservoirs and coast can be realized. Interference of floating objects and weather factors on water level detection is effectively avoided, and accuracy and robustness of water level monitoring are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of water level detection technology, and further relates to a complex environment water level monitoring system and method based on heterogeneous light field collaboration. The present invention can be used for real-time monitoring of water levels in rivers, reservoirs or coasts, and provide accurate data for formulating drought and flood prevention measures. Background Art

[0002] Real-time monitoring of water levels can provide important basis for the formulation of drought and flood prevention measures, river management planning, reservoir scheduling, water resources management, etc. There are two types of existing water level monitoring devices: contact and non-contact, both of which have certain limitations. For example, the contact float water level gauge requires a hydrostatic device; the non-contact ultrasonic and radar water level gauges require regular calibration and have high installation requirements. The image processing method has low installation requirements, but the effect is poor in complex environments. The general process of water level detection using image recognition methods is: water gauge segmentation, tilt correction, water level line positioning, character cutting, OCR detection, and water level calculation. Under ideal conditions, this method performs well. However, the images collected in natural waters are complex and diverse, and many schemes ignore exposure anomalies, rain, snow, fog, occlusion, etc. However, this is inevitable in practical applications, and it will cause different types and degrees of image degradation, which further makes it difficult for these schemes to obtain more accurate data in harsh environments and cannot be put into practical application. In response to the above problems, the present invention proposes a complex environment water level monitoring system and method based on heterogeneous light field collaboration. The water level monitoring system constructs a collaborative processing mechanism based on radar laser field and visible light field according to the reflection characteristics of lidar on different material media such as metal and water, which can make up for the shortcomings of water level detection using only image processing.

[0003] Chongqing Zhixing Digital Intelligent Technology Co., Ltd. disclosed a method for detecting water level in a river without a water gauge based on deep learning in its applied patent document "A method for detecting water level in a river without a water gauge based on deep learning" (patent application number CN202311379848.5, application publication number CN117974962A). The implementation steps of the method are as follows: (1) setting a calibration object at the water level detection point and obtaining an image containing the calibration object from the camera, and obtaining the mapping relationship between the pixel coordinate system and the world coordinate system through calibration; (2) using the SAM model to segment the water body area and the non-water body area of ​​the target water area and obtain a binary image; (3) cutting out the neighborhood of the water level calibration point on the binary image, and obtaining the water level segmentation line on the image through image processing; (4) binarizing the single-frame image obtained by the pull stream, obtaining the region of interest near the water level, dividing the region of interest into multiple sub-regions of interest, and calculating the Laplace value; (5) comparing the sub-regions of interest, determining the region where the Laplace value suddenly changes, which is intended to be the water level line region; (6) determining the position of the water level line by comparing steps three and five, and calculating the water level height through the mapping relationship. The method can use the calibration object to replace the water gauge, and obtain the mapping relationship between the pixel coordinate system and the world coordinate system after calibration, which solves the dependence on the physical water gauge to a certain extent, and can realize water level detection when the calibration object is blocked. However, this method still has the following three shortcomings: first, the size of the calibration object needs to be known, and the camera cannot be moved, and the camera needs to be recalibrated after the position and angle change; second, when there are large floating objects or attachments, the identified water level line will move downward, causing the detection value to be lower than the true value; third, the impact of weather on the detection process is not considered, and the impact of image degradation caused by abnormal exposure, rain, snow, fog, etc. on the detection accuracy is ignored.

[0004] Zhifeng Xie et al. proposed a water level detection method using a virtual water gauge in their paper "Application of Deep Learning Techniques in Water Level Measurement: Combining Improved SegFormer-UNet Model with Virtual Water Gauge" (Applied Sciences 2023.5.02). The implementation steps of this method are as follows: (1) Select the matching projection points required for perspective transformation and calculate the perspective matrix of the virtual water gauge; (2) Use the improved SegFormer-UNet segmentation network to accurately segment the water body and background in the image; (3) Use the Canny edge detection algorithm to determine the water level line based on their boundaries; (4) Use the perspective transformation matrix to transform the water level line from the pixel coordinate system to the virtual water gauge coordinate system to obtain the final water level value. This method improves the accuracy of water level detection by mapping the real water gauge coordinate system to the virtual water gauge coordinate system, and can adapt to some harsh scenes, such as night, dirty water gauge or even lost water gauge. However, this method still has the following three shortcomings: first, each time the virtual water gauge is installed, it is necessary to manually select matching projection points for calculating the perspective matrix, and the camera cannot be moved. The camera position and angle need to be recalibrated after changes; second, large floating objects will affect the segmentation of the water level line, causing the division of the water level line to deviate significantly from the true value; third, the impact of weather on the detection process is not fully considered, and the impact of rain, fog and other conditions on the detection effect is ignored. When rain, snow and fog blur the image information, the accuracy of water level detection by this method is low. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a water level monitoring system and method based on heterogeneous light field collaboration, so as to solve the deficiencies of the prior art that the camera position and angle need to be recalibrated after changes; the influence of abnormal exposure, rain, snow and fog on the detection effect is not considered; and large-volume floating objects will affect the segmentation of the water level line, causing the division of the water level line to deviate significantly from the true value.

[0006] The technical idea for achieving the purpose of the present invention is that the point cloud ROI positioning module in the system of the present invention extracts the water gauge point cloud by clustering, and the method of segmenting the water gauge image according to the water gauge point cloud and the joint external reference replaces the manual selection and image target detection method used in the prior art when segmenting the water gauge image, solving the problem that the prior art manually selects the water gauge image, the camera position and angle need to be recalibrated after the change, and the image target detection has poor robustness. The method of the present invention adds the image enhancement of exposure correction and rain, snow and fog removal operations on the water gauge image, which supplements the processing of abnormal exposure, rain, snow and fog that the prior art lacks. The method of the present invention utilizes the difference in the intensity of the laser radar point on water, metal and obstructions, accumulates the intensity of the water gauge point cloud, and then obtains the water level line according to the maximum gradient method, replacing the image segmentation method used in the prior art when locating the water level line, solving the problem that the prior art is easily affected by reflections and floating objects, causing the water level line to deviate from the true value.

[0007] The system of the present invention includes a calibration module, an image preprocessing module, a point cloud ROI positioning module, a water level positioning module, and a ruler detection module.

[0008] The calibration module is used to calibrate the camera and the laser radar;

[0009] The point cloud ROI positioning module is used to extract the water gauge point cloud by clustering, and segment the water gauge image according to the water gauge point cloud and the joint external parameters;

[0010] The image preprocessing module is used to further segment the water gauge image using an image segmentation algorithm after image enhancement, and to correct the tilted water gauge;

[0011] The water level positioning module is used to accumulate the strength of the water gauge point cloud and obtain the water level line through the maximum gradient method;

[0012] The ruler detection module is used to detect digital characters on the corrected water gauge image using a target detection algorithm, then use a character recognition algorithm to perform OCR recognition on the detected digital characters, and then calculate the water level value based on the digital characters, water level line and water gauge image.

[0013] The steps of the water level monitoring method of the present invention include the following:

[0014] Step 1: The calibration module calibrates the camera's intrinsic parameters and jointly calibrates the camera and lidar's extrinsic parameters;

[0015] Step 2: The point cloud ROI positioning module extracts the water gauge point cloud by clustering, and segments the water gauge image according to the water gauge point cloud and the joint external parameters;

[0016] Step 3: After the image preprocessing module performs image enhancement, the image segmentation algorithm is used to further segment the water gauge image and correct the tilted water gauge;

[0017] Step 4: The water level positioning module accumulates the intensity of the water gauge point cloud and obtains the water level line through the maximum gradient method;

[0018] Step 5: The ruler detection module uses the target detection algorithm to detect digital characters on the corrected water gauge image, and then uses the character recognition algorithm to perform OCR recognition on the detected digital characters, and then calculates the water level value based on the digital characters, water level line and water gauge image.

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] First, the point cloud ROI positioning module in the system of the present invention is used to extract water gauge point clouds through clustering, and segment the water gauge image according to the water gauge point clouds and joint external parameters, which overcomes the shortcomings of the prior art of manually selecting water gauge images, requiring recalibration after changes in camera position and angle, and poor robustness of image target detection. The present invention does not require calibration during installation, and the position and pitch angle of the equipment can be adjusted after installation, thereby improving the robustness of water gauge image segmentation.

[0021] Second, the method of the present invention adds image enhancement for exposure correction and rain, snow, and fog removal operations on the water level gauge image, supplementing the fact that the prior art does not consider the impact of abnormal exposure, rain, snow, fog and other abnormal climatic conditions on the detection effect, so that the present invention can process water level gauge images under complex environmental conditions, improve the accuracy and robustness of the water level gauge image, and at the same time expand the versatility of the water level monitoring system and method.

[0022] Third, the method of the present invention utilizes the differences in the intensity of lidar points on water, metal, and obstructions, accumulates the intensity of the water gauge point cloud, and then obtains the water level line according to the maximum gradient method. This overcomes the problem of the prior art that the water level line is easily affected by reflections and floating objects, causing the water level line to deviate from the true value. The present invention can avoid the influence of reflections and effectively reduce the degree of deviation of the water level line when floating objects are present. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a system block diagram of the present invention;

[0024] Figure 2 is a flow chart of the method of the present invention;

[0025] Figure 3 is a schematic diagram of the placement of the calibration plate in an embodiment of the method of the present invention;

[0026] Figure 4 It is a fusion effect diagram of camera image and laser radar point cloud data at different bird's-eye view angles after calibration in the simulation experiment of the present invention;

[0027] Figure 5It is the effect diagram of the implementation of prior filtering, discrete point filtering, KDTree point cloud clustering, and water gauge image segmentation of the point cloud ROI positioning module of the simulation experiment of the present invention;

[0028] Figure 6 It is a SAM image segmentation algorithm and a tilted water gauge correction implementation effect diagram of the image preprocessing module of the simulation experiment of the present invention;

[0029] Figure 7 It is a strength cumulative value sequence curve diagram of the water level positioning module of the simulation experiment of the present invention;

[0030] Figure 8 This is a digital ruler detection effect diagram of the ruler detection module YOLO v7 in the simulation experiment of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0032] Reference Figure 1 , the system structure of the present invention is further described.

[0033] The system of the present invention comprises a calibration module, a point cloud ROI positioning module, an image preprocessing module, a water level positioning module and a ruler detection module.

[0034] The calibration module is used to calibrate the camera and the laser radar.

[0035] The point cloud ROI positioning module is used to extract the water gauge point cloud by clustering, and segment the water gauge image according to the water gauge point cloud and the joint external parameters.

[0036] The image preprocessing module is used to further segment the water gauge image using an image segmentation algorithm after image enhancement, and to correct the tilted water gauge.

[0037] The water level positioning module is used to accumulate the intensity of the water gauge point cloud and obtain the water level line through the maximum gradient method.

[0038] The ruler detection module is used to detect digital characters on the corrected water gauge image using a target detection algorithm, then use a character recognition algorithm to perform OCR recognition on the detected digital characters, and then calculate the water level value based on the digital characters, water level line and water gauge image.

[0039] Reference Figure 2 , further describing the implementation steps of the embodiment of the method of the present invention.

[0040] Step 1: The calibration module calibrates the camera intrinsic parameters and jointly calibrates the camera and lidar extrinsic parameters.

[0041] The first step is to calibrate the camera intrinsic parameters. Using Zhang Zhengyou's calibration method, select 10-20 calibration pictures taken at different positions, angles, and postures, detect the corner points of the calibration plate in each picture, output the calibration average pixel error and the position of the calibration plate relative to the camera, reduce the average pixel error by screening pictures with excessive errors, and output the camera intrinsic parameters after calibration;

[0042] The second step is to jointly calibrate the external parameters of the camera and lidar. Use the ACSC calibration algorithm, select a relatively open calibration scene, set the lidar to be more than 3 meters away from the calibration plate, select at least 10 different angles and distances to place the calibration plate, collect the corresponding point cloud and image data, perform spatiotemporal integration and feature refinement, perform 3D corner point estimation and 2D corner point detection, and combine the 3D-2D corner point data to solve the external parameter parameters of the camera and lidar.

[0043] Reference Figure 3 , the placement position of the calibration plate in the method of the present invention is further described.

[0044] In the embodiment of the method of the present invention, when the camera intrinsic parameter calibration and the camera and laser radar extrinsic parameter joint calibration are performed, the calibration plate is placed in the following manner when the calibration image and the laser radar point cloud are collected: Figure 3 shown. Figure 3 The top camera and lidar to be calibrated are installed on the base. The calibration plates are placed starting from about 3 meters away from the camera and lidar, with each row spaced 2 meters apart, that is, the calibration plates are placed at 3 positions about 3 meters away from the camera and lidar, 5 positions at 5 meters, 5 positions at 7 meters, and 5 positions at 9 meters, and the placement angles are all facing the camera and lidar.

[0045] Step 2: The point cloud ROI positioning module extracts the water gauge point cloud through clustering, and segments the water gauge image based on the water gauge point cloud and joint external parameters.

[0046] The steps of extracting the water gauge point cloud by clustering and then segmenting the water gauge image according to the water gauge point cloud and the joint external reference are as follows:

[0047] The first step is to filter the LiDAR point cloud based on the prior range of the water surface in the LiDAR point cloud and the reflection intensity of water and metal to obtain the point cloud near the water surface.

[0048] The second step is to perform discrete point filtering on the laser radar point cloud. A number of neighbor points and a threshold parameter are specified. For each point in the laser radar point cloud, a set of neighbor points of the point is determined through K nearest neighbor search. For each point and its neighbor points, the average distance and standard deviation between these points are calculated. According to the set threshold parameter, if the average distance between a point and its neighbor points exceeds a certain multiple (determined by the threshold parameter) of the standard deviation, the point is considered to be a discrete point and is filtered out. The number of neighbor points and the threshold parameter are set according to the requirements for discrete point filtering.

[0049] The third step is to cluster the laser radar point cloud after the discrete point filtering process: use any of the KDTree, K-means and DBSCAN clustering algorithms to classify the neighboring points of the point cloud, classify the points with similar spatial position relationships into one category, and obtain the water level point cloud;

[0050] The fourth step is to use the calibrated external parameters of the camera and lidar to project the water gauge point cloud onto the image to obtain the water gauge pixel points, calculate the convex hull of the pixel points, and then expand and segment the water gauge image.

[0051] In the embodiment of the method of the present invention, the number of neighbor points used for discrete point filtering of the laser radar point cloud is 10000 and the threshold parameter is 1;

[0052] The clustering algorithm used in the embodiment of the method of the present invention is KDTree, which is characterized by being able to efficiently process large-scale point cloud data and quickly search for neighboring points, thereby improving the execution efficiency of the algorithm. The KDTree clustering algorithm is used to cluster the lidar point cloud after discrete points are filtered out, and the clustering tolerance is set to 0.02, the minimum cluster size is 100, and the maximum cluster size is 25000 to extract the water level point cloud.

[0053] Step 3: After the image preprocessing module performs image enhancement, the image segmentation algorithm is used to further segment the water gauge image and correct the tilted water gauge.

[0054] The steps of further segmenting the water gauge image using the image segmentation algorithm and correcting the tilt of the water gauge image after segmentation are as follows:

[0055] In the first step, an image segmentation algorithm is used to further segment the water gauge image to obtain the tilt angle of the water gauge in the water gauge image. The image segmentation algorithm is any one of SAM, SAM 2, and SEEM image segmentation algorithms.

[0056] In the second step, the water gauge image is rotated in the opposite direction of the tilt angle, and the rotation angle is equal to the tilt angle to obtain the corrected water gauge image.

[0057] Use any exposure correction algorithm in VECNet or DIDNet to enhance the water gauge image, correct the exposure effect of the image, and reduce the impact of abnormal lighting on the detection accuracy of subsequent steps. Use any severe weather image restoration algorithm in WGWS-Net or TransWeather to enhance the water gauge image, remove rain, snow, and fog in the image, and reduce the impact of weather on the detection accuracy of subsequent steps.

[0058] The image segmentation algorithm used in the embodiment of the method of the present invention is the VECNet exposure correction algorithm, which is characterized in that the DIME data set used for training and evaluation has precise spatial alignment; the model is based on the Retinex theory and adopts a dual-stream illumination mechanism to handle overexposure and underexposure problems respectively, which enables the model to handle mixed exposure problems well; the end-to-end solution simplifies the exposure correction process of overexposed and underexposed images; VECNet performs well in both quantitative indicators and qualitative evaluation. Since VECNet is an end-to-end exposure correction algorithm, when the algorithm is used on the water gauge image in the embodiment of the method of the present invention, the algorithm can detect the exposure effect of the water gauge image, and then perform exposure correction for the overexposed or underexposed image based on the detected exposure effect.

[0059] The image segmentation algorithm used in the embodiment of the method of the present invention is the WGWS-Net severe weather image restoration algorithm, which is characterized by adopting a two-stage training strategy. The first stage learns general weather features, and the second stage improves the performance under specific weather conditions by adaptively expanding weather-specific parameters. This adaptive expansion method avoids the design of redundant parameters, making the model more flexible and efficient; the end-to-end solution simplifies the severe weather image restoration process; and performs well on multiple benchmark data sets. Since WGWS-Net is an end-to-end severe weather image restoration algorithm, when the algorithm is used on the water gauge image in the embodiment of the method of the present invention, the algorithm can detect the severe weather type of the water gauge image, and then remove rain, snow, and fog according to the detected severe weather type.

[0060] The image segmentation algorithm used in the embodiment of the method of the present invention is the SAM image segmentation algorithm, which is characterized by having a powerful image encoder, a prompt encoder and a lightweight mask decoder; providing an interactive segmentation method, the user only needs to click or mark the area of ​​interest, and the algorithm can automatically perform high-precision segmentation; it can finely process complex boundaries and detail areas, and provide accurate segmentation results, especially when the object shape is complex and the background is chaotic; SAM is superior in computational efficiency and can complete complex image segmentation tasks in a short time. In the embodiment of the method of the present invention, the algorithm is used to further segment the water gauge image through an interactive segmentation method, and the central pixel point of the water gauge image and the water gauge image are input into the algorithm to obtain the further segmented water gauge image.

[0061] Step 4: The water level positioning module accumulates the intensity of the water gauge point cloud and obtains the water level line through the maximum gradient method.

[0062] The steps of performing intensity accumulation on the water gauge point cloud and obtaining the water level line by the maximum gradient method are as follows:

[0063] The first step is to accumulate the water gauge point cloud intensity values ​​in each plane of the water gauge point cloud to obtain the accumulated intensity value of the water gauge point cloud of the plane. The accumulated intensity values ​​of the water gauge point clouds of all planes constitute the accumulated intensity value sequence of the water gauge point cloud. The accumulated intensity value of the water gauge point cloud in the plane is calculated by the following formula:

[0064]

[0065] Among them, SUM represents the cumulative intensity value of the water gauge point cloud, w represents the maximum value of the horizontal coordinate of the water gauge point cloud, l represents the maximum value of the vertical coordinate of the water gauge point cloud, I(x, y) represents the intensity value of the water gauge point cloud at the (x, y) coordinate, x represents the horizontal coordinate of the plane, and y represents the vertical coordinate of the plane;

[0066] The second step is to calculate the gradient sequence corresponding to the cumulative value sequence of the water gauge point cloud intensity, and take the point with the maximum gradient as the water level line.

[0067] Step 5: The ruler detection module uses the target detection algorithm to detect digital characters on the corrected water gauge image, and then uses the character recognition algorithm to perform OCR recognition on the detected digital characters, and then calculates the water level value based on the digital characters, water level line and water gauge image.

[0068] The use of a target detection algorithm to detect digital characters refers to: using a target detection algorithm to detect digital characters on the water gauge image and extracting a digital character image, wherein the target detection algorithm can use any one of the YOLO series, EfficientDet, and RetinaNet target detection algorithms.

[0069] The character recognition algorithm refers to any one of the DB, CRNN, and SRN character recognition algorithms.

[0070] The water level value calculated based on the digital characters, water level line and water gauge image is calculated by the following formula:

[0071]

[0072] The coordinate system takes the upper left corner of the water gauge image as the origin coordinate (0, 0), the upper right corner is the positive direction of the x-axis (horizontal axis), and the lower left corner is the positive direction of the y-axis (vertical axis); wl represents the calculated water level value, num represents the OCR recognition result of the first numeric character along the reverse direction of the y-axis starting from the water level line position, h represents the actual height of the numeric character, and h num Indicates the pixel height of the digital character, y wl Indicates the ordinate of the water level, y num Indicates the vertical coordinate of the lowest point of the numeric character.

[0073] The target detection algorithm used in the embodiment of the method of the present invention is the YOLO v7 target detection algorithm. A water gauge image data set is established for retraining the algorithm. The algorithm is used on the water gauge image to detect and extract the digital character image of the water gauge scale. The YOLOv7 algorithm is characterized in that it can identify targets of different sizes in the same image, showing a strong multi-scale generalization ability; it has high-speed reasoning ability and can achieve target detection under low latency; while ensuring high precision, it maintains a small model size and low computational complexity. The water gauge image data set is characterized in that it is collected by downloading from the Internet and shooting on site, with a total of 411 images; the Internet download uses crawler technology to download images related to the water gauge to the local, and selects sample images with better quality through manual observation; the on-site shooting mode uses video monitoring equipment to collect real scene images of multiple hydrological monitoring points, collects high-definition images of 1920×1080 pixels, and each video image contains one or more water gauge objects; the data comes from different scenes as much as possible, and there are water gauges under different viewing angles, different lighting and different weather conditions.

[0074] The character recognition algorithm used in the embodiment of the method of the present invention is a DB character recognition algorithm, which is characterized in that it can detect texts of different shapes, directions and sizes; it generates precise character boundaries by learning a differentiable binarization threshold. Compared with the traditional binarization method, the DB algorithm can automatically adjust the threshold during the training process, making it easier for the model to extract character boundaries under complex backgrounds; the DB algorithm can extract fine character boundaries, especially when the characters are complex in shape, closely connected or partially blocked, and can still obtain relatively accurate detection results; the end-to-end solution simplifies the character detection process. Since DB is an end-to-end character recognition algorithm, when the algorithm is used on a digital character image in the embodiment of the method of the present invention, the algorithm can binarize the digital character image and then perform character detection based on the binarization result.

[0075] The technical effects of the present invention are further described below in conjunction with experiments.

[0076] 1. Simulation experiment conditions:

[0077] The hardware platform of the simulation experiment of the present invention is: the processor is Inter(R)Core(TM)i7-9750H CPU, the main frequency is 2.90GHz, the memory is 8GB, the camera is MV-CA050-10GM / GC, the lens is MVL-MF1228M-8MP, and the laser radar is Livox Avia.

[0078] The simulation experiment software platform of the present invention is: Ubuntu18.04 operating system, MATLAB 2023b and python3.7.

[0079] The present invention collects the laser radar point cloud and image data sets required for the simulation experiment. The collection method is as follows: the hardware platform is installed on the shore, the camera and laser radar have a downward viewing angle of 15°, and the water level gauge is placed within the field of view of the camera and laser radar. A total of 121 pairs of data are collected, which include data under 7 different conditions, as shown in Table 1, hereinafter referred to as the water level gauge data set.

[0080] Table 1. Composition of water gauge dataset

[0081]

[0082] 2. Simulation content and result analysis:

[0083] There are six simulation experiments of the present invention.

[0084] Simulation experiment 1, simulation of the calibration effect of the calibration module.

[0085] The simulation experiment 1 of the present invention is calibrated according to step 1 of the method of the present invention, and the average reprojection error of the calibration is 0.502pixel. Then, a camera and a laser radar are used to respectively shoot data with a pair of camera and laser radar with a top view of 0°, 15°, and 30°, and the obtained external parameter parameters of the camera and laser radar are used to fuse the data. The fusion effect is shown in FIG. Figure 4 shown.

[0086] Figure 4 (a) is the data fusion image of the image and the lidar point cloud when the top view angle is 0°. Figure 4 (b) is the data fusion image of the image and the lidar point cloud when the top view angle is 15°. Figure 4 (c) is the data fusion image of the image and the lidar point cloud when the top view angle is 30°. Figure 4 (a) Figure 4 (b) Figure 4 The overlap effect of the edges of the calibration plate and the surrounding scenery in (c) shows that the data fusion effect of the camera image and the lidar point cloud can maintain good consistency at different bird's-eye view angles.

[0087] Simulation experiment 2 simulates the implementation effects of prior filtering, discrete point filtering, KDTree point cloud clustering, and water level image segmentation of the point cloud ROI positioning module. The simulation results are shown in the figure below. Figure 5 shown.

[0088] Figure 5 (a) is a diagram showing the effect of prior filtering on the lidar point cloud of the water gauge dataset. Figure 5 (b) is the effect diagram of discrete point filtering, where the number of neighbor points is set to 10,000 and the threshold parameter is set to 1. Figure 5 (c) is the effect diagram after KDTree point cloud clustering processing, where the clustering tolerance is set to 0.02, the minimum cluster size is 100, and the maximum cluster size is 25000. Figure 5 (d) is to use the extrinsic parameters of the calibrated camera and lidar to project the water level gauge point cloud onto the corresponding image in the water level gauge data set to obtain water level gauge pixel points, calculate the convex hull of the pixel points, perform expansion, and then segment the water level gauge image.

[0089] Simulation experiment 3 simulates the SAM image segmentation algorithm and tilt scale correction of the image preprocessing module. The results are as follows: Figure 6 shown.

[0090] Figure 6 (a) is the inclined water gauge image segmented from simulation experiment 2. Figure 6 (b) is the water gauge image obtained after further segmentation using the SAM image segmentation algorithm. Figure 6(c) is the effect diagram after the tilted water gauge is corrected.

[0091] Simulation experiment 4 is a simulation of the water level positioning effect of the water level positioning module.

[0092] The simulation experiment 4 of the present invention accumulates the intensity values ​​of the water gauge point cloud in each plane extracted by the simulation experiment 2 to obtain the accumulated intensity value of the water gauge point cloud of the plane. The accumulated intensity values ​​of the water gauge point cloud of all planes constitute the accumulated intensity value sequence of the water gauge point cloud. The relationship between the accumulated intensity value and the pixel coordinates of the water gauge image is plotted as Figure 7 .

[0093] Simulation experiment 5 simulates the target detection, OCR recognition, and water level value calculation effects of the ruler detection module. The results are as follows: Figure 8 shown.

[0094] In the simulation experiment 5 of the present invention, the YOLO v7 target detection algorithm is used to detect digital characters on the water gauge image corrected in the simulation experiment 3. The detection effect is referenced to Figure 8 (a) Figure 8 (b) Use DB character recognition algorithm to perform OCR recognition on the detected digital characters, and calculate the water level value based on the digital characters, water level line and water gauge image. The detection error of the water level value of the water gauge data set is shown in Table 2.

[0095] Table 2. Water gauge data set, water level value detection error list

[0096]

[0097] The effects of the present invention are further described below in conjunction with simulation diagrams.

[0098] Figure 4 (a) Figure 4 (b) Figure 4 The colored discrete points in (c) are the effect of projecting the laser radar point cloud onto the camera image. Figure 4 (a) Figure 4 (b) Figure 4 (c) It can be seen that the shape of the calibration board chessboard is neat and there is no misalignment. It can be judged that the calibration effect is good and changing the camera and lidar's top-down angle does not affect the calibration effect.

[0099] Figure 5 (a) Figure 5 (b) Figure 5 (c) Figure 5 (d) is a result diagram of a pair of normal water gauge data processed according to step 2 of the method of the present invention. Figure 5(a) is the result of a priori filtering of the lidar point cloud, and it can be seen that there are some discrete points in the lower left; Figure 5 (b) is the result after discrete point filtering. It can be observed that Figure 5 (a) The discrete points on the lower left are filtered out; Figure 5 (c) is the result of KDTree point cloud clustering processing. It can be observed that the lidar point cloud is divided into two categories. The blue part in the lower right is the water level point cloud; Figure 5 (d) is the result of water gauge image segmentation. It can be seen that the water gauge image is completely segmented.

[0100] Figure 6 (a) is a water gauge image obtained by processing a pair of inclined water gauge data according to step 2 of the method of the present invention; Figure 6 (b) is the result of SAM segmenting the water gauge image again. It can be seen that the water gauge part in the image is covered with blue as a hint; Figure 6 (c) is the result image after the tilt correction of the water ruler image. It can be seen that the original tilted water ruler is corrected and cut out.

[0101] Figure 7 The horizontal axis represents the pixel coordinates of the water gauge point cloud intensity accumulation value sequence projected on the water gauge image, in pixels, and the vertical axis represents the water gauge point cloud intensity accumulation value, which is a unitless scalar value. It can be observed that the water gauge point cloud intensity accumulation value has changed significantly at 465 pixels, and when it is greater than 465 pixels, the water gauge point cloud intensity accumulation value is stable at a low level.

[0102] Figure 8 (a) is the result of using YOLO v7 to detect digital characters on the water gauge image. It can be seen that the digital characters on the water gauge are successfully detected with high confidence levels of 0.89 and 0.84 respectively. Figure 8 (b) shows the digital characters extracted after detection.

Claims

1. A complex environment water level monitoring system based on heterogeneous light field collaboration, including a calibration module, an image preprocessing module, a water level positioning module, and a scale detection module, characterized in that: It also includes a point cloud ROI positioning module; The calibration module is used to calibrate the camera and the laser radar; The point cloud ROI positioning module is used to extract the water gauge point cloud by clustering, and segment the water gauge image according to the water gauge point cloud and the joint external parameters; The image preprocessing module is used to further segment the water gauge image using an image segmentation algorithm after image enhancement, and to correct the tilted water gauge; The water level positioning module is used to accumulate the strength of the water gauge point cloud and obtain the water level line through the maximum gradient method; The ruler detection module is used to detect digital characters on the corrected water gauge image using a target detection algorithm, then use a character recognition algorithm to perform OCR recognition on the detected digital characters, and then calculate the water level value based on the digital characters, water level line and water gauge image.

2. A complex environment water level monitoring method based on heterogeneous light field synergy using the water level monitoring system of claim 1, characterized in that: The water gauge image is segmented by using the water gauge point cloud and the joint external parameter; the water gauge image is further segmented by using the image segmentation algorithm, and the inclined water gauge is corrected; the steps of the water level monitoring method include the following: Step 1: The calibration module calibrates the camera's intrinsic parameters and jointly calibrates the camera and lidar's extrinsic parameters; Step 2: The point cloud ROI positioning module extracts the water gauge point cloud by clustering, and segments the water gauge image according to the water gauge point cloud and the joint external parameters; Step 3: After the image preprocessing module performs image enhancement, the image segmentation algorithm is used to further segment the water gauge image and correct the tilted water gauge; Step 4: The water level positioning module accumulates the intensity of the water gauge point cloud and obtains the water level line through the maximum gradient method; Step 5: The ruler detection module uses the digital character target detection algorithm on the corrected water gauge image, uses the character recognition algorithm to recognize the digital characters, and then calculates the water level value based on the digital characters, the water level line and the water gauge image; Step 5: The ruler detection module uses the target detection algorithm to detect digital characters on the corrected water gauge image, uses the character recognition algorithm to perform OCR recognition on the detected digital characters, and calculates the water level value based on the digital characters, water level line and water gauge image.

3. The complex environment water level monitoring method based on heterogeneous light field synergy according to claim 2 is characterized in that: The steps for calibrating the camera intrinsic parameters in step 1 and jointly calibrating the camera and lidar extrinsic parameters are as follows: The first step is to calibrate the camera intrinsic parameters. Using Zhang Zhengyou's calibration method, select 10-20 calibration pictures taken at different positions, angles, and postures, detect the corner points of the calibration plate in each picture, output the calibration average pixel error and the position of the calibration plate relative to the camera, reduce the average pixel error by screening pictures with excessive errors, and output the camera intrinsic parameters after calibration; The second step is to jointly calibrate the external parameters of the camera and lidar. Use the ACSC calibration algorithm, select a relatively open calibration scene, set the lidar to be more than 3 meters away from the calibration plate, select at least 10 different angles and distances to place the calibration plate, collect the corresponding point cloud and image data, perform spatiotemporal integration and feature refinement, perform 3D corner point estimation and 2D corner point detection, and combine the 3D-2D corner point data to solve the external parameter parameters of the camera and lidar.

4. The complex environment water level monitoring method based on heterogeneous light field synergy according to claim 2 is characterized in that: The steps for segmenting the water gauge image according to the water gauge point cloud and the joint external reference described in step 2 are as follows: The first step is to filter the LiDAR point cloud based on the prior range of the water surface in the LiDAR point cloud and the reflection intensity of water and metal to obtain the point cloud near the water surface. The second step is to perform discrete point filtering on the laser radar point cloud, specify a number of neighbor points and a threshold parameter, determine a set of neighbor points for each laser radar point cloud through K nearest neighbor search, calculate the average distance and standard deviation between each point and its neighbor points, and according to the set threshold parameter, if the average distance between a point and its neighbor points is greater than or equal to the threshold parameter, the point is determined as a discrete point and filtered out; the number of neighbor points and the threshold parameter are set according to the requirements for discrete point filtering; The third step is to use any of the KDTree, K-means and DBSCAN clustering algorithms to cluster the laser radar point cloud after discrete point filtering, classify points with similar spatial position relationships into one category, and obtain the water level point cloud; The fourth step is to use the calibrated external parameters of the camera and lidar to project the water level gauge point cloud to obtain the water level gauge pixel points, calculate the convex hull of the pixel points, and then expand and segment the water level gauge image.

5. The complex environment water level monitoring method based on heterogeneous light field synergy according to claim 2 is characterized in that: The image enhancement described in step 3 refers to using any exposure correction algorithm in VECNet or DIDNet to enhance the water gauge image, correct the exposure effect of the image, and reduce the impact of abnormal lighting on the detection accuracy of subsequent steps.

6. According to claim 2, the method for complex environment water level monitoring based on heterogeneous light field synergy is characterized in that: The steps of further segmenting the water gauge image using the image segmentation algorithm described in step 3 and correcting the tilt of the water gauge image after segmentation are as follows: The first step is to further segment the water gauge image using an image segmentation algorithm to obtain the tilt angle of the water gauge in the water gauge image, wherein the image segmentation algorithm is any one of SAM, SAM 2, and SEEM image segmentation algorithms; In the second step, the water gauge image is rotated in the opposite direction of the tilt angle to obtain a corrected water gauge image, where the rotation angle is equal to the tilt angle.

7. The complex environment water level monitoring method based on heterogeneous light field synergy according to claim 2 is characterized in that: The steps for accumulating the intensity of the water gauge point cloud in step 4 and obtaining the water level line by the maximum gradient method are as follows: The first step is to accumulate the water gauge point cloud intensity values ​​in each plane of the water gauge point cloud to obtain the accumulated water gauge point cloud intensity value of the plane. The accumulated water gauge point cloud intensity values ​​of all planes constitute a water gauge point cloud intensity accumulated value sequence; the accumulated water gauge point cloud intensity value in the plane is calculated by the following formula: Among them, SUM represents the cumulative intensity value of the water gauge point cloud, w represents the maximum value of the horizontal coordinate of the water gauge point cloud, l represents the maximum value of the vertical coordinate of the water gauge point cloud, I(x,y) represents the intensity value of the water gauge point cloud at the (x,y) coordinate, x represents the horizontal coordinate of the plane, and y represents the vertical coordinate of the plane; The second step is to calculate the gradient sequence corresponding to the cumulative value sequence of the water gauge point cloud intensity, and take the point with the maximum gradient as the water level line.

8. According to claim 2, the complex environment water level monitoring method based on heterogeneous light field synergy is characterized in that: The use of a target detection algorithm to detect digital characters in step 5 refers to: using a target detection algorithm to detect digital characters on the water gauge image, and extracting a digital character image, wherein the target detection algorithm can use any one of the YOLO series, EfficientDet, and RetinaNet target detection algorithms.

9. The complex environment water level monitoring method based on heterogeneous light field synergy according to claim 2 is characterized in that: The character recognition algorithm described in step 5 refers to any one of DB, CRNN, and SRN character recognition algorithms.

10. The complex environment water level monitoring method based on heterogeneous light field synergy according to claim 2 is characterized in that: The water level value calculated according to the digital characters, water level line and water gauge image in step 5 is calculated by the following formula: Among them, wl represents the upper left corner of the water gauge image as the origin coordinate (0,0), the upper right corner is in the positive direction of the horizontal x-axis, and the lower left corner is in the positive direction of the vertical y-axis. The water level value is calculated, num represents the OCR recognition result of the first numeric character in the opposite direction of the y-axis starting from the water level line, and h represents the actual height of the numeric character. num Indicates the pixel height of the digital character, y wl Indicates the ordinate of the water level, y num Indicates the vertical coordinate of the lowest point of the numeric character.

Images