Water level monitoring device and water level monitoring equipment based on radar multi-modal fusion

By using the multimodal fusion water level monitoring device, which combines video and radar data, the efficiency and accuracy problems of traditional water level monitoring methods have been solved, achieving high-precision and robust water level monitoring.

CN118640997BActive Publication Date: 2024-11-08WUHAN UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411108503.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-08
Estimated Expiration
2044-08-13

AI Technical Summary

Technical Problem

Traditional water level monitoring methods are inefficient, lack intelligence and automation, and single sensors are not adaptable and accurate enough in complex environments. In particular, cameras are greatly affected by weather and lighting, and radar has weak signal reflection and large errors when measured at an angle.

Method used

A water level monitoring device based on radar vision multimodal fusion is adopted, which combines RGB video image data and radar signal data, and performs feature extraction and fusion through deep residual network and global attention mechanism to achieve water level segmentation and recognition.

Benefits of technology

It achieves high precision and robustness in water level monitoring, can adapt to complex environments, reduces the impact of sensor failures on measurements, and expands the measurement range.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118640997B_ABST
    Figure CN118640997B_ABST
Patent Text Reader

Abstract

The present invention discloses a water level monitoring device and water level monitoring equipment based on radar-vision multimodal fusion, the device comprises an acquisition module, a radar data processing module and a radar-vision multimodal fusion output module, wherein the acquisition module is used to acquire the original data information of the water level in the area to be detected, the original data information of the water level comprises RGB video image data and radar signal data; the radar data processing module is used to process the radar signal data to obtain RDM image data; the radar-vision multimodal fusion output module is used to perform feature extraction and feature fusion processing on the RGB video image data and the RDM image data, and obtain a water level segmentation result map, so as to output a water level recognition result according to the water level segmentation result map. The present invention can fully integrate the data features of each sensor, realize strong robustness and high-precision measurement of the water level, and can realize water level monitoring in complex environments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of water level monitoring, and in particular to a water level monitoring device and water level monitoring equipment based on radar-vision multi-modal fusion. Background Art

[0002] Water level monitoring is one of the most basic contents of hydrological testing and is of great significance to water conservancy project construction, water resources planning and management, and water disaster prevention and control. Traditional water level monitoring methods are inefficient, lack intelligence and automation, and have observation risks in harsh environments. With the increasing demand for timeliness, accuracy, and security of hydrological data acquisition, non-contact measurement methods based on video, radar and other technologies have begun to be applied to water level monitoring, which is considered to be an important development trend in the future.

[0003] The video uses computer vision and digital image processing technology to intelligently segment and identify water levels, which is of great significance for real-time water level data acquisition and the construction of flood monitoring and early warning systems. However, camera sensors are greatly affected by weather and light when collecting data, and are difficult to adapt to complex and changing environmental conditions, and have poor robustness. Radar calculates the distance to the water surface based on the delay of the echo signal of electromagnetic waves, and then infers the water level. Its main advantage is that it is not affected by light and climate conditions. However, affected by conditions such as beam width and bandwidth, radar has problems such as weak signal reflection and large error under tilt measurement conditions, which makes the radar water level meter only able to measure perpendicular to the water surface. When installing the equipment, it is often necessary to build additional cables or cantilevers, which not only increases the measurement cost, but also increases the uncertainty of the measurement. It can be seen that due to the structure and characteristics of the sensor, a single sensor has certain limitations in adapting to complex environments and the accuracy of water level detection. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the first object of the present invention is to provide a water level monitoring device based on radar multimodal fusion, which can fully integrate the data features of each sensor, achieve strong robustness and high-precision measurement of water level, and can realize water level monitoring in complex environments.

[0005] A second object of the present invention is to provide a water level monitoring device.

[0006] To achieve the above object, the present invention is implemented through the following technical solutions:

[0007] A water level monitoring device based on radar multi-modal fusion, comprising:

[0008] An acquisition module is used to acquire raw water level data information of the area to be detected, wherein the raw water level data information includes RGB video image data and radar signal data;

[0009] A radar data processing module, used for processing the radar signal data to obtain RDM image data;

[0010] The Lei Shi multimodal fusion output module is used to perform feature extraction and feature fusion processing on the RGB video image data and the RDM image data, and obtain a water level segmentation result map, so as to output a water level recognition result according to the water level segmentation result map.

[0011] Preferably, the device further comprises:

[0012] The screening module is used to perform frame alignment on the RGB video image data and the radar signal data in a time series before the radar data processing module processes them, and perform downsampling screening processing according to a preset alignment interval to obtain preprocessed RGB video image data and radar signal data.

[0013] Preferably, the radar data processing module is also used to perform constant false alarm processing on the RDM image data to remove clutter interference data.

[0014] Preferably, the radar vision multimodal fusion output module includes:

[0015] The backbone network submodule includes an image network backbone unit and a radar backbone network unit. The image network backbone unit and the radar backbone network unit are composed of a deep residual network and are respectively used to perform convolution, regularization, activation function activation and maximum pooling processing on the RGB video image data and the RDM image data to achieve feature extraction.

[0016] Preferably, the image network backbone unit and the radar backbone network unit include:

[0017] The CBAM attention mechanism subunit is used to generate two types of attention feature map information in the two dimensions of channel and space, and multiply the two types of attention feature map information with the initially extracted feature map data to adaptively correct the initially extracted feature map.

[0018] Preferably, the backbone network submodule also includes a first convolutional layer, a second convolutional layer and a dilated space convolutional pooling pyramid, wherein the first convolutional layer and the dilated space convolutional pooling pyramid are connected to the back end of the image network backbone unit, and the second convolutional layer is connected to the back end of the radar backbone network unit, and the first convolutional layer, the dilated space convolutional pooling pyramid and the second convolutional layer are respectively used to improve the resolution of the corresponding adaptively corrected feature map.

[0019] Preferably, the backbone network submodule extracts features from the RGB video image data and the RDM image data to obtain image features and radar features, and the radar vision multimodal fusion output module further includes:

[0020] The feature fusion submodule is used to adopt a global attention mechanism to fuse the image features and radar features through an adaptive weight calculation method to generate fused features and obtain a fused feature map.

[0021] Preferably, the radar vision multimodal fusion output module further includes:

[0022] The prediction output submodule includes a third convolutional layer, an upsampling unit and an output unit connected in sequence. The prediction output submodule is used to restore the size and dimensional information of the fused feature map through convolution and upsampling operations, and output a water surface area probability distribution map of the area to be detected according to a preset probability, and output a water level segmentation result map based on the water surface area probability distribution map, and output a water level recognition result based on the water level segmentation result map.

[0023] Preferably, when the prediction output submodule outputs the water level recognition result according to the water level segmentation result map, it is specifically used to:

[0024] Obtaining pixel coordinates in the water level segmentation result image, and converting the pixel coordinates to obtain spatial three-dimensional coordinates;

[0025] Determine a water level reference plane, and project the three-dimensional spatial coordinates onto the water level reference plane to obtain the projection point coordinates;

[0026] The water level height is obtained by performing distance calculation based on the three-dimensional spatial coordinates and the projection point coordinates.

[0027] In order to achieve the above-mentioned object, a second aspect of the present invention provides a water level monitoring device, including the water level monitoring device based on radar-vision multi-modal fusion as described above.

[0028] The present invention has at least the following technical effects:

[0029] The present invention provides a water level monitoring device and water level monitoring equipment based on radar-visual multimodal fusion, wherein the device can automatically collect water level raw data information, and can pre-process the water level raw data information to obtain preferred water level raw data, and then can fully deeply fuse the RGB video image data and radar signal data in the water level raw data, and complement each other's advantages, so as to obtain all-round and high-precision water level information of the area to be detected, and at the same time can expand the measurement perception range, and give the multimodal fusion model the ability to adapt to uncertain environments. In addition, the present invention adopts a global attention mechanism, and fuses image features and radar features through an adaptive weight calculation method. When the camera or radar fails, or the sensor data is poor, the device can automatically adjust the image feature or radar feature fusion weight, thereby reducing the performance degradation of the model and realizing high-precision and strong robustness measurement of the water level.

[0030] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a structural block diagram of a water level monitoring device based on radar-vision multi-modal fusion according to an embodiment of the present invention.

[0032] Figure 2 The figure is a schematic diagram of the arrangement of a video sensor and a radar sensor according to an embodiment of the present invention.

[0033] Figure 3 Schematic diagram of the radar multi-modal fusion output process of an embodiment of the present invention.

[0034] Figure 4 4 is a structural block diagram of a water level monitoring device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present embodiment is described in detail below, and examples of the embodiment are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0036] The water level monitoring device and water level monitoring equipment based on radar-vision multi-modal fusion of this embodiment will be described below with reference to the accompanying drawings.

[0037] Figure 1 FIG. 1 is a structural block diagram of a water level monitoring device based on multi-modal fusion of radar and vision according to an embodiment of the present invention. Figure 1As shown, the water level monitoring device 100 based on radar-visual multimodal fusion includes an acquisition module 10, a radar data processing module 20 and a radar-visual multimodal fusion output module 30 which are connected in sequence, wherein the radar-visual multimodal fusion output module 30 is also connected to the acquisition module 10.

[0038] In one embodiment of the present invention, the acquisition module 10 is used to collect the original water level data information of the area to be detected, and the original water level data information includes RGB (Red Green Blue) video image data and radar signal data; the radar data processing module 20 is used to process the radar signal data to obtain RDM (Range Doppler Map) image data; the radar multimodal fusion output module 30 is used to perform feature extraction and feature fusion processing on the RGB video image data and the RDM image data, and obtain a water level segmentation result map, so as to output the water level recognition result according to the water level segmentation result map.

[0039] Figure 2 FIG. 1 is a schematic diagram of the arrangement of a video sensor and a radar sensor according to an embodiment of the present invention. Figure 2 As shown, the acquisition module 10 can be a multi-sensor combination consisting of a video sensor and a radar sensor. In this embodiment, the water level raw data information of the area to be detected can be collected by the video sensor and the radar sensor, wherein the RGB video image data is represented by continuous RGB image frames, and the radar signal data is continuous signal frames.

[0040] In one embodiment of the present invention, the water level monitoring device 100 based on radar-vision multimodal fusion also includes a screening module, which is used to frame align the RGB video image data and radar signal data in time series before processing by the radar data processing module 20, and perform downsampling and screening processing according to a preset alignment interval to obtain pre-processed preferred RGB video image data and radar signal data.

[0041] Specifically, since the sampling rates of the video sensor and the radar sensor are different, the collected RGB video image data and radar signal data need to be aligned at the same time to ensure the data correlation. At the same time, downsampling screening is performed, that is, a suitable preset alignment interval is set to prevent the continuous frames with high similarity from causing diminishing returns and overfitting of the results.

[0042] After obtaining the preferred RGB video image data and radar signal data, the radar signal data is further processed by the radar data processing module 20 to obtain high-precision RDM image data.

[0043] Specifically, in order to more effectively fuse radar signal data with RGB video image data, the collected radar signal data needs to be further processed, that is, the collected radar signal data is subjected to a two-dimensional fast Fourier transform to achieve the conversion of data from the time domain to the frequency domain and highlight the data features. At the same time, the RDM image data is subjected to constant false alarm calculation, that is, the existence of the target is determined by the detection threshold, and the clutter interference of the data is removed to improve the data accuracy. The constant false alarm is calculated by selecting a detection unit on the RDM image data. If the signal strength of the detection unit is greater than the detection threshold, the target is considered to exist. The target refers to the effective signal excluding clutter, such as the water level signal in this article.

[0044] In this embodiment, the two-dimensional fast Fourier transform is expressed as follows:

[0045] (1)

[0046] In the formula, , are the frequency domain horizontal and vertical coordinates of the radar signal data, is the RDM image data, For the preferred radar signal data, , are the time domain horizontal and vertical coordinates of the radar signal data, e is a natural constant.

[0047] The constant false alarm calculation formula is:

[0048] (2)

[0049] (3)

[0050] In the formula, is the noise expectation of the detection unit, , are the frequency domain horizontal and vertical coordinates of the last pair of radar signal data, S To detect the signal strength of the unit, T is the detection threshold, is the threshold factor.

[0051] In one embodiment of the present invention, the radar multimodal fusion output module 30 includes a backbone network submodule. The backbone network submodule includes an image network backbone unit and a radar backbone network unit, that is, Figure 3In the feature extraction backbone part, the image network backbone unit and the radar backbone network unit contain the same feature extractor, both of which are composed of deep residual networks, and can be used to perform convolution, regularization, activation function activation and maximum pooling on RGB video image data and RDM image data respectively to achieve feature extraction. It should be noted that when the image network backbone unit and the radar backbone network unit perform convolution, regularization, activation function activation and maximum pooling on RGB video image data and RDM image data respectively, the size and dimension of the input data can be converted at the same time, and the input data can be mapped to a high-dimensional space, so that it is more suitable for deep learning.

[0052] In one embodiment of the present invention, the image network backbone unit and the radar backbone network unit include a CBAM (convolutional block attention model) attention mechanism sub-unit, which can serially generate two types of attention feature map information in the two dimensions of channel and space, and multiply the two types of attention feature map information with the initially extracted feature map data, i.e., the feature map data after maximum pooling processing, to adaptively correct the initially extracted feature map, thereby improving the model accuracy and further improving the accuracy of the prediction results.

[0053] In one embodiment of the present invention, the backbone network submodule further includes a first convolutional layer, a second convolutional layer and a dilated spatial convolutional pooling pyramid, wherein the first convolutional layer is Figure 3 The 1x1 convolution layer in the upper middle part and the dilated space convolution pooling pyramid are connected to the back end of the image network backbone unit. The second convolution layer, that is, the 1x1 convolution layer below, is connected to the back end of the radar backbone network unit. The first convolution layer, the dilated space convolution pooling pyramid, and the second convolution layer are respectively used to improve the resolution of the corresponding adaptively corrected feature map, wherein the dilated space convolution pooling pyramid can also integrate the multi-scale information of the image, that is, obtain a multi-size feature map. In this embodiment, the prediction accuracy of the model can be improved by improving the resolution of the feature map and obtaining a multi-size feature map.

[0054] In one embodiment of the present invention, the backbone network submodule extracts features from the RGB video image data and the RDM image data to obtain image features. and radar signature The Leishi multimodal fusion output module also includes a feature fusion submodule, which can be used to use the global attention mechanism to integrate image features through an adaptive weight calculation method. and radar signature Fusion to generate fusion features , and obtain the fused feature map.

[0055] In this embodiment, the feature fusion submodule adopts a global attention mechanism, observes all sensor channels through a multi-layer global attention network, and performs feature fusion through an adaptive weight calculation method, which can achieve sensor behavior complementarity and increase the robustness of the fusion model. The feature fusion formula is expressed as follows:

[0056] (4)

[0057] In the formula, is the sigmoid (an activation function) function, , The shared weights for the RGB video image data and the RDM image data input, is the ReLU (an activation function) function, is the maximum pooling function.

[0058] In one embodiment of the present invention, the multimodal fusion output module of the radar vision further includes a prediction output submodule, which includes a third convolutional layer connected in sequence, namely Figure 3 The 3x3 convolutional layer, upsampling unit and output unit in the prediction output submodule are used to restore the size and dimensional information of the fused feature map through convolution and upsampling operations, and output the water surface area probability distribution map of the area to be detected according to the preset probability, and output the water level segmentation result map according to the water surface area probability distribution map, and output the water level recognition result according to the water level segmentation result map.

[0059] Specifically, a preset probability can be set, and then the device outputs a probability distribution map of the water surface area to be detected according to the preset probability. When the probability of the water surface area is 0, it means that there is no water in the area. When the probability of the water surface area is 1, it means that there is water in the area. Thus, a water level segmentation result map can be output.

[0060] Furthermore, the prediction output submodule can output the water level recognition result according to the water level segmentation result map. When the prediction output submodule outputs the water level recognition result according to the water level segmentation result map, it is specifically used to: obtain the pixel point coordinates in the water level segmentation result map, and convert the pixel point coordinates to obtain the spatial three-dimensional coordinates; determine the water level reference plane, and project the spatial three-dimensional coordinates onto the water level reference plane to obtain the projection point coordinates; perform distance calculation based on the spatial three-dimensional coordinates and the projection point coordinates to obtain the water level height.

[0061] Specifically, the water level recognition result is determined by the relationship between the pre-set spatial three-dimensional coordinates and the pixel coordinates of the water level segmentation result map. Assume that in the water level segmentation result map, the pixel point set coordinates of the water level segmentation line are , , For the The horizontal and vertical coordinate values ​​of the pixel points, is the number of pixels, and the coordinates of this pixel set correspond to the three-dimensional coordinates in the real world: , , and Respectively The spatial three-dimensional coordinate value corresponding to the pixel point, the water level reference plane equation is , is the water level reference function, then the water level Can be represented as a three-dimensional coordinate point Instead of projecting a point on the water level datum The distance , and is the three-dimensional coordinate value of the corresponding projection point, water level It is expressed as follows:

[0062] (5)

[0063] In one embodiment of the present invention, the device also includes an input display module, which is respectively communicated with the acquisition module 10, the radar data processing module 20, the radar vision multimodal fusion output module 30, and the screening module and other modules, and can be used to allow the user to input operation instructions and display the data obtained and processed by the corresponding module in the form of text, list, static graph or dynamic trend graph.

[0064] Furthermore, the present invention also provides a water level monitoring device. Figure 4 FIG. 1 is a structural block diagram of a water level monitoring device according to an embodiment of the present invention. Figure 4 As shown, the water level monitoring equipment 1000 includes the above-mentioned water level monitoring device 100 based on radar-vision multi-modal fusion.

[0065] In summary, the present invention provides a water level monitoring device and water level monitoring equipment based on radar-visual multimodal fusion, wherein the device can automatically collect water level raw data information, and can pre-process the water level raw data information to obtain the preferred water level raw data, and then can fully deeply fuse the RGB video image data and radar signal data in the water level raw data, and complement each other's advantages, so as to obtain all-round and high-precision water level information of the area to be detected, and at the same time can expand the measurement perception range, and give the multimodal fusion model the ability to adapt to uncertain environments. In addition, the present invention adopts a global attention mechanism, and fuses image features and radar features through an adaptive weight calculation method. When the camera or radar fails, or the sensor data is poor, the device can automatically adjust the image feature or radar feature fusion weight, thereby reducing the performance degradation of the model and achieving high-precision and strong robustness measurement of the water level.

[0066] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0067] Although the content of the present invention has been described in detail through the above preferred embodiments, it should be appreciated that the above description should not be considered as a limitation of the present invention. After reading the above content, it will be apparent to those skilled in the art that various modifications and substitutions of the present invention will occur. Therefore, the protection scope of the present invention should be limited by the appended claims.

Claims

1. A water level monitoring device based on multi-modal fusion of radar and vision, characterized in that: include: An acquisition module is used to acquire raw water level data information of the area to be detected, wherein the raw water level data information includes RGB video image data and radar signal data; A radar data processing module, used for processing the radar signal data to obtain RDM image data; A multi-modal fusion output module for performing feature extraction and feature fusion processing on the RGB video image data and the RDM image data, and obtaining a water level segmentation result map, so as to output a water level recognition result according to the water level segmentation result map; The radar multi-modal fusion output module includes: A backbone network submodule, comprising an image network backbone unit and a radar backbone network unit, wherein the image network backbone unit and the radar backbone network unit are composed of a deep residual network and are respectively used to perform convolution, regularization, activation function activation and maximum pooling processing on the RGB video image data and the RDM image data to achieve feature extraction; The image network backbone unit and the radar backbone network unit include: The CBAM attention mechanism subunit is used to generate two kinds of attention feature map information in the two dimensions of channel and space, and multiply the two kinds of attention feature map information with the initially extracted feature map data to adaptively correct the initially extracted feature map; The backbone network submodule further includes a first convolutional layer, a second convolutional layer and a dilated space convolutional pooling pyramid, wherein the first convolutional layer and the dilated space convolutional pooling pyramid are connected to the rear end of the image network backbone unit, and the second convolutional layer is connected to the rear end of the radar backbone network unit, and the first convolutional layer, the dilated space convolutional pooling pyramid and the second convolutional layer are respectively used to improve the resolution of the corresponding adaptively corrected feature map; The backbone network submodule extracts features from the RGB video image data and the RDM image data to obtain image features and radar features. The radar vision multimodal fusion output module also includes: A feature fusion submodule, for adopting a global attention mechanism to fuse the image features and radar features through an adaptive weight calculation method, generate fused features, and obtain a fused feature map; The radar multi-modal fusion output module also includes: A prediction output submodule, comprising a third convolution layer, an upsampling unit and an output unit connected in sequence, wherein the prediction output submodule is used to restore the size and dimension information of the fused feature map through convolution and upsampling operations, and output a water surface area probability distribution map of the area to be detected according to a preset probability, and output a water level segmentation result map according to the water surface area probability distribution map, and output a water level recognition result according to the water level segmentation result map; When the prediction output submodule outputs the water level recognition result according to the water level segmentation result map, it is specifically used to: Obtaining pixel coordinates in the water level segmentation result image, and converting the pixel coordinates to obtain spatial three-dimensional coordinates; Determine a water level reference plane, and project the three-dimensional spatial coordinates onto the water level reference plane to obtain the projection point coordinates; The water level height is obtained by performing distance calculation based on the three-dimensional spatial coordinates and the projection point coordinates.

2. The water level monitoring device based on radar multi-modal fusion according to claim 1, characterized in that: Also includes: The screening module is used to perform frame alignment on the RGB video image data and the radar signal data in a time series before the radar data processing module processes them, and perform downsampling screening processing according to a preset alignment interval to obtain preprocessed RGB video image data and radar signal data.

3. The water level monitoring device based on radar multi-modal fusion according to claim 1, characterized in that: The radar data processing module is also used to perform constant false alarm processing on the RDM image data to remove clutter interference data.

4. A water level monitoring device, characterized in that: It comprises a water level monitoring device based on radar multi-modal fusion as described in any one of claims 1-3.

Citation Information

Patent Citations

  • Water level data fusion method and device based on Leighting multi-sensor data

    CN114911789A

  • Drainage well water level monitoring method, device, storage medium and system

    CN117213581A

  • System and method for measuring water depth using radar image processing

    KR102558955B1