A damage monitoring and early warning method and system based on a radar-visual integrated device

Abstract

The present invention discloses a damage monitoring and early warning method and system based on a radar-visual integrated device. The method comprises: obtaining a radar point cloud image corresponding to each radar scanning frame within a current preset duration, thereby obtaining multiple radar point cloud images corresponding to the multiple radar scanning frames; each radar point cloud image represents the position coordinates of each corner reflector in the monitoring area; obtaining a set of second position coordinates corresponding to each image frame within the current preset duration, thereby obtaining multiple sets of second position coordinates corresponding to the multiple image frames; the second position coordinates corresponding to each image frame are the position coordinates of each corner reflector; synchronizing, fusing, and linearly fitting the multiple sets of first position coordinates and the multiple sets of second position coordinates represented by the multiple radar point cloud images to obtain a curve corresponding to the current preset duration; and determining the current monitoring result of the monitoring area based on the characteristics of the curves corresponding to the current preset duration and historical preset durations. The present invention can improve the accuracy and timeliness of monitoring.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent monitoring technology, and in particular relates to a damage monitoring and early warning method and system based on a radar-visual integrated machine. Background Art

[0002] On traffic roads, environmental factors such as earthquakes, heavy rains, floods, and underground land displacement can lead to road collapse, ground subsidence, mudslides, and other phenomena. This will not only cause road congestion, but also cause vehicle damage and even casualties. Therefore, road monitoring and prediction for the above situations are very important. Through prediction, areas where disasters may occur can be observed in advance, early warnings can be issued, and early management can be carried out, which can also reduce the losses caused by natural disasters.

[0003] In recent years, the fusion of radar and vision technology has gradually attracted attention. This fusion technology can fully utilize the respective advantages of radar and vision to achieve data complementarity and information fusion, thereby improving the accuracy and reliability of monitoring.

[0004] Currently, widely used disaster monitoring and early warning technologies primarily include systems based on synthetic aperture radar (SAR) and synthetic aperture radar interferometry (InSAR). For example, ground-based synthetic aperture interferometric radar (GBInSAR) systems achieve high-resolution imaging through synthetic aperture technology, but are only suitable for high-precision monitoring of small areas. Satellite-based synthetic aperture radar, on the other hand, can cover large areas, but its wide-area, periodic monitoring model cannot guarantee real-time performance. Alternatively, cameras are deployed on slopes to capture images of the monitored area and use image recognition algorithms to monitor and warn of landslide hazards. However, the image information provided by visual technology also has certain limitations. Summary of the Invention

[0005] In order to solve the above-mentioned problems existing in the prior art, the present invention provides a damage monitoring and early warning method and system based on a radar-visual integrated device.

[0006] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0007] The present invention provides a damage monitoring and early warning method based on a radar-visual integrated device, comprising:

[0008] Within the current preset time duration, a radar point cloud image corresponding to each radar scanning frame is obtained to obtain a plurality of radar point cloud images corresponding to the plurality of radar scanning frames; a point in each radar point cloud image represents the position coordinates of each corner reflector in the monitoring area; each radar point cloud image represents a set of first position coordinates; the duration of the preset time duration is the first time duration;

[0009] Within the current preset time length, a set of second position coordinates corresponding to each image frame is obtained, thereby obtaining multiple sets of second position coordinates corresponding to multiple image frames one by one; the set of second position coordinates corresponding to each image frame is the position coordinates of each corner reflector in the monitoring area;

[0010] Performing synchronization processing, fusion processing, and linear fitting processing on the multiple groups of first position coordinates represented by the multiple radar point cloud images and the multiple groups of second position coordinates to obtain a curve corresponding to the current preset time length;

[0011] The current monitoring result of the monitoring area is determined based on the characteristics of the curve corresponding to the current preset time period and the characteristics of the curve corresponding to the historical preset time periods of the current preset time period.

[0012] The present invention also provides a damage monitoring and early warning system based on a radar-visual integrated device, comprising:

[0013] A data acquisition module is used to scan the monitoring area in real time using a radar to obtain a real-time radar point cloud map of the monitoring area, and to capture images of the monitoring area in real time using a camera, and to identify the position of each corner reflector in the captured image frame in real time;

[0014] a data processing module configured to obtain, from the data acquisition module, a radar point cloud image corresponding to each radar scanning frame within a current preset time duration, to obtain a plurality of radar point cloud images corresponding to the plurality of radar scanning frames; wherein the points in each radar point cloud image represent the position coordinates of each corner reflector in the monitoring area; wherein each radar point cloud image represents a set of first position coordinates; wherein the duration of the current preset time duration is the preset time duration; and, within the current preset time duration, obtain a set of second position coordinates corresponding to each image frame, to obtain a plurality of sets of second position coordinates corresponding to the plurality of image frames; wherein the set of second position coordinates corresponding to each image frame is the position coordinates of each corner reflector in the monitoring area; perform synchronization processing, fusion processing, and linear fitting processing on the plurality of sets of first position coordinates and the plurality of sets of second position coordinates represented by the plurality of radar point cloud images to obtain a curve corresponding to the current preset time duration; determine a current monitoring result of the monitoring area based on the characteristics of the curve corresponding to the current preset time duration and the characteristics of the curve corresponding to the historical preset time duration of the current preset time duration; generate warning information based on the current monitoring result; and send the warning information to the information release module;

[0015] The information release module is used to send the warning information to the target device.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] By combining radar and camera, the present invention can realize all-weather real-time monitoring of the monitoring area, and is not easily interfered by natural factors such as vegetation, weather and terrain. It can work stably in complex environments such as rainy days, foggy days and nights, ensuring the continuity and accuracy of monitoring data. The present invention synchronizes, fuses and linearly fits the coordinates of the corner reflector detected by the radar and the coordinates of the corner reflector detected by the camera to obtain a corresponding curve, and performs damage monitoring of the monitoring area based on the characteristics of the obtained curve. It can accurately and timely identify slight position changes and curve shape changes of the corner reflector, and has strong anti-interference ability, further improving the timeliness, reliability and accuracy of monitoring.

[0018] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of a damage monitoring and early warning method based on a radar-visual integrated device provided by an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of an exemplary equipment layout in a monitoring area provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0022] Figure 1 This is a flow chart of a damage monitoring and early warning method based on a radar-visual integrated device provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0023] S101. Within a current preset duration, obtain a radar point cloud image corresponding to each radar scanning frame to obtain multiple radar point cloud images corresponding to the multiple radar scanning frames; the points in each radar point cloud image represent the position coordinates of each corner reflector in the monitoring area; each radar point cloud image represents a set of first position coordinates; the duration of the preset duration is the first time length.

[0024] Here, the value of the first time length can be set according to actual needs, for example, it can be 2 seconds or 4 seconds, etc., and the present invention is not limited to this.

[0025] Here, multiple corner reflectors are fixedly installed in different areas of the monitoring area, and radar and cameras are also installed in the monitoring area. In some embodiments, the radar and camera can be included in a single device, such as a radar-visual integrated device; in other embodiments, the radar and camera can be two different devices and installed separately.

[0026] The damage described in the present invention can be man-made damage, such as damage to buildings, mountains, roads, rivers, etc., or natural damage (i.e., natural disasters), such as natural road collapse, mountain collapse, mud and rock flow, landslide, etc. Accordingly, the monitoring area described in the present invention can be buildings, roads, mountains, etc.

[0027] For example, when the radar and camera include a radar-vision integrated device, and the monitoring area is a road, the radar-vision integrated device can be set up above the road prone to damage and at intervals (for example, 5 meters) above the road while ensuring the effective detection range of the radar-vision integrated device, to ensure that the radar-vision integrated device can cover the entire area; then, the radar-vision integrated device is firmly fixed to the ground or other stable structure using a bracket to ensure that it does not move during long-term use. At the same time, corner reflectors are arranged at intervals (for example, 10 meters) on both sides of the road in the monitoring area to ensure that the corner reflectors on each side of the road in the monitoring area are connected into a curve. Then, the corner reflectors are firmly mounted on the ground or rock and ensure that the corner reflectors are mounted on a bracket perpendicular to the ground to avoid the influence of external factors such as wind and rain. For example, Figure 2 This is an example equipment layout diagram for a monitoring area, where 1 represents a corner reflector and 2 represents a radar-visual integrated device. After completing this layout, the corner reflector positions can be calibrated to eliminate or reduce system errors and improve the measurement accuracy and reliability of the millimeter-wave radar. During calibration, it is important to adjust the positions and angles of the radar and corner reflectors based on the initial measurement data to ensure accurate detection of road damage.

[0028] Here, the second position coordinates are obtained by the camera performing target recognition on the image frame through a target recognition algorithm, for example, the YoLov8 target recognition algorithm.

[0029] S102. Within the current preset time length, obtain a set of second position coordinates corresponding to each image frame to obtain multiple sets of second position coordinates corresponding to multiple image frames; the set of second position coordinates corresponding to each image frame is the position coordinates of each corner reflector in the monitoring area.

[0030] It should be noted that the above S101 and S102 are performed simultaneously.

[0031] S103: Perform synchronization processing, fusion processing, and linear fitting processing on multiple groups of first position coordinates represented by multiple radar point cloud images and multiple groups of second position coordinates to obtain a curve corresponding to the current preset time length.

[0032] S104: Determine a current monitoring result of the monitoring area based on the characteristics of the curve corresponding to the current preset duration and the characteristics of the curve corresponding to the historical preset duration of the current preset duration.

[0033] By combining radar and camera, the present invention can realize all-weather real-time monitoring of the monitoring area, and is not easily interfered by natural factors such as vegetation, weather and terrain. It can work stably in complex environments such as rainy days, foggy days and nights, ensuring the continuity and accuracy of monitoring data. The present invention synchronizes, fuses and linearly fits the coordinates of the corner reflector detected by the radar and the coordinates of the corner reflector detected by the camera to obtain a corresponding curve, and performs damage monitoring of the monitoring area based on the characteristics of the obtained curve. It can accurately and timely identify slight position changes and curve shape changes of the corner reflector, and has strong anti-interference ability, further improving the timeliness, reliability and accuracy of monitoring.

[0034] In the present invention, each radar point cloud image has a radar timestamp, and each image frame has a camera timestamp. Based on this, the above S103 is implemented by the following steps:

[0035] S1031. Time synchronize multiple image frames with multiple radar scan frames to obtain at least one pair of synchronized frames; each pair of synchronized frames includes a radar scan frame and an image frame, where the time difference between the radar timestamp and the camera timestamp is less than a preset time difference; the camera timestamp or the radar timestamp of the pair of synchronized frames is the time point of the pair of synchronized frames.

[0036] Here, time synchronization can be performed based on the camera timestamp. Specifically, for each of the multiple image frames, a radar scan frame can be found from the multiple radar scan frames, where the time difference between the radar timestamp and the camera timestamp of the image frame is less than a preset time difference. This radar scan frame is used as the radar scan frame synchronized with the image frame. Through this method, after executing the time synchronization of multiple image frames and multiple radar scan frames, at least one pair of synchronized frames can be obtained. The introduction of a time synchronization mechanism can improve the integration and analysis accuracy of radar and camera data, which has a direct impact on the response time and early warning capabilities of the real-time monitoring system. Through precise time alignment, the rapid processing and timely feedback of real-time data can be ensured, thereby improving the efficiency and effectiveness of monitoring.

[0037] It should be noted that the radar and camera system clocks need to be regularly calibrated to ensure synchronization with millisecond accuracy or better. This can be achieved through methods such as the Network Time Protocol (NTP) to synchronize the system clocks with Coordinated Universal Time (UTC).

[0038] In some embodiments, in order to improve the purity and accuracy of the radar point cloud image, thereby improving the data quality, after obtaining the radar point cloud image, an adaptive Kalman filtering algorithm can be used to filter the radar point cloud image for noise, remove unnecessary interference signals, and obtain a radar point cloud image without noise for subsequent data processing.

[0039] S1032. For each pair of synchronous frames, project a set of first position coordinates represented by a radar point cloud image corresponding to a radar scanning frame in the pair of synchronous frames into a camera coordinate system corresponding to an image frame in the pair of synchronous frames to obtain a set of projected coordinates.

[0040] Specifically, when projecting a set of first position coordinates represented by the radar point cloud image in a pair of synchronized frames onto the camera coordinate system corresponding to the image frame in the pair of synchronized frames, the first position coordinates can be projected one-to-one using a homography matrix perspective transformation method to ensure high spatial consistency between the camera and radar data. Furthermore, during the coordinate transformation process, dynamic calibration technology can be employed. This dynamic calibration technology allows the system to adaptively adjust the homography matrix based on changes in the position of the corner reflector to ensure the accuracy and real-time nature of the transformation process, thereby enhancing the flexibility and robustness of position coordinate tracking. These steps enable precise monitoring and tracking of the corner reflector position, ensuring the efficient operation of the monitoring system in complex environments.

[0041] S1033. Match a set of projection coordinates with a set of second position coordinates corresponding to the image frames in the pair of synchronization frames to obtain a set of matching point pairs corresponding to the pair of synchronization frames; a set of matching point pairs includes multiple matching point pairs, and a matching point pair includes a projection coordinate and a second position coordinate.

[0042] Specifically, when matching the projection coordinates with the second position coordinates, a nearest neighbor algorithm can be used for matching, and a distance error range can be set to filter and match camera projection points within each radar distance error range. Specifically, first, a reasonable distance error range is defined to limit the maximum acceptable matching distance. Then, for each projection point (i.e., each projection coordinate), a camera point (i.e., the second position coordinate) is searched within this distance error range. From the searched camera points, the camera point with the smallest distance to the projection point is selected, and the projection point and the camera point are considered a matching point pair. This matching achieves an effective association between radar data and camera data.

[0043] S1034: Each matching point pair in a set of matching point pairs corresponding to the pair of synchronization frames is fused into a fusion point to obtain a set of fusion points corresponding to the pair of synchronization frames.

[0044] Specifically, S1034 is implemented by the following steps:

[0045] S1. Calculate the variance of all projection coordinates in at least one set of matching point pairs to obtain the first variance

[0046] For example, when the at least one set of matching point pairs obtained after executing S1033 above is n matching point pairs in total, the first variance The expression is as follows:

[0047]

[0048] Among them, X1+X2+…+X n are the n projection coordinates of n matching point pairs.

[0049] S2. Calculate the variance of all second position coordinates in at least one set of matching point pairs to obtain a second variance.

[0050] For example, when the at least one set of matching points obtained after executing S1033 above has a total of n matching points, the second variance The expression is as follows:

[0051]

[0052]

[0053] Among them, Y1+Y2+…+Y n are the n second position coordinates of the n matching point pairs.

[0054] S3. According to the first variance and the second variance Calculate the Kalman gain value K for the current preset duration.

[0055] For example, the expression of the Kalman gain value K is as follows:

[0056]

[0057] S4. Using the Kalman gain value K of the current preset time length, perform weighted fusion on the projection coordinates and the coordinate values ​​of the second position coordinates of each matching point pair in a set of matching point pairs corresponding to the pair of synchronization frames to obtain a fusion point formed by the fusion of each matching point pair.

[0058] For example, when the projection coordinates of a matching point pair are The second position coordinates are When fusion is formed, a fusion point is formed The expression is: is the horizontal coordinate value after fusion, is the vertical coordinate value after fusion.

[0059] S5. Pair the matching points corresponding to the pair of synchronization frames, perform weighted fusion on each matching point pair, and obtain all fused points as the set of fused points corresponding to the pair of synchronization frames.

[0060] S1035 . Perform linear fitting on a group of fusion points corresponding to the pair of synchronization frames to obtain a curve corresponding to the pair of synchronization frames.

[0061] S1036 : Use at least one curve in one-to-one correspondence between at least one pair of synchronization frames as the curve corresponding to the current preset duration.

[0062] Specifically, the least square method can be used to perform curve fitting on each set of fusion points, or polynomial fitting can be combined to use a polynomial of appropriate order to perform curve fitting, thereby obtaining at least one curve that corresponds one-to-one to at least one pair of synchronization frames.

[0063] In the present invention, there is at least one curve corresponding to the current preset duration, and one curve corresponds one-to-one to a pair of synchronization frames, each pair of synchronization frames has a time point, and each pair of synchronization frames includes a radar scanning frame and an image frame; the above S104 is implemented by the following steps:

[0064] S1041. Based on the characteristics of the curve corresponding to the current preset duration and the characteristics of the curve corresponding to the preset duration in the history of the current preset duration, determine a mutation measurement value F(t) for each curve corresponding to the current preset duration; the mutation measurement value is used to measure whether the curve has undergone a mutation.

[0065] In the present invention, the historical preset duration of the current preset duration is the P preset durations before the current preset duration, where P is an integer greater than or equal to 1 (for example, P can be 1). Based on this, S1041 can be implemented by the following steps:

[0066] S11 . Taking a pair of time points of synchronous frames corresponding to each curve as the time points corresponding to the curve.

[0067] S12. Sort all curves corresponding to the first P preset durations and the current preset duration in chronological order to obtain Q curves; wherein the curve corresponding to the current preset duration is the U-th curve to the Q-th curve among the Q curves; Q is an integer greater than P, U is a positive integer, and U is less than or equal to Q.

[0068] Here, the number of curves corresponding to each preset duration is not necessarily the same.

[0069] For example, when Q is 7 and U is 3, the curves corresponding to the current preset duration are the last four curves of the seven curves.

[0070] S13. For the c-th curve among the U-th to Q-th curves, calculate the curvature difference between corresponding points in every two adjacent curves in a set of curves consisting of the c-th curve and the first L curves of the c-th curve, to obtain a set of curvature differences corresponding to the c-th curve; the value of c is 1 to (Q-U+1); and L is a preset positive integer.

[0071] Here, the value of L can be set according to actual needs. For example, L can be 3, 4, or 2, etc. Continuing with the example where the curves corresponding to the current preset duration are the last four curves among the seven curves, when L is 2 and the current c is 3, the curvature differences between the first curve and each corresponding point in the second curve, as well as the curvature differences between each corresponding point in the second curve and the third curve among the seven curves are calculated. These obtained curvature differences are used as a set of curvature differences corresponding to the third curve among the seven curves. Exemplarily, when each curve contains 2 points, a set of curvature differences corresponding to the third curve is expressed as: {ΔR1, ΔR2, ΔR3, ΔR4}, where ΔR1 represents the curvature difference between the first curve and the first point on the second curve among the 7 curves, ΔR2 represents the curvature difference between the first curve and the second point on the second curve among the 7 curves, ΔR3 represents the curvature difference between the second curve and the first point on the third curve among the 7 curves, and ΔR4 represents the curvature difference between the second curve and the second point on the third curve among the 7 curves.

[0072] S14. Determine a mutation measurement value of the c-th curve based on a set of curvature differences corresponding to the c-th curve.

[0073] Specifically, a set of curvature differences corresponding to the c-th curve can be used as the input value of the PELT algorithm, and the PELT algorithm is used to calculate the minimum loss value F(t) corresponding to the c-th curve. c ; The minimum loss value corresponding to the c-th curve is used as a mutation measurement value of the c-th curve. For example, the mutation measurement value F(t) corresponding to the third curve is 3 The expression is as follows:

[0074]

[0075] Here, β is a penalty term that balances the loss and the number of change points.

[0076] S1042: Determine a current monitoring result of the monitoring area based on a magnitude relationship between a mutation measurement value of each curve corresponding to a current preset time length and a preset threshold.

[0077] In some embodiments, the preset threshold includes: a plurality of different sub-thresholds, each of which constitutes N different threshold intervals with different numerical ranges and successively increasing upper limits; N is a first preset integer greater than 1. Based on this, the above S1042 can be implemented by the following steps:

[0078] S21. When the mutation measurement value of each curve corresponding to the current preset time duration belongs to the i-th threshold interval, generate a monitoring result corresponding to the i-th threshold interval; i is a positive integer, and the value of i ranges from 1 to N;

[0079] S22. The generated monitoring result is used as the current monitoring result of the monitoring area.

[0080] Here, the number of sub-thresholds and the value of N can be set according to actual needs and are not limited in the present invention. For example, the number of sub-thresholds can be 6, and accordingly, N can be 3. The above-mentioned multiple different sub-thresholds then constitute three different threshold intervals with different numerical ranges and successively increasing upper limits. These three different threshold intervals correspond one-to-one to three levels of mutation severity: the first threshold interval corresponds to a low mutation severity, the second threshold interval corresponds to a medium mutation severity, and the third threshold interval corresponds to a high mutation severity. Therefore, when the mutation metric values ​​of each curve corresponding to the current preset duration all fall within the first threshold interval, a monitoring result indicating a low mutation severity is generated. When the mutation metric values ​​of each curve corresponding to the current preset duration all fall within the second threshold interval, a monitoring result indicating a medium mutation severity is generated. When the mutation metric values ​​of each curve corresponding to the current preset duration all fall within the third threshold interval, a monitoring result indicating a high mutation severity is generated.

[0081] In some embodiments, the above S1042 may also be implemented by the following steps:

[0082] S31 . For the mutation measurement value of each curve corresponding to the current preset duration, when the mutation measurement value is greater than a preset threshold, a pair of time points of synchronization frames corresponding to the curve corresponding to the mutation measurement value is taken as a mutation time point.

[0083] Here, the value of the preset threshold can be set according to actual needs, and the present invention does not limit this.

[0084] S32. Determine the number of consecutive mutation time points within a preset analysis duration; the consecutive mutation time points are at least two consecutive mutation time points; the preset analysis duration is a duration that at least includes the current preset duration, and the duration of the preset analysis duration is the second duration.

[0085] For example, the preset analysis time can be set according to actual needs, and the present invention does not limit this. For example, the preset analysis time can be 4 seconds, and the preset time is 1 second.

[0086] The meaning of the continuity of time points described in the present invention is as follows: for example, when the current preset time length corresponds to 3 curves, the time points corresponding to these three curves are three consecutive time points, the time point corresponding to the first curve of the three curves and the time point corresponding to the second curve are two consecutive time points, and the time point corresponding to the second curve of the three curves and the time point corresponding to the third curve are two consecutive time points.

[0087] For example, when the preset analysis time length is 4 seconds and the preset time length is 2 seconds, the number of consecutive mutation time points in the consecutive 4 seconds consisting of the current 2 seconds and the previous 2 seconds of the current 2 seconds can be determined through S32.

[0088] S33. When the number of consecutive mutation time points within the preset analysis time length belongs to the j-th time point quantity interval among the preset M different time point quantity intervals, a monitoring result corresponding to the j-th time point quantity interval is generated; the quantity ranges of the M different time point quantity intervals are different, and the upper limit values ​​increase successively, j is a positive integer, and the value of j ranges from 1 to M; M is a second preset integer greater than 1.

[0089] Here, the value of M can be set according to actual needs, and the present invention does not limit this.

[0090] Exemplarily, M can be 3, and the above-mentioned M different time point number intervals have different number ranges and three different time point number intervals with increasing upper limits; these three different time point number intervals correspond one-to-one to three mutation severity levels, that is, the mutation severity level corresponding to the first time point number interval is low, the mutation severity level corresponding to the second time point number interval is medium, and the mutation severity level corresponding to the third time point number interval is high. Therefore, when the number of consecutive mutation time points within the preset analysis time length belongs to the first time point number interval, a monitoring result can be generated to characterize a low degree of mutation; when the number of consecutive mutation time points within the preset analysis time length belongs to the second time point number interval, a monitoring result can be generated to characterize a medium degree of mutation; when the number of consecutive mutation time points within the preset analysis time length belongs to the third time point number interval, a monitoring result can be generated to characterize a high degree of mutation.

[0091] S34. The generated monitoring result is used as the current monitoring result of the monitoring area.

[0092] In some embodiments, after the current monitoring result is generated in the above S104, step S105 is further included:

[0093] S105. Generate early warning information based on current monitoring results.

[0094] For example, when the current monitoring results indicate a low level of sudden change, the warning message may indicate a low risk of damage to the monitored area. This indicates the possibility of minor geological activity. In this case, the risk of geological change is low, and a low-level warning message is used to remind relevant personnel to observe and record, but no immediate action is required. In this case, the main purpose is to detect potential problems early, conduct continuous monitoring, and ensure preparation before the situation worsens. When the current monitoring results indicate a moderate level of sudden change, the warning message may indicate a low risk of damage to the monitored area. This indicates a high risk of geological activity, and a medium-level warning is used to remind relevant personnel to be vigilant and prepare for possible geological activity. This may include increasing monitoring frequency, preparing emergency supplies, and notifying relevant departments and personnel to take preventive measures to reduce potential losses and dangers. When current monitoring results indicate a high degree of sudden change, the warning may indicate a high risk of damage in the monitored area. This indicates the possibility of a severe geological disaster, prompting a high-level warning and the initiation of emergency response measures. These measures may include immediate evacuation of personnel, sealing off the danger zone, initiating emergency rescue plans, and notifying government and emergency management departments to coordinate rescue efforts. High-level warnings are intended to maximize the protection of life and property and minimize the impact and losses caused by the disaster.

[0095] In some embodiments, the device executing the aforementioned damage monitoring and early warning method based on an integrated radar and visual device is a data processing device, such as an edge server. The edge server can obtain real-time data on the monitoring area detected by radar and cameras from the integrated radar and visual device located within the monitoring area, or obtain real-time data on the monitoring area detected by radar and cameras from radar and cameras located within the monitoring area.

[0096] In some embodiments, cameras or integrated radar and video cameras within the monitoring area are further configured to perform damage and disaster identification on real-time images of the monitoring area, generating image recognition results. These image recognition results may include, for example, whether there is road collapse, road damage, or road waterlogging. The edge server may also obtain real-time image recognition results from the cameras or integrated radar and video cameras.

[0097] The present invention also provides a damage monitoring and early warning system based on a radar-visual integrated machine, comprising: a data acquisition module, a data processing module and an information release module.

[0098] The data acquisition module is used to scan the monitoring area in real time through radar to obtain a real-time radar point cloud map of the monitoring area, and to capture images of the monitoring area in real time through a camera, and to identify the position of each corner reflector in the captured image frame in real time.

[0099] The data processing module is used to obtain a radar point cloud image corresponding to each radar scanning frame from the data acquisition module within a current preset time length, and obtain multiple radar point cloud images corresponding to multiple radar scanning frames one by one; the points in each radar point cloud image represent the position coordinates of each corner reflector in the monitoring area; each radar point cloud image represents a set of first position coordinates; the duration of the current preset time length is the preset time length; and, within the current preset time length, obtain a set of second position coordinates corresponding to each image frame, and obtain multiple sets of second position coordinates corresponding to multiple image frames one by one; the set of second position coordinates corresponding to each image frame is the position coordinates of each corner reflector in the monitoring area; synchronize, fuse and linearly fit the multiple sets of first position coordinates and the multiple sets of second position coordinates represented by the multiple radar point cloud images to obtain a curve corresponding to the current preset time length; determine the current monitoring result of the monitoring area based on the characteristics of the curve corresponding to the current preset time length and the characteristics of the curve corresponding to the historical preset time length of the current preset time length; generate warning information based on the current monitoring result; and send the warning information to the information release module.

[0100] The information release module is used to send the risk warning level to the target device.

[0101] In some embodiments, the data acquisition module is also used to identify damage and disasters in images of the monitoring area taken in real time to obtain image recognition results; the data processing module is also used to obtain image recognition results of the current preset time length from the data acquisition module, and send the image recognition results and risk warning levels to the information release module through a dedicated warning information channel.

[0102] In some embodiments, the data acquisition module, data processing module, and information release module can all be hardware modules. For example, the data acquisition module can be a radar-visual integrated device or a radar and camera; the data processing module can be an edge server communicatively connected to the radar-visual integrated device (or radar and camera); and the information release module can be a cloud server, which can be a server at the monitoring center for the monitored area. In some embodiments, the data acquisition module and the data processing module can both be software programs that require hardware to implement their corresponding functions.

[0103] Here, the target device can be an electronic device installed in the monitoring area, such as a road information board, as well as the mobile phones of vehicles and personnel in the monitoring area. It can also be a display device in the monitoring center of the monitoring area, or the mobile phones and computers of relevant personnel in the monitoring center of the monitoring area. The warning information and image recognition results can be sent in the form of text messages, emails, APP push, etc. Promptly sending warning information to relevant departments and personnel can ensure the accuracy and timeliness of the warning information, making it easier for relevant personnel to take timely response measures. In particular, by sending warning information to road information boards, the warning information can be displayed on the information board as soon as it is available, alerting passing vehicles and reducing the occurrence of safety accidents.

[0104] Here, the warning information release channel can be a communication channel dedicated to transmitting warning information and image recognition results. If the possibility of damage is predicted, the warning information can be uploaded through a dedicated channel in a timely manner to reduce the delay in the transmission process.

[0105] In some embodiments, the data processing module can also send the acquired radar data and image data, as well as the processed data, to the information release module through the daily data backup channel for daily backup. Similarly, the daily data backup channel can also be a communication channel.

[0106] In general, the present invention has the following technical effects:

[0107] Real-time performance: This system enables 24 / 7 real-time monitoring and transmits data to a cloud server via two upload channels: one for daily data backup and the other for warning information release. This ensures rapid delivery of warning information. Once a potential disaster is predicted, the warning is immediately sent to the communication terminals of relevant departments or personnel, ensuring they can take necessary countermeasures as quickly as possible.

[0108] Cost: Radar-based vision systems are less expensive than other technology solutions, and corner reflectors are also more affordable. Radar provides precise location and deformation data, while vision technology provides clear images and feature recognition. By fusing these two data sets, a more comprehensive understanding of road conditions in the monitored area can be achieved, enabling timely detection and early warning of potential road disasters and other damage.

[0109] Accuracy: The present invention can ensure reliable geological disaster early warning support under various environmental conditions, is not easily interfered with by natural factors such as vegetation, weather, and terrain, and can work stably in complex environments such as rainy days, foggy days, and at night, ensuring the continuity and accuracy of monitoring data. The integrated radar vision device can accurately identify small position changes and curve shape changes of the corner reflector. The corner reflector design with strong anti-interference ability improves the reliability of the signal and the accuracy of the monitoring data. By comparing and analyzing historical data and real-time data, the accuracy of anomaly detection is improved, and early signals of damage such as geological disasters can be more accurately identified.

[0110] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0111] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0112] In the specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0113] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the scope of protection of the present invention.

Claims

1. A damage monitoring and early warning method based on a radar-visual integrated device, characterized in that: include: Within the current preset time length, a radar point cloud image corresponding to each radar scanning frame is obtained, and multiple radar point cloud images corresponding to multiple radar scanning frames are obtained; The points in each radar point cloud image represent the position coordinates of each corner reflector in the monitoring area; each radar point cloud image represents a set of first position coordinates; the time length of the preset time length is the first time length; Each radar point cloud image has a radar timestamp; Within the current preset time duration, a set of second position coordinates corresponding to each image frame is obtained, thereby obtaining multiple sets of second position coordinates corresponding to multiple image frames one by one; the set of second position coordinates corresponding to each image frame is the position coordinates of each corner reflector in the monitoring area, and each image frame has a camera timestamp; Time synchronization is performed on the multiple image frames and the multiple radar scan frames to obtain at least one pair of synchronized frames; each pair of synchronized frames includes a radar scan frame and an image frame having a time difference between a radar timestamp and a camera timestamp that is less than a preset time difference; the camera timestamp or the radar timestamp of the pair of synchronized frames is a time point of the pair of synchronized frames; For each pair of synchronous frames, projecting a set of first position coordinates represented by a radar point cloud image corresponding to a radar scanning frame in the pair of synchronous frames into a camera coordinate system corresponding to an image frame in the pair of synchronous frames to obtain a set of projected coordinates; Matching the set of projection coordinates with a set of second position coordinates corresponding to the image frames in the pair of synchronous frames to obtain a set of matching point pairs corresponding to the pair of synchronous frames; A set of matching point pairs includes multiple matching point pairs, and a matching point pair includes a projection coordinate and a second position coordinate; Each matching point pair in a set of matching point pairs corresponding to the pair of synchronization frames is fused into a fusion point to obtain a set of fusion points corresponding to the pair of synchronization frames; Performing linear fitting on a set of fusion points corresponding to the pair of synchronous frames to obtain a curve corresponding to the pair of synchronous frames; Using at least one curve in one-to-one correspondence between the at least one pair of synchronization frames as the curve corresponding to the current preset duration; Determining a mutation measurement value for each curve corresponding to the current preset duration based on characteristics of the curve corresponding to the current preset duration and characteristics of curves corresponding to historical preset durations of the current preset duration; The mutation measurement value is used to measure whether the curve has a mutation; The current monitoring result of the monitoring area is determined based on the magnitude relationship between the mutation measurement value of each curve corresponding to the current preset time length and a preset threshold.

2. The damage monitoring and early warning method based on the integrated radar and visual device according to claim 1 is characterized in that: There is at least one curve corresponding to the current preset time length, and one curve corresponds one-to-one to a pair of synchronization frames, each pair of synchronization frames has a time point, and each pair of synchronization frames includes a radar scanning frame and an image frame.

3. The damage monitoring and early warning method based on the integrated radar and visual device according to claim 2 is characterized in that: The preset threshold includes: a plurality of different sub-thresholds, wherein the plurality of different sub-thresholds constitute N different threshold intervals with different numerical ranges and successively increasing upper limits; N is a first preset integer greater than 1; the current monitoring result of the monitoring area is determined based on the relationship between the mutation measurement value of each curve corresponding to the current preset time length and the preset threshold, including: When the mutation measurement value of each curve corresponding to the current preset time length belongs to the i-th threshold interval, a monitoring result corresponding to the i-th threshold interval is generated; i is a positive integer, and the value of i ranges from 1 to N; The generated monitoring result is used as the current monitoring result of the monitoring area.

4. The damage monitoring and early warning method based on the integrated radar and visual device according to claim 2 is characterized in that: The determining of the current monitoring result of the monitoring area based on the magnitude relationship between the mutation measurement value of each curve corresponding to the current preset time length and a preset threshold value includes: For the mutation measurement value of each curve corresponding to the current preset duration, when the mutation measurement value is greater than the preset threshold, a pair of time points of synchronization frames corresponding to the curve corresponding to the mutation measurement value is used as a mutation time point; Determining the number of consecutive mutation time points within a preset analysis duration; the consecutive mutation time points are at least two consecutive mutation time points; the preset analysis duration is a duration that at least includes the current preset duration, and the duration of the preset analysis duration is a second duration; When the number of consecutive mutation time points within the preset analysis time length belongs to the jth time point number interval among the preset M different time point number intervals, a monitoring result corresponding to the jth time point number interval is generated; the number ranges of the M different time point number intervals are different, and the upper limits increase successively, j is a positive integer, and the value of j ranges from 1 to M; M is a second preset integer greater than 1; The generated monitoring result is used as the current monitoring result of the monitoring area.

5. The damage monitoring and early warning method based on the integrated radar and visual device according to claim 2 is characterized in that: The historical preset duration of the current preset duration is P preset durations preceding the current preset duration, where P is an integer greater than or equal to 1; and determining a mutation measurement value for each curve corresponding to the current preset duration based on characteristics of the curve corresponding to the current preset duration and characteristics of curves corresponding to historical preset durations of the current preset duration includes: The time points of a pair of synchronous frames corresponding to each curve are taken as the time points corresponding to the curve; Sorting all the curves corresponding to the first P preset durations and the current preset duration in order of the time points to obtain Q curves; wherein the curve corresponding to the current preset duration is the Uth curve to the Qth curve among the Q curves; Q is an integer greater than P, U is a positive integer, and U is less than or equal to Q; For the cth curve among the Uth curve to the Qth curve, calculate the curvature difference between corresponding points in every two adjacent curves in a set of curves consisting of the cth curve and the L curves preceding the cth curve, to obtain a set of curvature differences corresponding to the cth curve; the value of c ranges from 1 to (Q-U+1); and L is a preset positive integer; A mutation measurement value of the c th curve is determined according to a set of curvature differences corresponding to the c th curve.

6. The damage monitoring and early warning method based on the integrated radar and visual device according to claim 5 is characterized in that: Determining a mutation measurement value of the c-th curve according to a set of curvature differences corresponding to the c-th curve includes: Using a set of curvature differences corresponding to the c-th curve as input values ​​of a PELT algorithm, and using the PELT algorithm to calculate a minimum loss value corresponding to the c-th curve; The minimum loss value corresponding to the c-th curve is used as a mutation measurement value of the c-th curve.

7. The damage monitoring and early warning method based on the integrated radar and visual device according to claim 1 is characterized in that: The step of fusing each matching point pair in a set of matching point pairs corresponding to the pair of synchronization frames into a fusion point to obtain a set of fusion points corresponding to the pair of synchronization frames includes: Calculate the variance of all projection coordinates in at least one set of matching point pairs and obtain the first variance ; Calculate the variance of all second position coordinates in the at least one set of matching point pairs to obtain a second variance ; According to the first variance and the second variance , calculate the Kalman gain value of the current preset time length ; The Kalman gain value of the current preset time length is used , performing weighted fusion on the projection coordinates and the coordinate values ​​of the second position coordinates of each matching point pair in a set of matching point pairs corresponding to the pair of synchronous frames, and obtaining a fusion point formed by fusion of each matching point pair; In a set of matching point pairs corresponding to the pair of synchronization frames, all fusion points obtained after weighted fusion of each matching point pair are used as a set of fusion points corresponding to the pair of synchronization frames.

8. A damage monitoring and early warning system based on a radar and visual integrated device, characterized in that: include: A data acquisition module is used to scan the monitoring area in real time using a radar to obtain a real-time radar point cloud map of the monitoring area, and to capture images of the monitoring area in real time using a camera, and to identify the position of each corner reflector in the captured image frame in real time; a data processing module configured to acquire, from the data acquisition module, a radar point cloud image corresponding to each radar scanning frame within a current preset duration, to obtain a plurality of radar point cloud images corresponding to the plurality of radar scanning frames; wherein the points in each radar point cloud image represent the position coordinates of each corner reflector in the monitoring area; and wherein each radar point cloud image represents a set of first position coordinates; and wherein the current preset duration is the preset duration; Each radar point cloud image has a radar timestamp; And, within the current preset time length, a group of second position coordinates corresponding to each image frame is obtained to obtain multiple groups of second position coordinates corresponding to multiple image frames one by one; a group of second position coordinates corresponding to each image frame is the position coordinates of each corner reflector in the monitoring area, and each image frame has a camera timestamp; the multiple image frames are time-synchronized with the multiple radar scanning frames to obtain at least one pair of synchronized frames; each pair of synchronized frames includes a radar scanning frame and an image frame whose time difference between the radar timestamp and the camera timestamp is less than the preset time difference; the camera timestamp or radar timestamp of a pair of synchronized frames is the time point of the pair of synchronized frames; for each pair of synchronized frames, a group of first position coordinates represented by a radar point cloud corresponding to the radar scanning frame in the pair of synchronized frames is projected into the camera coordinate system corresponding to the image frame in the pair of synchronized frames to obtain a group of projected coordinates; the group of projected coordinates is matched with a group of second position coordinates corresponding to the image frame in the pair of synchronized frames to obtain a group of matching point pairs corresponding to the pair of synchronized frames; A set of matching point pairs includes multiple matching point pairs, each matching point pair includes a projection coordinate and a second position coordinate; each matching point pair in the set of matching point pairs corresponding to the pair of synchronous frames is fused into a fusion point to obtain a set of fusion points corresponding to the pair of synchronous frames; a linear fit is performed on the set of fusion points corresponding to the pair of synchronous frames to obtain a curve corresponding to the pair of synchronous frames; and the at least one curve corresponding to the at least one pair of synchronous frames is used as the curve corresponding to the current preset time length; Determining a mutation measurement value for each curve corresponding to the current preset duration based on characteristics of the curve corresponding to the current preset duration and characteristics of curves corresponding to historical preset durations of the current preset duration; The mutation measurement value is used to measure whether the curve has a mutation; Determining a current monitoring result of the monitoring area based on a magnitude relationship between the mutation measurement value of each curve corresponding to the current preset time length and a preset threshold; generating early warning information according to the current monitoring results; Sending the warning information to the information release module; The information release module is used to send the warning information to the target device.

9. The damage monitoring and early warning system based on the integrated radar and visual device according to claim 8 is characterized in that: The data acquisition module is further used to perform damage and disaster identification on the images of the monitoring area taken in real time to obtain image recognition results; The data processing module is further configured to obtain the image recognition result of the current preset time length from the data acquisition module, and send the image recognition result and the warning information together to the information release module through a dedicated warning information channel.

Images