Damage monitoring and early warning method and system based on all-in-one machine

Abstract

The invention discloses a damage monitoring and early warning method and system based on an all-in-one machine, and the method comprises the steps: obtaining a radar point cloud picture corresponding to each radar scanning frame in a current preset time period, and obtaining a plurality of radar point cloud pictures corresponding to a plurality of radar scanning frames in a one-to-one manner; each radar point cloud picture represents the position coordinate of each corner reflector in the monitoring area; acquiring a group of second position coordinates corresponding to each image frame within the current preset duration, and obtaining multiple groups of second position coordinates corresponding to the multiple image frames one by one; the second position coordinate corresponding to each image frame is the position coordinate of each corner reflector; performing synchronization, fusion and linear fitting on the multiple groups of first position coordinates and the multiple groups of second position coordinates represented by the multiple radar point cloud pictures to obtain a curve corresponding to the current preset duration; and determining a current monitoring result of the monitoring area based on the characteristics of the curve corresponding to the current preset duration and the historical preset duration. According to the invention, the monitoring accuracy and timeliness can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent monitoring technology, and in particular relates to a damage monitoring and early warning method and system based on a radar-vision integrated device. Background Art

[0002] On traffic roads, environmental factors such as earthquakes, heavy rains, floods, underground land displacement, etc. can lead to road collapse, ground subsidence, mudslides, etc., which will not only cause road congestion, but also cause vehicle damage and even casualties. Therefore, road monitoring and prediction for the above situations are very important. Through prediction, areas where disasters may occur can be observed in advance, and early warnings can be issued to control them in advance, which can also reduce the losses caused by natural disasters.

[0003] In recent years, the fusion application of radar and vision technology has gradually attracted attention. This fusion technology can make full use of the respective advantages of radar and vision, realize data complementarity and information fusion, and thus improve the accuracy and reliability of monitoring.

[0004] At present, the widely used disaster monitoring and early warning technologies mainly include systems based on synthetic aperture radar (SAR) and synthetic aperture radar interferometry (InSAR). For example, the ground-based synthetic aperture interferometric radar (GBInSAR) system achieves high-resolution imaging through synthetic aperture technology, but it is only suitable for small-scale high-precision monitoring; while satellite synthetic aperture radar can cover a large area, but it is a wide-area periodic monitoring and cannot guarantee real-time performance. In addition, there is currently a technology that deploys cameras in the slope area, collects images of the monitoring area through the camera, and monitors and warns landslide disasters based on image recognition-related algorithms, but the image information that visual technology can provide also has certain limitations. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a damage monitoring and early warning method and system based on a radar-vision integrated device.

[0006] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0007] The present invention provides a damage monitoring and early warning method based on a radar-visual integrated device, comprising:

[0008] Within the current preset time length, a radar point cloud image corresponding to each radar scanning frame is obtained, and a plurality of radar point cloud images corresponding to the plurality of radar scanning frames are obtained; the points in each radar point cloud image represent the position coordinates of each corner reflector in the monitoring area; each radar point cloud image represents a set of first position coordinates; the time length of the preset time length is the first time length;

[0009] Within the current preset time length, a set of second position coordinates corresponding to each image frame is obtained to obtain multiple sets of second position coordinates corresponding to multiple image frames one by one; a set of second position coordinates corresponding to each image frame is the position coordinates of each corner reflector in the monitoring area;

[0010] Performing synchronization processing, fusion processing, and linear fitting processing on the multiple groups of first position coordinates represented by the multiple radar point cloud images and the multiple groups of second position coordinates to obtain a curve corresponding to the current preset time length;

[0011] Based on the characteristics of the curve corresponding to the current preset time length and the characteristics of the curve corresponding to the historical preset time length of the current preset time length, the current monitoring result of the monitoring area is determined.

[0012] The present invention also provides a damage monitoring and early warning system based on a radar-visual integrated machine, comprising:

[0013] A data acquisition module is used to scan the monitoring area in real time by radar to obtain a real-time radar point cloud map of the monitoring area, and to take images of the monitoring area in real time by a camera, and to identify the position of each corner reflector in the taken image frame in real time;

[0014] A data processing module is used to obtain a radar point cloud image corresponding to each radar scanning frame from the data acquisition module within a current preset time length, and obtain a plurality of radar point cloud images corresponding to the plurality of radar scanning frames one by one; the points in each radar point cloud image represent the position coordinates of each corner reflector in the monitoring area; each radar point cloud image represents a set of first position coordinates; the duration of the current preset time length is the preset time length; and, within the current preset time length, obtain a set of second position coordinates corresponding to each image frame, and obtain a plurality of sets of second position coordinates corresponding to the plurality of image frames one by one; a set of second position coordinates corresponding to each image frame is the position coordinates of each corner reflector in the monitoring area; synchronize, fuse and linearly fit the plurality of sets of first position coordinates represented by the plurality of radar point cloud images, and the plurality of sets of second position coordinates, and obtain a curve corresponding to the current preset time length; determine the current monitoring result of the monitoring area based on the characteristics of the curve corresponding to the current preset time length and the characteristics of the curve corresponding to the historical preset time length of the current preset time length; generate warning information according to the current monitoring result; and send the warning information to the information release module;

[0015] The information publishing module is used to send the warning information to the target device.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] The present invention combines radar and camera to realize all-weather real-time monitoring of the monitoring area, and is not easily disturbed by natural factors such as vegetation, weather and terrain, and can work stably in complex environments such as rainy days, foggy days and nights, ensuring the continuity and accuracy of monitoring data; the present invention synchronizes, fuses and linearly fits the coordinates of the corner reflector detected by the radar with the coordinates of the corner reflector detected by the camera to obtain a corresponding curve, and performs damage monitoring of the monitoring area based on the characteristics of the obtained curve, and can accurately and timely identify slight position changes of the corner reflector and changes in the curve shape, and has strong anti-interference ability, further improving the timeliness, reliability and accuracy of monitoring.

[0018] The present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of a damage monitoring and early warning method based on a radar-visual integrated device provided by an embodiment of the present invention;

[0020] Figure 2 It is a schematic diagram of an exemplary equipment layout of a monitoring area provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0022] Figure 1 is a flow chart of a damage monitoring and early warning method based on a radar-visual integrated device provided by an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0023] S101. Within the current preset time length, obtain a radar point cloud image corresponding to each radar scanning frame, and obtain multiple radar point cloud images corresponding to multiple radar scanning frames one by one; the points in each radar point cloud image represent the position coordinates of each corner reflector in the monitoring area; each radar point cloud image represents a set of first position coordinates; the time length of the preset time length is the first time length.

[0024] Here, the value of the first time length can be set according to actual needs, for example, it can be 2 seconds or 4 seconds, etc., and the present invention is not limited to this.

[0025] Here, multiple corner reflectors are fixedly installed in different areas of the monitoring area, and radar and camera are also installed in the monitoring area. In some embodiments, the radar and camera can be included in one device, for example, on a radar-vision integrated device; in other embodiments, the radar and camera can be two different devices and are separately arranged.

[0026] The damage described in the present invention may be man-made damage, such as man-made damage to buildings, mountains, roads, rivers, etc., or natural damage (i.e., natural disasters), such as natural road collapse, mountain collapse, mud and rock flow, landslide, etc. Accordingly, the monitoring area described in the present invention may be buildings, roads, mountains, etc.

[0027] Exemplarily, when the radar and camera include a radar-vision integrated device, and the monitoring area is a road, the radar-vision integrated device can be set up above the road (for example, 5 meters) at intervals (for example, 500 meters) above the road, while ensuring the effective detection range of the radar-vision integrated device, to ensure that the radar-vision integrated device can cover the entire area; then, use a bracket to firmly fix the radar-vision integrated device to the ground or other stable structure to ensure that it will not move during long-term use. At the same time, corner reflectors are arranged at intervals (for example, 10 meters) on both sides of the road in the monitoring area to ensure that the corner reflectors on each side of the road in the monitoring area are connected to form a curve. Then, the corner reflectors are firmly mounted on the ground or rock and ensure that the corner reflectors are mounted on a bracket perpendicular to the ground to avoid the influence of external factors such as wind and rain. Exemplarily, Figure 2 This is an exemplary equipment layout diagram of the monitoring area, where number 1 represents the corner reflector and number 2 represents the radar-visual integrated device. After completing these layouts, in order to eliminate or reduce system errors and improve the measurement accuracy and reliability of the millimeter-wave radar, the position of the corner reflector can be calibrated, and during the calibration process, it should be noted that the position and angle of the radar and corner reflector should be adjusted according to the initial measurement data to ensure that the damage on the road can be accurately monitored in the future.

[0028] Here, the second position coordinates are obtained by the camera performing target recognition on the image frame through a target recognition algorithm, for example, the YoLov8 target recognition algorithm.

[0029] S102. Within the current preset time length, obtain a set of second position coordinates corresponding to each image frame to obtain multiple sets of second position coordinates corresponding to multiple image frames one by one; the set of second position coordinates corresponding to each image frame is the position coordinates of each corner reflector in the monitoring area.

[0030] It should be noted that the above S101 and S102 are performed simultaneously.

[0031] S103, performing synchronization processing, fusion processing and linear fitting processing on multiple groups of first position coordinates represented by multiple radar point cloud images and multiple groups of second position coordinates to obtain a curve corresponding to the current preset time length.

[0032] S104: Determine a current monitoring result of the monitoring area based on the characteristics of the curve corresponding to the current preset time length and the characteristics of the curve corresponding to the historical preset time length of the current preset time length.

[0033] The present invention combines radar and camera to realize all-weather real-time monitoring of the monitoring area, and is not easily disturbed by natural factors such as vegetation, weather and terrain, and can work stably in complex environments such as rainy days, foggy days and nights, ensuring the continuity and accuracy of monitoring data; the present invention synchronizes, fuses and linearly fits the coordinates of the corner reflector detected by the radar with the coordinates of the corner reflector detected by the camera to obtain a corresponding curve, and performs damage monitoring of the monitoring area based on the characteristics of the obtained curve, and can accurately and timely identify slight position changes of the corner reflector and changes in the curve shape, and has strong anti-interference ability, further improving the timeliness, reliability and accuracy of monitoring.

[0034] In the present invention, each radar point cloud image has a radar timestamp, and each image frame has a camera timestamp; based on this, the above S103 is implemented by the following steps:

[0035] S1031. Time synchronize multiple image frames with multiple radar scanning frames to obtain at least one pair of synchronization frames; each pair of synchronization frames includes a radar scanning frame and an image frame, wherein the time difference between the radar timestamp and the camera timestamp is less than a preset time difference; the camera timestamp or the radar timestamp of a pair of synchronization frames is the time point of the pair of synchronization frames.

[0036] Here, time synchronization can be performed based on the camera timestamp. Specifically, for each of the multiple image frames, a radar scan frame whose time difference between the radar timestamp and the camera timestamp of the image frame is less than the preset time difference can be found from the multiple radar scan frames as the radar scan frame synchronized with the image frame. Through this method, after executing the time synchronization of multiple image frames and multiple radar scan frames, at least one pair of synchronized frames can be obtained. The introduction of a time synchronization mechanism can improve the integration of radar and camera data and the accuracy of analysis, which has a direct impact on the response time and early warning capability of the real-time monitoring system. Through precise time alignment, rapid processing and timely feedback of real-time data can be ensured, thereby improving the efficiency and effectiveness of monitoring.

[0037] It should be noted that the system clocks of radar and camera need to be regularly adjusted to ensure that the clock synchronization is accurate to the millisecond level or higher. This can be achieved through methods such as the Network Time Protocol (NTP) to synchronize the system clock with the Universal Time Coordinated (UTC).

[0038] In some embodiments, in order to improve the purity and accuracy of the radar point cloud image, thereby improving the data quality, after acquiring the radar point cloud image, an adaptive Kalman filtering algorithm can be used to filter the radar point cloud image for noise, remove unnecessary interference signals, and obtain a radar point cloud image without noise for subsequent data processing.

[0039] S1032. For each pair of synchronous frames, project a set of first position coordinates represented by a radar point cloud image corresponding to a radar scanning frame in the pair of synchronous frames into a camera coordinate system corresponding to an image frame in the pair of synchronous frames to obtain a set of projection coordinates.

[0040] Specifically, when a set of first position coordinates represented by the radar point cloud in a pair of synchronous frames is projected to the camera coordinate system corresponding to the image frame in the pair of synchronous frames, the first position coordinates can be projected one by one through the homography matrix perspective transformation method to ensure the high spatial consistency of the camera and radar data. In addition, in the coordinate transformation process, dynamic calibration technology can be used. The dynamic calibration technology allows the system to adaptively adjust the homography matrix according to the position change of the corner reflector to ensure the accuracy and real-time performance of the transformation process, and enhances the flexibility and robustness of position coordinate tracking. Through these steps, accurate monitoring and tracking of the corner reflector position can be achieved, ensuring the efficient operation of the monitoring system in complex environments.

[0041] S1033. Match a set of projection coordinates with a set of second position coordinates corresponding to the image frames in the pair of synchronization frames to obtain a set of matching point pairs corresponding to the pair of synchronization frames; a set of matching point pairs includes multiple matching point pairs, and a matching point pair includes a projection coordinate and a second position coordinate.

[0042] Specifically, when matching the projection coordinates with the second position coordinates, the nearest neighbor algorithm can be used for matching, and the distance error range can be set to screen and match the camera projection points within each radar distance error range. Specifically, first, a reasonable distance error range is defined to limit the maximum acceptable matching distance; then, for each projection point (i.e., each projection coordinate), the camera point (i.e., the second position coordinate) is searched within this distance error range, and a camera point with the smallest distance to the projection point is selected from the searched camera points, and the projection point and the camera point are used as a matching point pair. The effective association of radar data and camera data is achieved through matching.

[0043] S1034. Each matching point pair in a group of matching point pairs corresponding to the pair of synchronization frames is merged into a fusion point to obtain a group of fusion points corresponding to the pair of synchronization frames.

[0044] Specifically, S1034 is implemented by the following steps:

[0045] S1. Calculate the variance of all projection coordinates in at least one set of matching point pairs to obtain the first variance

[0046] Exemplarily, when at least one set of matching point pairs obtained after executing S1033 above has a total of n matching point pairs, the first variance The expression is as follows:

[0047]

[0048] Among them, X1+X2+…+X n are the n projection coordinates of the n matching point pairs.

[0049] S2. Calculate the variance of all second position coordinates in at least one set of matching point pairs to obtain a second variance.

[0050] Exemplarily, when the at least one group of matching points obtained after executing S1033 above has a total of n matching points, the second variance The expression is as follows:

[0051]

[0052]

[0053] Among them, Y1+Y2+…+Y n are the n second position coordinates of the n matching point pairs.

[0054] S3. According to the first variance and the second variance Calculate the Kalman gain value K for the current preset duration.

[0055] Exemplarily, the expression of the Kalman gain value K is as follows:

[0056]

[0057] S4. Using the Kalman gain value K of the current preset time length, weighted fusion is performed on the projection coordinates and the coordinate values ​​of the second position coordinates of each matching point pair in a set of matching point pairs corresponding to the pair of synchronization frames to obtain a fusion point formed by the fusion of each matching point pair.

[0058] For example, when the projection coordinates of a matching point pair are The second position coordinates are When the fusion is formed, a fusion point The expression is: is the horizontal coordinate value after fusion, is the vertical coordinate value after fusion.

[0059] S5. Among a group of matching point pairs corresponding to the pair of synchronization frames, all fusion points obtained after weighted fusion of each matching point pair are used as a group of fusion points corresponding to the pair of synchronization frames.

[0060] S1035 . Perform linear fitting on a group of fusion points corresponding to the pair of synchronization frames to obtain a curve corresponding to the pair of synchronization frames.

[0061] S1036: Use at least one curve in one-to-one correspondence between at least one pair of synchronization frames as the curve corresponding to the current preset duration.

[0062] Specifically, the least square method may be used to perform curve fitting on each group of fusion points, or polynomial fitting may be combined to perform curve fitting using a polynomial of an appropriate order, thereby obtaining at least one curve corresponding to at least one pair of synchronization frames.

[0063] In the present invention, there is at least one curve corresponding to the current preset duration, and one curve corresponds to a pair of synchronization frames one by one, each pair of synchronization frames has a time point, and each pair of synchronization frames includes a radar scanning frame and an image frame; the above S104 is implemented by the following steps:

[0064] S1041. Based on the characteristics of the curve corresponding to the current preset duration and the characteristics of the curve corresponding to the historical preset duration of the current preset duration, determine a mutation measurement value F(t) for each curve corresponding to the current preset duration; the mutation measurement value is used to measure whether the curve has undergone a mutation.

[0065] In the present invention, the historical preset duration of the current preset duration is the first P preset durations of the current preset duration, where P is an integer greater than or equal to 1 (for example, P may be 1). Based on this, S1041 may be implemented by the following steps:

[0066] S11. Taking a pair of time points of synchronization frames corresponding to each curve as the time points corresponding to the curve.

[0067] S12. Sort all curves corresponding to the first P preset durations and the current preset duration in order of time points to obtain Q curves; wherein the curve corresponding to the current preset duration is the Uth curve to the Qth curve among the Q curves; Q is an integer greater than P, U is a positive integer, and U is less than or equal to Q.

[0068] Here, the number of curves corresponding to each preset duration may not be the same.

[0069] For example, when Q is 7 and U is 3, the curves corresponding to the current preset duration are the last 4 curves of the 7 curves.

[0070] S13. For the c-th curve among the U-th curve to the Q-th curve, calculate the curvature difference between each corresponding point in every two adjacent curves in a group of curves consisting of the c-th curve and the first L curves of the c-th curve, and obtain a group of curvature differences corresponding to the c-th curve; the value of c is 1 to (Q-U+1); L is a preset positive integer.

[0071] Here, the value of L can be set according to actual needs, for example, L can be 3 or 4, or 2, etc. Continuing to take the case where the curve corresponding to the current preset duration is the last 4 curves of the 7 curves as an example, when L is 2 and the current c is 3, the curvature difference between each corresponding point in the first curve and the second curve, and the curvature difference between each corresponding point in the second curve and the third curve are calculated among the three curves of the 7 curves, namely, the first curve, the second curve and the third curve, and the curvature difference between each corresponding point in the second curve and the third curve, and the obtained curvature differences are used as a set of curvature differences corresponding to the third curve of the 7 curves. Exemplarily, when each curve contains 2 points, a set of curvature differences corresponding to the third curve is expressed as: {ΔR1, ΔR2, ΔR3, ΔR4}, wherein ΔR1 represents the curvature difference between the first curve among the 7 curves and the first point on the second curve, ΔR2 represents the curvature difference between the first curve among the 7 curves and the second point on the second curve, ΔR3 represents the curvature difference between the second curve among the 7 curves and the first point on the third curve, and ΔR4 represents the curvature difference between the second curve among the 7 curves and the second point on the third curve.

[0072] S14. Determine a mutation measurement value of the c th curve according to a set of curvature differences corresponding to the c th curve.

[0073] Specifically, a set of curvature differences corresponding to the c-th curve can be used as the input value of the PELT algorithm, and the PELT algorithm is used to calculate the minimum loss value F(t) corresponding to the c-th curve. c ; The minimum loss value corresponding to the c-th curve is used as a mutation measurement value of the c-th curve. For example, the mutation measurement value F(t) corresponding to the third curve is 3 The expression is as follows:

[0074]

[0075] Among them, β is a penalty term used to balance the loss and the number of change points.

[0076] S1042: Determine the current monitoring result of the monitoring area based on the relationship between the mutation measurement value of each curve corresponding to the current preset time length and the preset threshold.

[0077] In some embodiments, the above-mentioned preset threshold includes: a plurality of different sub-thresholds, and the plurality of different sub-thresholds constitute N different threshold intervals with different numerical ranges and upper limits increasing in sequence; N is a first preset integer greater than 1; based on this, the above-mentioned S1042 can be implemented by the following steps:

[0078] S21. When the mutation measurement value of each curve corresponding to the current preset time length belongs to the i-th threshold interval, a monitoring result corresponding to the i-th threshold interval is generated; i is a positive integer, and the value of i ranges from 1 to N;

[0079] S22. Use the generated monitoring result as the current monitoring result of the monitoring area.

[0080] Here, the number of sub-thresholds and the value of N can be set according to actual needs, and the present invention does not limit this. Exemplarily, the number of sub-thresholds can be 6, and correspondingly, N can be 3, then the above-mentioned multiple different sub-thresholds constitute 3 different threshold intervals with different numerical ranges and upper limits increasing in sequence; these 3 different threshold intervals correspond to 3 mutation severity levels, that is, the mutation severity level corresponding to the first threshold interval is low, the mutation severity level corresponding to the second threshold interval is medium, and the mutation severity level corresponding to the third threshold interval is high. Therefore, when the mutation measurement value of each curve corresponding to the current preset time length belongs to the first threshold interval, a monitoring result for characterizing a low degree of mutation can be generated, when the mutation measurement value of each curve corresponding to the current preset time length belongs to the second threshold interval, a monitoring result for characterizing a medium degree of mutation can be generated, and when the mutation measurement value of each curve corresponding to the current preset time length belongs to the third threshold interval, a monitoring result for characterizing a high degree of mutation can be generated.

[0081] In some embodiments, the above S1042 may also be implemented by the following steps:

[0082] S31. For the mutation measurement value of each curve corresponding to the current preset duration, when the mutation measurement value is greater than a preset threshold, a pair of time points of synchronization frames corresponding to the curve corresponding to the mutation measurement value is taken as a mutation time point.

[0083] Here, the value of the preset threshold can be set according to actual needs, and the present invention does not limit this.

[0084] S32, determining the number of consecutive mutation time points within a preset analysis duration; the consecutive mutation time points are at least two mutation time points that are consecutive in time; the preset analysis duration is a duration that at least includes the current preset duration, and the time length of the preset analysis duration is the second time length.

[0085] Exemplarily, the preset analysis time length can be set according to actual needs, and the present invention is not limited to this. For example, the preset analysis time length can be 4 seconds (seconds), and the preset time length is 1 second.

[0086] The meaning of the continuity of time points described in the present invention is as follows: for example, when the current preset time length corresponds to 3 curves, the time points corresponding to the three curves are three time points that are continuous, the time point corresponding to the first curve of the three curves and the time point corresponding to the second curve are two time points that are continuous, and the time point corresponding to the second curve of the three curves and the time point corresponding to the third curve are two time points that are continuous.

[0087] Exemplarily, when the preset analysis time length is 4s and the preset time length is 2s, the number of consecutive mutation time points in the consecutive 4s consisting of the current 2s and the previous 2s of the current 2s can be determined through S32.

[0088] S33. When the number of continuous mutation time points within the preset analysis time length belongs to the jth time point quantity interval among the preset M different time point quantity intervals, a monitoring result corresponding to the jth time point quantity interval is generated; the quantity ranges of the M different time point quantity intervals are different, and the upper limit values ​​increase successively, j is a positive integer, and the value of j is 1 to M; M is a second preset integer greater than 1.

[0089] Here, the value of M can be set according to actual needs, and the present invention does not limit this.

[0090] Exemplarily, M can be 3, then the above-mentioned M different time point quantity intervals have different quantity ranges and three different time point quantity intervals with increasing upper limits; these three different time point quantity intervals correspond one-to-one to three mutation severity levels, that is, the mutation severity level corresponding to the first time point quantity interval is low, the mutation severity level corresponding to the second time point quantity interval is medium, and the mutation severity level corresponding to the third time point quantity interval is high. Therefore, when the number of continuous mutation time points within the preset analysis time length belongs to the first time point quantity interval, a monitoring result for characterizing a low degree of mutation can be generated; when the number of continuous mutation time points within the preset analysis time length belongs to the second time point quantity interval, a monitoring result for characterizing a medium degree of mutation can be generated; when the number of continuous mutation time points within the preset analysis time length belongs to the third time point quantity interval, a monitoring result for characterizing a high degree of mutation can be generated.

[0091] S34. Use the generated monitoring result as the current monitoring result of the monitoring area.

[0092] In some embodiments, after the current monitoring result is generated by the above S104, step S105 is further included:

[0093] S105. Generate early warning information based on current monitoring results.

[0094] Exemplarily, when the current monitoring results are used to characterize the low degree of mutation, the warning information can be a warning information that the risk level of damage in the monitoring area is low; this situation indicates that there may be slight geological activities. In this case, the risk of geological changes is low, and low-level warning information is used to remind relevant personnel to pay attention to observation and recording, but no immediate action is required. At this time, it is mainly to detect potential problems early, conduct continuous monitoring, and ensure preparation before the situation deteriorates. When the current monitoring results are used to characterize the medium degree of mutation, the warning information can be a warning information that the risk level of damage in the monitoring area is low; this situation indicates that there is a greater risk of geological activities, and a medium-level warning is used to remind relevant personnel to be vigilant and prepare for possible geological activities. This may include increasing the frequency of monitoring, preparing emergency supplies, and notifying relevant departments and personnel to take preventive measures to reduce potential losses and dangers. When the current monitoring results are used to characterize the high degree of mutation, the warning information can be a warning information that the risk level of damage in the monitoring area is high; it means that a serious geological disaster may occur, a high-level warning is issued, and emergency response measures are initiated. These measures may include immediate evacuation of personnel, blocking dangerous areas, launching emergency rescue plans, and notifying the government and emergency management departments to coordinate rescue work. High-level warnings are intended to protect life and property safety to the greatest extent and reduce the impact and losses caused by disasters.

[0095] In some embodiments, the device that executes the above-mentioned damage monitoring and early warning method based on the integrated radar and visual device is a data processing device, for example, an edge server. The edge server can obtain real-time data of the monitoring area detected by radar and camera from the integrated radar and visual device set in the monitoring area, or obtain real-time data of the monitoring area detected by radar and camera from the radar and camera set in the monitoring area.

[0096] In some embodiments, the camera or the integrated radar and vision device in the monitoring area is also used to identify damage and disasters on the real-time images of the monitoring area to obtain image recognition results, which may be, for example, whether there is a road collapse, whether there is road damage, or whether there is water accumulation on the road. The edge server may also obtain real-time image recognition results from the camera or the integrated radar and vision device.

[0097] The present invention also provides a damage monitoring and early warning system based on a radar-vision integrated machine, comprising: a data acquisition module, a data processing module and an information publishing module.

[0098] The data acquisition module is used to scan the monitoring area in real time through radar to obtain a real-time radar point cloud map of the monitoring area, and to take images of the monitoring area in real time through a camera, and to identify the position of each corner reflector in the captured image frame in real time.

[0099] The data processing module is used to obtain, from the data acquisition module, a radar point cloud image corresponding to each radar scanning frame within a current preset time length, and obtain a plurality of radar point cloud images corresponding to the plurality of radar scanning frames one by one; the points in each radar point cloud image represent the position coordinates of each corner reflector in the monitoring area; each radar point cloud image represents a set of first position coordinates; the duration of the current preset time length is the preset time length; and, within the current preset time length, obtain a set of second position coordinates corresponding to each image frame, and obtain a plurality of sets of second position coordinates corresponding to the plurality of image frames one by one; a set of second position coordinates corresponding to each image frame is the position coordinates of each corner reflector in the monitoring area; synchronize, fuse and linearly fit the plurality of sets of first position coordinates represented by the plurality of radar point cloud images, and the plurality of sets of second position coordinates, and obtain a curve corresponding to the current preset time length; determine the current monitoring result of the monitoring area based on the characteristics of the curve corresponding to the current preset time length and the characteristics of the curve corresponding to the historical preset time length of the current preset time length; generate warning information according to the current monitoring result; and send the warning information to the information release module.

[0100] The information release module is used to send the risk warning level to the target device.

[0101] In some embodiments, the data acquisition module is also used to identify damage and disasters in images of the monitoring area taken in real time to obtain image recognition results; the data processing module is also used to obtain image recognition results of the current preset time length from the data acquisition module, and send the image recognition results and risk warning levels to the information release module through a dedicated warning information channel.

[0102] In some embodiments, the data acquisition module, the data processing module and the information release module may all be hardware modules. Exemplarily, the data acquisition module may be a radar-visual integrated device or a radar and a camera; the data processing module may be an edge server that is in communication with the radar-visual integrated device (or a radar and a camera); the information release module may be a cloud server, and the cloud server may be a server of the monitoring center of the monitoring area. In some embodiments, the data acquisition module and the data processing module may also be software programs that require the use of hardware to implement corresponding functions.

[0103] Here, the target device can be an electronic device installed in the monitoring area, such as a road information board, as well as the mobile phones of vehicles and personnel in the monitoring area, etc. It can also be a display device in the monitoring center of the monitoring area, or a mobile phone, computer, etc. of relevant personnel in the monitoring center of the monitoring area. The early warning information and image recognition results can be sent in the form of text messages, emails, APP push, etc. Sending early warning information to relevant departments and personnel quickly can ensure the accuracy and timeliness of the early warning information, so that relevant personnel can take timely response measures. In particular, by sending early warning information to the road information board, the early warning information can be displayed on the information board as soon as it is available, reminding passing vehicles to pay attention and reducing the occurrence of safety accidents.

[0104] Here, the warning information release channel can be a communication channel dedicated to transmitting warning information and image recognition results. If the possibility of damage is predicted, the warning information can be uploaded through a dedicated channel in a timely manner to reduce the delay in the transmission process.

[0105] In some embodiments, the data processing module can also send the acquired radar data and image data, as well as the processed data, to the information publishing module through the daily data backup channel for daily backup. Similarly, the daily data backup channel can also be a communication channel.

[0106] In general, the present invention has the following technical effects:

[0107] In terms of real-time performance: the present invention can achieve all-weather real-time monitoring, and transmit data to the cloud server through two data upload channels, one for daily data backup and the other for early warning information release, to ensure the rapid transmission of early warning information. Once the possibility of a disaster is predicted, the early warning information will be sent to the communication terminals of relevant departments or personnel as soon as possible to ensure that they can take necessary response measures as soon as possible.

[0108] Cost: Compared with other technical solutions, the cost of radar-vision integrated machine is lower, and the cost of corner reflectors is also lower. Radar can provide accurate location information and deformation data, while visual technology can provide clear images and feature recognition capabilities. By fusing the data of the two, we can have a more comprehensive understanding of the road conditions in the monitored area, and timely discover and warn of potential road disasters and other damages.

[0109] Accuracy: The present invention can ensure reliable geological disaster early warning support under various environmental conditions, is not easily interfered by natural factors such as vegetation, weather and terrain, and can work stably in complex environments such as rainy days, foggy days and nights, ensuring the continuity and accuracy of monitoring data. The integrated radar vision device can accurately identify small position changes and curve shape changes of the corner reflector. The corner reflector design with strong anti-interference ability improves the reliability of the signal and the accuracy of the monitoring data. By comparing and analyzing historical data and real-time data, the accuracy of anomaly detection is improved, and early signals of damage such as geological disasters can be more accurately identified.

[0110] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0111] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0112] In the specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality of components. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good effects.

[0113] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A damage monitoring and early warning method based on a radar-visual integrated device, characterized in that: include: Within the current preset time length, a radar point cloud image corresponding to each radar scanning frame is obtained, and multiple radar point cloud images corresponding to multiple radar scanning frames are obtained; The points in each radar point cloud image represent the position coordinates of each corner reflector in the monitoring area; each radar point cloud image represents a set of first position coordinates; the time length of the preset time length is the first time length; Within the current preset time length, a set of second position coordinates corresponding to each image frame is obtained to obtain multiple sets of second position coordinates corresponding to multiple image frames one by one; a set of second position coordinates corresponding to each image frame is the position coordinates of each corner reflector in the monitoring area; Performing synchronization processing, fusion processing, and linear fitting processing on the multiple groups of first position coordinates represented by the multiple radar point cloud images and the multiple groups of second position coordinates to obtain a curve corresponding to the current preset time length; Based on the characteristics of the curve corresponding to the current preset time length and the characteristics of the curve corresponding to the historical preset time length of the current preset time length, the current monitoring result of the monitoring area is determined.

2. The damage monitoring and early warning method based on the integrated radar and visual device according to claim 1 is characterized in that: There is at least one curve corresponding to the current preset time length, and one curve corresponds to a pair of synchronization frames one by one, each pair of synchronization frames has a time point, and each pair of synchronization frames includes a radar scanning frame and an image frame; the current monitoring result of the monitoring area is determined based on the characteristics of the curve corresponding to the current preset time length and the characteristics of the curve corresponding to the historical preset time length of the current preset time length, including: Based on the characteristics of the curve corresponding to the current preset duration and the characteristics of the curve corresponding to the historical preset duration of the current preset duration, a mutation measurement value of each curve corresponding to the current preset duration is determined; the mutation measurement value is used to measure whether the curve has a mutation; Based on the magnitude relationship between the mutation measurement value of each curve corresponding to the current preset time length and a preset threshold, the current monitoring result of the monitoring area is determined.

3. The damage monitoring and early warning method based on the integrated radar and visual device according to claim 2 is characterized in that: The preset threshold includes: a plurality of different sub-thresholds, wherein the plurality of different sub-thresholds constitute N different threshold intervals with different numerical ranges and upper limits increasing in sequence; N is a first preset integer greater than 1; the current monitoring result of the monitoring area is determined based on the relationship between the mutation measurement value of each curve corresponding to the current preset time length and the preset threshold, including: When the mutation measurement value of each curve corresponding to the current preset time length belongs to the i-th threshold interval, a monitoring result corresponding to the i-th threshold interval is generated; i is a positive integer, and the value of i ranges from 1 to N; The generated monitoring result is used as the current monitoring result of the monitoring area.

4. The damage monitoring and early warning method based on the integrated radar and vision device according to claim 2 is characterized in that: The determining of the current monitoring result of the monitoring area based on the relationship between the mutation measurement value of each curve corresponding to the current preset time length and a preset threshold value includes: For the mutation measurement value of each curve corresponding to the current preset duration, when the mutation measurement value is greater than the preset threshold, a pair of time points of synchronization frames corresponding to the curve corresponding to the mutation measurement value is taken as a mutation time point; Determine the number of consecutive mutation time points within a preset analysis time; the consecutive mutation time points are at least two mutation time points that are consecutive in time; the preset analysis time is a time that at least includes the current preset time, and the time length of the preset analysis time is a second time length; When the number of continuous mutation time points within the preset analysis time length belongs to the jth time point quantity interval among the preset M different time point quantity intervals, a monitoring result corresponding to the jth time point quantity interval is generated; the number ranges of the M different time point quantity intervals are different, and the upper limit values ​​increase successively, j is a positive integer, and the value of j ranges from 1 to M; M is a second preset integer greater than 1; The generated monitoring result is used as the current monitoring result of the monitoring area.

5. The damage monitoring and early warning method based on the integrated radar and vision device according to claim 2 is characterized in that: The historical preset duration of the current preset duration is the first P preset durations of the current preset duration, where P is an integer greater than or equal to 1; and determining a mutation measurement value of each curve corresponding to the current preset duration based on the characteristics of the curve corresponding to the current preset duration and the characteristics of the curve corresponding to the historical preset duration of the current preset duration includes: The time points of a pair of synchronization frames corresponding to each curve are taken as the time points corresponding to the curve; According to the order of the time points, all the curves corresponding to the first P preset durations and the current preset duration are sorted to obtain Q curves; wherein the curve corresponding to the current preset duration is the Uth curve to the Qth curve among the Q curves; Q is an integer greater than P, U is a positive integer, and U is less than or equal to Q; For the cth curve among the Uth curve to the Qth curve, calculate the curvature difference between corresponding points in every two adjacent curves in a group of curves consisting of the cth curve and the first L curves of the cth curve, and obtain a group of curvature differences corresponding to the cth curve; the value of c is 1 to (Q-U+1); L is a preset positive integer; A mutation measurement value of the c th curve is determined according to a set of curvature differences corresponding to the c th curve.

6. The damage monitoring and early warning method based on the integrated radar and visual device according to claim 5 is characterized in that: Determining a mutation measurement value of the c-th curve according to a set of curvature differences corresponding to the c-th curve includes: A set of curvature differences corresponding to the c-th curve is used as an input value of a PELT algorithm, and the PELT algorithm is used to calculate a minimum loss value corresponding to the c-th curve; The minimum loss value corresponding to the c-th curve is used as a mutation measurement value of the c-th curve.

7. The damage monitoring and early warning method based on the integrated radar and visual device according to claim 1 is characterized in that: Each radar point cloud image has a radar timestamp, and each image frame has a camera timestamp; the multiple groups of first position coordinates represented by the multiple radar point cloud images and the multiple groups of second position coordinates are synchronized, fused, and linearly fitted to obtain a curve corresponding to the current preset time length, including: The multiple image frames and the multiple radar scan frames are time synchronized to obtain at least one pair of synchronized frames; each pair of synchronized frames includes a radar scan frame and an image frame, the time difference between the radar timestamp and the camera timestamp of which is less than a preset time difference; the camera timestamp or the radar timestamp of a pair of synchronized frames is the time point of the pair of synchronized frames; For each pair of synchronous frames, projecting a set of first position coordinates represented by a radar point cloud image corresponding to a radar scanning frame in the pair of synchronous frames into a camera coordinate system corresponding to an image frame in the pair of synchronous frames to obtain a set of projection coordinates; Matching the set of projection coordinates with a set of second position coordinates corresponding to the image frame in the pair of synchronization frames to obtain a set of matching point pairs corresponding to the pair of synchronization frames; a set of matching point pairs includes multiple matching point pairs, and a matching point pair includes a projection coordinate and a second position coordinate; Each matching point pair in a set of matching point pairs corresponding to the pair of synchronization frames is fused into a fusion point to obtain a set of fusion points corresponding to the pair of synchronization frames; Performing linear fitting on a set of fusion points corresponding to the pair of synchronization frames to obtain a curve corresponding to the pair of synchronization frames; At least one curve in one-to-one correspondence between the at least one pair of synchronization frames is used as the curve corresponding to the current preset time length.

8. The damage monitoring and early warning method based on the integrated radar and visual device according to claim 7 is characterized in that: The step of fusing each matching point pair in a set of matching point pairs corresponding to the pair of synchronization frames into a fusion point to obtain a set of fusion points corresponding to the pair of synchronization frames includes: Calculate the variance of all projection coordinates in the at least one set of matching point pairs to obtain a first variance Calculate the variance of all second position coordinates in the at least one set of matching point pairs to obtain a second variance According to the first variance and the second variance Calculate the Kalman gain value K of the current preset time length; Using the Kalman gain value K of the current preset time length, weighted fusion is performed on the projection coordinates and the coordinate values ​​of the second position coordinates in each matching point pair in a set of matching point pairs corresponding to the pair of synchronization frames, so as to obtain a fusion point formed by fusion of each matching point pair; Among a group of matching point pairs corresponding to the pair of synchronization frames, all fusion points obtained after weighted fusion of each matching point pair are used as a group of fusion points corresponding to the pair of synchronization frames.

9. A damage monitoring and early warning system based on a radar-visual integrated device, characterized in that: include: A data acquisition module is used to scan the monitoring area in real time by radar to obtain a real-time radar point cloud map of the monitoring area, and to take images of the monitoring area in real time by a camera, and to identify the position of each corner reflector in the taken image frame in real time; A data processing module is used to obtain, from the data acquisition module, a radar point cloud image corresponding to each radar scanning frame within a current preset time length, to obtain a plurality of radar point cloud images corresponding to the plurality of radar scanning frames one by one; the points in each radar point cloud image represent the position coordinates of each corner reflector in the monitoring area; each radar point cloud image represents a set of first position coordinates; the duration of the current preset time length is the preset time length; And, within the current preset time length, a set of second position coordinates corresponding to each image frame is obtained to obtain multiple sets of second position coordinates corresponding to multiple image frames one by one; a set of second position coordinates corresponding to each image frame is the position coordinates of each corner reflector in the monitoring area; Performing synchronization processing, fusion processing, and linear fitting processing on the multiple groups of first position coordinates represented by the multiple radar point cloud images and the multiple groups of second position coordinates to obtain a curve corresponding to the current preset time length; Determine a current monitoring result of the monitoring area based on the characteristics of the curve corresponding to the current preset time length and the characteristics of the curve corresponding to the historical preset time length of the current preset time length; Generate early warning information according to the current monitoring results; Sending the warning information to the information release module; The information publishing module is used to send the warning information to the target device.

10. The damage monitoring and early warning method based on the integrated radar and visual device according to claim 9 is characterized in that: The data acquisition module is also used to identify damage and disasters on the images of the monitoring area taken in real time to obtain image recognition results; The data processing module is also used to obtain the image recognition result of the current preset time length from the data acquisition module, and send the image recognition result and the warning information to the information release module through a dedicated warning information channel.

Images