Multi-sensor based waterway patrol method and apparatus

Abstract

The present disclosure provides a multi-sensor-based water area patrol method and device, relating to the technical field of monitoring. The method comprises: acquiring visible light data, infrared data, and echo data of a patrol area in real time; determining a label of a real-time environmental state of the patrol area based on the visible light data and the echo data; dynamically allocating weights of the visible light data, the infrared data, and the echo data in a multi-modal hidden danger identification network based on the identified label of the real-time environmental state; performing weighted fusion analysis on multi-source data collected by the multi-sensor based on the allocated weights of each type of data, to identify whether a preset abnormal event exists in the patrol area; if the preset abnormal event is identified, triggering a corresponding level of patrol prompt according to the type and location of the preset abnormal event, and marking the location of the abnormal event on an electronic map of the patrol area. Based on the technical solution, the accuracy of water area monitoring can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of monitoring technology, and in particular to a method and apparatus for water patrol based on multiple sensors. Background Technology

[0002] With the development of surveillance technology, the accuracy requirements for identifying targets to be monitored are becoming increasingly higher.

[0003] During the inspection of water conservancy projects, the main task is to establish a standard database of potential hazards to be inspected based on the inspection checklist, and to select points to deploy cameras and use voice broadcasts for grid management. The parameters of each camera and four feature points within the monitoring range are bound together to perform image geolocation and video recognition positioning.

[0004] However, current monitoring systems have significant blind spots, leading to a high false alarm rate. Each false alarm can trigger unnecessary emergency response team deployments, equipment damage, and subsequent handling costs. This forces security personnel to continuously deal with false alarms, resulting in a huge waste of manpower and attention, eroding their trust in the monitoring system, and causing them to hesitate even in the face of real alarms, potentially leading to serious accidents. Therefore, there is an urgent need to improve the accuracy of waterway monitoring systems in identifying abnormal events in waterways.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] To overcome the problems existing in related technologies, this disclosure provides a multi-sensor-based method and apparatus for water area patrol, which can improve the accuracy of water area monitoring systems in identifying targets to be monitored.

[0007] According to a first aspect of the present disclosure, a multi-sensor-based waterway patrol method is provided. The multi-sensor includes a visible light sensor, an infrared sensor, and an active sensor. The method includes: acquiring visible light data collected by the visible light sensor, infrared data collected by the infrared sensor, and echo data collected by the active sensor in real time in the patrol area; determining a label for the real-time environmental state of the patrol area based on the visible light data and echo data, wherein the environmental state includes light pattern and weather pattern; dynamically assigning weights to the visible light data, infrared data, and echo data in a multimodal hazard identification network based on the identified real-time environmental state labels; performing weighted fusion analysis on the multi-source data collected by the multi-sensor based on the weights assigned to each type of data through the multimodal hazard identification network to identify whether a preset abnormal event exists in the patrol area; if a preset abnormal event is identified, triggering a patrol alert of the corresponding level according to the type and location of the preset abnormal event, and marking the location of the abnormal event on an electronic map of the patrol area.

[0008] According to a second aspect of the present disclosure, a multi-sensor-based water patrol device is provided. The multi-sensor includes a visible light sensor, an infrared sensor, and an active sensor. The water patrol device includes a data acquisition module, an environmental perception module, a weight allocation module, an intelligent identification module, and a patrol prompting module. The data acquisition module is used to acquire in real-time visible light data collected by the visible light sensor, infrared data collected by the infrared sensor, and echo data collected by the active sensor in the patrol area. The environmental perception module is used to determine a label representing the real-time environmental state of the patrol area based on the visible light data and the echo data. The environmental state includes... The system includes light and weather modes; a weight allocation module, which dynamically assigns weights to visible light, infrared, and echo data in the multimodal hazard identification network based on the identified real-time environmental status labels; an intelligent identification module, which performs weighted fusion analysis on multi-source data collected by multiple sensors based on the weights assigned to each type of data through the multimodal hazard identification network to identify whether preset abnormal events exist in the patrol area; and a patrol alert module, which, if a preset abnormal event is identified, triggers a corresponding level of patrol alert based on the type and location of the preset abnormal event, and marks the location of the abnormal event on the electronic map of the patrol area.

[0009] According to a third aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the multi-sensor-based water patrol method as described in the first aspect.

[0010] According to a fourth aspect of the present disclosure, a computer device apparatus is provided, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the computer-readable instructions, when executed by the processor, implement the multi-sensor-based water patrol method as described in the first aspect.

[0011] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: In this embodiment, the intelligent patrol system first uses multi-source sensors for monitoring. During monitoring, it first assesses the environmental state based on visible light and infrared data in real time. Based on real-time environmental state labels, it assigns fusion weights to data from different sources. Then, a multimodal hazard identification network performs weighted fusion analysis on the multi-source data collected by multiple sensors based on the fusion weights assigned to each type of data to identify whether preset abnormal events exist in the patrol area. Because the intelligent patrol system comprehensively applies data collected by multiple sensors, especially by adjusting the identification weights of each type of sensor according to the environmental state before performing the identification task, favoring data that accurately expresses characteristics, the multimodal hazard identification network combines visible light, infrared, and echo data for fusion judgment during target identification. This allows the intelligent patrol system to accurately identify abnormal events even at night and in extreme weather conditions. Firstly, the introduction of infrared and echo data enables 24 / 7, all-weather, and comprehensive precise monitoring, greatly improving patrol efficiency. Secondly, it significantly enhances the accuracy of identifying and locating potential hazards in waterways under poor lighting conditions, allowing for timely detection and alerts to abnormal events within the patrol area, and precise location of these events. This facilitates timely measures by security personnel to effectively prevent safety incidents, improving the reliability of the intelligent patrol system and increasing security personnel's trust in it. Thirdly, it reduces the high false alarm rate caused by technical defects, which leads to ineffective deployments and unnecessary emergency team deployments, equipment wear and tear, and subsequent handling costs associated with security personnel continuously dealing with false alarms.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this disclosure, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0014] Figure 1 This is a schematic diagram of an intelligent patrol system architecture provided in an embodiment of the present disclosure.

[0015] Figure 2 This is a flowchart illustrating a multi-sensor-based water patrol method provided in an embodiment of the present disclosure.

[0016] Figure 3 This is a schematic plan view of a dam patrol route provided in an embodiment of this disclosure.

[0017] Figure 4 This is a schematic diagram of a multimodal sensor provided in an embodiment of the present disclosure.

[0018] Figure 5 This is a schematic diagram of a multimodal hazard identification network structure provided in an embodiment of this disclosure.

[0019] Figure 6 This is a hardware structure diagram of a computer device used in the multi-sensor-based water patrol method according to an embodiment of this disclosure.

[0020] Figure 7 This is a schematic diagram of a multi-sensor-based water patrol device provided in an embodiment of this disclosure. Detailed Implementation

[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0022] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0023] It should be understood that although this disclosure may use the terms first, second, third, etc., to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0024] The embodiments of this disclosure will now be described in detail.

[0025] Figure 1 This is a schematic diagram of an intelligent patrol system architecture provided in an embodiment of this disclosure. Figure 1 As shown, the intelligent patrol system 100 may include a heterogeneous multi-sensor monitoring device 101, a data transmission network 102, and a monitoring center 103. The multi-sensor monitoring device 101 includes a visible light sensor 1011, an infrared sensor 1012, and an active sensor 1013. The visible light sensor 1011 collects visible light data of the patrol area, the infrared sensor 1012 collects infrared data of the patrol area, and the active sensor 1013 collects echo data of the patrol area. The monitoring center 103 includes a data analysis platform 1031, a patrol prompting unit 1032, an electronic map 1033, and a data storage unit 1034. The data analysis platform 1031 is a deep learning analysis platform based on a multimodal hazard identification network built on a neural network. The multi-sensor monitoring device 101 transmits the monitored data to the monitoring center 103 in real time through the data transmission network 102. The multimodal hazard identification network of the data analysis platform 1031 performs fusion analysis on the received multi-source data to identify abnormal events. When the multimodal hazard network identifies an abnormal event, the patrol notification unit 1032 is triggered to issue a tiered notification, and the electronic map 1033 is triggered to display a map of the patrol area where the abnormal event occurred, with the location of the abnormal event marked on the map in real time. Optionally, the patrol notification unit 1032 can also notify relevant personnel to conduct tiered patrols immediately via sound and light, SMS, or short message applications. The data storage unit 1034 stores the multi-source data collected by the multi-sensor monitoring device 101 and records the identification results.

[0026] For example, the aforementioned visible light sensor can be a high-definition camera to collect images, videos, and other data of the patrol area; the infrared sensor can be a thermal imaging camera to detect temperature information in the patrol area, which can compensate for the shortcomings of the visible light sensor at night or in bad weather conditions; and the active sensor can be a millimeter-wave radar to detect the distance, speed, and orientation of moving targets in the patrol area and to accurately locate the targets.

[0027] like Figure 2 As shown, Figure 2 The flowchart of a multi-sensor-based water patrol method provided in this disclosure includes the following steps S201 to S205.

[0028] S201. Real-time acquisition of visible light data collected by visible light sensors, infrared data collected by infrared sensors, and echo data collected by active sensors in the patrol area.

[0029] Specifically, the above three types of sensors are simultaneously deployed at various locations along the patrol area of ​​the water area to be monitored, so that the intelligent patrol system can identify abnormal water events in real time based on the multi-source data collected from various locations throughout the area.

[0030] Figure 3 This is a schematic diagram of a dam patrol route provided in an embodiment of the present disclosure, such as... Figure 3 As shown, the area above and below the dam is a water surface area. There are patrol lines on both sides of the water surface area, and multi-modal sensors are installed on the patrol lines. The flood discharge outlet is located below the dam, and the hydrological equipment is located on land. Figure 4 This is a schematic diagram of a multimodal sensor provided in an embodiment of the present disclosure, as shown below. Figure 4 As shown, each set of multimodal sensors includes a high-definition visible light camera, an infrared camera, and a millimeter-wave radar, all mounted on a pillar at the monitoring location. Solar photovoltaic panels are installed on the pillar to power each set of multimodal sensors.

[0031] It should be noted that before collecting data for identifying anomalous events in the water area, the intelligent patrol system performs data preprocessing. This preprocessing includes: establishing a unified spatiotemporal reference and generating a water surface mask. This facilitates accurate target location and tracking of the location of anomalous events in the water area.

[0032] Unified spatiotemporal benchmark.

[0033] Taking a visible light sensor as a visible light camera, an infrared sensor as an infrared camera, and an active sensor as a radar as an example, this study unifies the spatiotemporal reference. Using the Zhang Zhengyou calibration method, checkerboard images of visible light and infrared data are acquired from multiple angles. Based on the acquired checkerboard images, the intrinsic parameter matrices and distortion coefficients of the visible light and infrared cameras in the camera coordinate system are solved separately. Furthermore, using a joint calibration target plate integrating a checkerboard and corner reflectors, and based on the obtained camera intrinsic parameter matrices and distortion coefficients, a 3D-3D (3D to 3D Registration) registration method using Singular Value Decomposition (SVD) is employed to solve for the optimal rotation and translation matrix from the radar coordinate system to the camera coordinate system, which is then used as the transformation matrix. Based on the transformation matrix, the echo data coordinates are transformed to the image pixel coordinate system through perspective projection, and a boundary check algorithm is used to filter out out-of-bounds points, thus completing the unification of the spatial reference. Finally, a timestamp-based interpolation method was used to establish a unified time reference for all data, ultimately achieving spatiotemporal synchronization of visible light data, infrared data, and echo data.

[0034] Generate a water surface mask.

[0035] Specifically, an initial water surface mask is first generated from the visible light data image using HSV (Hue, Saturation, Value) color space thresholding. A motion compensation algorithm is then used to eliminate spurious motion interference caused by water surface ripples in the initial mask. For the echo data, feature extraction and rasterization techniques are applied to generate radar channel data for radar feature channels and dense image channels. Based on the optical mask and radar channel data, a final water surface mask is generated, providing a precise perceptual basis for subsequent analysis and monitoring of whether abnormal events occur on the water surface.

[0036] S202. Based on visible light data and echo data, determine the labels of the real-time environmental status of the patrol area.

[0037] The environmental conditions include: light mode and weather mode. Light mode includes: daytime mode and nighttime mode. Weather mode includes: sunny mode, cloudy mode, foggy mode, and rain / snow mode.

[0038] S203. Based on the identified real-time environmental status labels, dynamically allocate the weights of visible light data, infrared data, and echo data in the multimodal hazard identification network.

[0039] Understandably, during clear daytime conditions, visible light images provide rich texture and detail features, resulting in high target recognition accuracy. However, in dense fog or at night, the signal-to-noise ratio of visible light images acquired by visible light sensors drops sharply, or even becomes completely ineffective. If target recognition is primarily based on visible light images, the recognition task will be impossible. Active sensors, on the other hand, are less affected by lighting conditions and common weather conditions, and can stably provide distance and velocity information for the target to be identified. However, in clear weather, the recognition accuracy and resolution of active sensors are lower than those of visible light sensors.

[0040] Therefore, the intelligent patrol system dynamically adjusts the weights of abnormal event identification based on the real-time environmental status indicated by different environmental status labels. When the intelligent patrol system is identifying abnormal events, it can automatically allocate resources to the feature data of the most advantageous sensors under any light and weather conditions. This makes the performance of the intelligent patrol system unaffected by light and weather conditions, thereby significantly improving the robustness, accuracy, and 24 / 7 operation capability of the intelligent patrol system in reservoir patrol target identification tasks.

[0041] For example, poor light and low visibility at night and during extreme weather provide cover for activities such as illegal fishing, sabotage, and trespassing. Since visible light sensors have poor accuracy, a fusion weighting approach can be used to identify anomalies, with lower weights for visible light data, higher weights for infrared data, and higher weights for echo data.

[0042] S204. Based on the weights assigned to each type of data, the multi-source data collected by multiple sensors is weighted and fused through a multi-modal hazard identification network to identify whether there are preset abnormal events in the patrol area.

[0043] For example, abnormal behaviors in water areas may include: intrusion of living beings, fishing, foreign objects on the water surface (floating garbage, illegal facilities), falling into the water, swimming, fire, etc. within the reservoir area.

[0044] S205. If a preset abnormal event is detected, a corresponding level of patrol prompt will be triggered according to the type and location of the preset abnormal event, and the location of the abnormal event will be marked on the electronic map of the patrol area.

[0045] It is understandable that the level of patrol prompts varies depending on the type and location of the preset abnormal event.

[0046] Optionally, the intelligent patrol system stores the data collected from multiple sensors, enabling post-event traceability and analysis. By combining the results of manual patrols with feedback on the accuracy of identification, the monitoring performance of the intelligent patrol system can be optimized.

[0047] In a multi-sensor-based waterway patrol method provided in this embodiment, the intelligent patrol system first uses multi-source sensors for monitoring. During monitoring, the system first assesses the environmental state based on visible light and infrared data in real time. Based on real-time environmental state labels, it assigns fusion weights to data from different sources. Then, a multi-modal hazard identification network performs weighted fusion analysis on the multi-source data collected by the multiple sensors based on the assigned fusion weights for each type of data to identify whether preset abnormal events exist in the patrol area. Because the intelligent patrol system comprehensively applies data collected by multiple sensors, especially by adjusting the identification weights of each type of sensor according to the environmental state before performing the identification task, favoring data that accurately expresses characteristics, the multi-modal hazard identification network combines visible light, infrared, and echo data for fusion judgment during target identification. This allows the intelligent patrol system to accurately identify abnormal events even at night and in extreme weather conditions. Firstly, the introduction of infrared and echo data enables 24 / 7, all-weather, and comprehensive precise monitoring, greatly improving patrol efficiency. Secondly, it significantly enhances the accuracy of identifying and locating potential hazards in waterways under poor lighting conditions, allowing for timely detection and alerts to abnormal events within the patrol area, and precise location of these events. This facilitates timely measures by security personnel to effectively prevent safety incidents, improving the reliability of the intelligent patrol system and increasing security personnel's trust in it. Thirdly, it reduces the high false alarm rate caused by technical defects, which leads to ineffective deployments and unnecessary emergency team deployments, equipment wear and tear, and subsequent handling costs associated with security personnel continuously dealing with false alarms.

[0048] Optionally, in the multi-sensor-based water patrol method provided in the embodiments of this disclosure, the above-mentioned S202 can be specifically executed through the following S202a and S202b.

[0049] S202a: Calculate the global average brightness and image contrast of the image based on visible light data, and calculate the real-time effective detection distance of the active sensor and the point cloud density of the preset detection position based on echo data.

[0050] The preset detection position is a location at a preset percentage of the real-time effective detection distance from the active sensor. The preset detection position indicates the far-end region of the active sensor.

[0051] Specifically, the visible light data is converted into a grayscale matrix, and the global average brightness corresponding to the visible light data is calculated based on the grayscale matrix and formula (1). The image contrast corresponding to the visible light data is calculated based on formula (2).

[0052] ;Formula (1) ;Formula (2) in, This represents the global average brightness. Indicates image contrast. This indicates the row number of the grayscale matrix. This indicates the number of columns in the grayscale matrix. Indicates the first Line number The grayscale value of the column.

[0053] S202b: Determine the lighting pattern based on the global average brightness, determine the weather pattern based on image contrast, real-time effective detection distance, and point cloud density at preset detection locations, and obtain a label for the real-time environmental status of the patrol area.

[0054] Specifically, if the global average brightness is greater than the preset average brightness, the lighting mode is determined to be daytime mode; if the global average brightness is less than or equal to the preset average brightness, the lighting mode is determined to be nighttime mode. After determining the lighting mode based on the global average brightness of visible light data, if the lighting mode is daytime mode, the weather mode is determined based on the contrast of visible light data; if the lighting mode is nighttime mode, the weather mode is determined based on the real-time effective detection distance of echo data and the point cloud density at the preset detection location.

[0055] Based on this scheme, the environmental state can be accurately determined based on visible light data and echo data. This allows for the adjustment of the identification weight values ​​of data collected by different sensors based on the real-time environmental state, thereby providing data bias guidance for subsequent abnormal event identification, so as to focus on monitoring data that is more effectively identified.

[0056] Optionally, in the multi-sensor-based water patrol method provided in this embodiment, in S202a above, the light pattern can be divided into two main categories: night mode and day mode based on the global brightness. Then, the weather mode is specifically determined based on the light pattern.

[0057] Scenario 1: Night mode - global average brightness is less than the brightness threshold; 1-1. If the real-time effective detection distance is less than the distance threshold and the point cloud density at the preset detection location is higher than the density threshold, the environmental status label is "Night - Rain / Snow". 1-2. If the real-time effective detection distance is less than the distance threshold and the point cloud density at the preset detection location is not higher than the density threshold, the environmental status label is "Night - Foggy Day". 1-3. If the real-time effective detection distance is greater than or equal to the distance threshold, the environmental status label is "Night - Clear".

[0058] Scenario 2: Daytime mode - Global average brightness is greater than or equal to the brightness threshold; 2-1. If the image contrast is greater than the first contrast threshold and the real-time effective detection distance is greater than or equal to the distance threshold, then the environmental status label is "Daytime - Sunny". 2-2. If the image contrast is less than the second contrast threshold, and the real-time effective detection distance is less than the distance threshold and the point cloud density at the preset detection position is higher than the density threshold, then the environmental status label is "daytime - rain / snow". The first contrast threshold is greater than the second contrast threshold.

[0059] 2-3. If the image contrast is less than the second contrast threshold, the real-time effective detection distance is less than the distance threshold, and the point cloud density at the preset detection position is not higher than the density threshold, then the environmental status label is "daytime - foggy day". 2-4. If none of the above conditions are met, the environmental status label will be "Daytime - Cloudy".

[0060] Based on this scheme, the light mode can be determined as either daytime or nighttime based on the global average brightness of visible light data. If the light mode is nighttime, the weather mode can be determined based on the effective detection distance of the echo data and the point cloud density at the preset detection location. If the light mode is daytime, the weather mode can be determined based on the effective detection distance of the image contrast echo data and the point cloud density at the preset detection location, thus accurately obtaining the real-time environmental status.

[0061] Optionally, in a multi-sensor-based water patrol method provided in this embodiment, S204 may specifically include S204a to S204e.

[0062] S204a. Extract the detailed texture and color features from the visible light data to obtain the first feature map.

[0063] In daytime mode, the first feature map has rich detail textures and color channels, making it easy to express the target's morphological features. However, in nighttime mode, the first feature map struggles to express the target's morphological features.

[0064] It is understandable that the detailed texture and color features extracted from visible light data are used to determine the target, location, and specific behavior of the abnormal event.

[0065] S204b: Extract the thermal radiation profile features from the infrared data to obtain the second feature map.

[0066] Among them, the second feature map can capture the thermal radiation contour features of living organisms or fire sources that are difficult to obtain with visible light.

[0067] Specifically, the extracted thermal radiation profile features can be used to determine whether an organism is a living being and what type of organism it is. Especially in nighttime mode, where the reliability of visible light data is low, infrared data can be used to assist in determining the shape of living organisms, and the motion characteristics of behavior can be determined based on continuous thermal radiation profile features.

[0068] It should be noted that the thermal radiation profiles of different species of life forms are different.

[0069] S204c: Analyze the echo data to generate a point cloud containing target distance, velocity, and orientation information, and extract spatial structure and motion features from the point cloud to obtain a third feature map.

[0070] The third feature map contains enhanced spatial and motion information and can also be used for fuzzy localization.

[0071] S204d. Based on the weights assigned to the environmental state labels, perform static weighted fusion of the first feature map, the second feature map, and the third feature map.

[0072] Specifically, the weights of the feature maps corresponding to each of the multi-source heterogeneous sensors can be assigned in real time based on environmental labels and a preset weight correspondence table. Table 1 is a fusion weight relationship table of real-time environmental and various sensor data provided in an embodiment of this disclosure. Indicates the real-time effective detection distance. Indicates the distance threshold. =150m, The point cloud density represents the first detection range. This represents the density threshold of the point cloud. This represents the global average brightness, with a global brightness threshold of 30. This represents the image contrast, with a first contrast threshold of 40 and a second contrast threshold of 80.

[0073] Table 1. Fusion Weighting Relationship between Environment and Sensors As can be seen from the table, if it is determined to be night mode, visible light data is not used for identification.

[0074] S204e, Based on the fused features, output the identification and location results of abnormal events.

[0075] For example, static weighted fusion is performed based on formula (3) and the three types of feature weights corresponding to the real-time environment label.

[0076] ;Formula (3) in, This indicates the result after weighted fusion. This represents the first feature map extracted from visible light data. This represents the second feature map extracted from infrared data. This represents the third feature map extracted from the echo data. This indicates the weight ratio of the first feature map. This indicates the weight ratio of the second feature map. This represents the weight ratio of the third feature map.

[0077] Based on the fused results, the multimodal hazard identification network makes comprehensive judgments, enabling accurate identification of abnormal events in various lighting and weather scenarios. In target identification tasks, traditional single sensors cannot cope with diverse environmental lighting and weather conditions. This solution employs multi-source heterogeneous sensors working collaboratively, reducing the impact of ambient lighting and weather conditions on the data quality of different sensor modalities. This reduces fluctuations in the identification model's performance and maintains stable model performance under various lighting modes and weather conditions, thereby enhancing all-weather monitoring capabilities. For example, it can accurately identify illegal fishing, facility damage, and unauthorized intrusion not only during clear days but also at night or in extreme weather conditions, enabling timely warnings and effective deterrence of potential threats, effectively ensuring the safety of reservoir infrastructure and its surrounding environment. In low-visibility environments, in the event of accidents such as people falling into the water, it can shorten the golden time from the occurrence of the accident to discovery and rescue, minimizing the occurrence of tragic casualties.

[0078] Optionally, the multimodal hazard identification network includes: a visible light feature extraction network, an infrared feature extraction network, an echo feature extraction network, a feature fusion unit, and a detection head; the visible light feature extraction network is constructed from a target detection backbone network structure and is used to obtain a first feature map; the infrared feature extraction network is constructed from a lightweight convolutional network structure and is used to obtain a second feature map; the echo feature extraction network is constructed from sparse convolutional layers and is used to obtain a third feature map; the feature fusion unit is used to perform static weighted fusion of the first, second, and third feature maps according to the weights assigned by the environmental state labels; the detection head is used to output the identification and location results of preset abnormal events based on the fused features.

[0079] For example, Figure 5 This is a schematic diagram of a multimodal hazard identification network structure provided in an embodiment of this disclosure. Figure 5 As shown, the visible light feature extraction network of the multimodal hazard identification network based on the YOLO (You Only Look Once)-Multi network (YOLO's multi-task model) uses the YOLO backbone, which is mainly suitable for target detection in well-lit environments, and can obtain visible light features at the first, second, and third scales. The infrared feature extraction network uses the MobileNetV3 backbone, which is mainly suitable for capturing the thermal signals of targets in low-light environments (overcast days, cloudy days, or nighttime), and can obtain infrared features at the first, second, and third scales. This can enhance the robustness of the multimodal monitoring system under complex lighting conditions. The echo feature extraction network uses coefficient convolutional layers to process point cloud data of remote sensing echoes. Combined with max pooling operations and scale adjustment and feature compression, it extracts the spatial structure and distance information of the echo point cloud, and can obtain echo features at the first, second, and third scales. It can compensate for the limitations of visible light sensors in harsh weather conditions (rain, fog, smoke and dust) and improve the reliability of multimodal monitoring systems in identifying abnormal events in various environments.

[0080] Static weighted fusion is performed on the visible light, infrared, and echo features at the first scale to obtain the fused first-scale features. Static weighted fusion is then performed on the visible light, infrared, and echo features at the second scale to obtain the fused second-scale features. Finally, static weighted fusion is performed on the visible light, infrared, and echo features at the third scale to obtain the fused third-scale echo features. These three fused scale features are then sequentially input into the YOLOneck (neck module) and the YOLO head (detection head) to output the final recognition result.

[0081] It should be noted that, based on the aforementioned network structure and the collected sample data, anomaly event monitoring training can be performed to obtain a trained network for identifying aquatic anomalies. This trained network can then be used for aquatic monitoring. Specifically, the collection of sample data from three types of sensors and the aforementioned monitoring data for various aquatic anomalies covers a wide time span, including multi-source data across all seasons, day and night, various meteorological conditions, different distances, attitudes, and states. This ensures sufficient sample diversity, guaranteeing that the trained model can cope with the complex and variable meteorological conditions in real reservoir patrol environments. After collecting the raw data, data cleaning, spatial registration, and modality-specific preprocessing are performed. A human-machine collaborative annotation strategy is adopted to label each target with its category and detection box, assigning a globally unique ID (Identity) and state attributes, establishing cross-modal instance-level associations, and providing accurate ground truth for model fusion learning. Through systematic collection and annotation of three types of target scenarios, a multimodal dataset containing at least three thousand high-quality samples is constructed. This multimodal dataset fully covers target morphologies in different seasons, time periods, meteorological conditions, and complex backgrounds, providing rich and balanced learning materials for model training. The scale and quality of this data effectively ensured the convergence stability and generalization ability of the network model during training, significantly improving its recognition accuracy and robustness for multiple target classes in real-world scenarios. The processed dataset was divided into training, validation, and test sets in a 7:2:1 ratio, ensuring a balanced distribution of target categories and scene conditions. Based on this, targeted data augmentation was implemented, such as simulating optical occlusion, severe weather imaging, and multimodal consistent geometric transformations, effectively enhancing the model's adaptability to small targets and complex environments.

[0082] Optionally, in a multi-sensor-based waterway patrol method provided in this embodiment, the patrol area is divided into a primary monitoring zone and a secondary monitoring zone according to the level of safety accident risk. Abnormal events are classified into three categories, and patrol alerts are classified into three categories.

[0083] Among them, the safety accident risk in the first-level monitoring area is higher than that in the second-level monitoring area; the first preset abnormal event is an event that is threatening life and health safety, the second preset abnormal event is an event that may threaten life and health safety, and the third preset abnormal event is an event that may threaten the safety of water equipment; the first-level patrol prompts and instructs to carry rescue equipment to the scene for rescue immediately, the second-level patrol prompts and instructs to go to the scene for handling immediately, and the third-level patrol prompts to go to the scene for verification within the preset time.

[0084] The highest risk of safety incidents occurs in the Level 1 monitoring zone, and immediate intervention is required if any abnormal events are detected.

[0085] For example, the primary monitoring zone includes (core): water area, hydrophilic hazard zone, and hydrodynamic high-risk zone. The hydrophilic hazard zone is the area extending from the water boundary into the land at a first distance (e.g., 5 meters). The hydrodynamic high-risk zone includes the land area within a first radius (e.g., 20 meters) centered on critical facilities; critical facilities include: dam spillways and pump inlets. The first preset distance is less than the second preset distance.

[0086] There is a potential safety risk in the secondary monitoring zone; after an abnormal event is detected in the secondary monitoring zone, intervention will be carried out within a preset time (e.g., within 30 minutes).

[0087] For example, the secondary monitoring zone (buffer) includes: a second distance (5 meters) extending outward from the primary monitoring zone, an activity area on the passageway above the dam, and an area within a second radius (5 meters) centered on the basic equipment; the basic equipment includes: hydrological monitoring equipment, power supply boxes, etc. The third distance and the preset radius can be the same or different.

[0088] Based on this scheme, water areas are divided into different monitoring levels. For areas with different monitoring levels, different patrol reminders and patrol requirements can be issued for abnormal events in the area. This can provide precise information guidance for responders and shorten their preparation time for rescue or expulsion.

[0089] Optionally, in the multi-sensor-based water patrol method provided in this disclosure, different patrol alert levels correspond to different behaviors in different monitoring areas. Specifically, it can include three levels of patrol alerts, which are, in descending order of processing level, level one patrol alert, level two patrol alert, and level three patrol alert.

[0090] Level 1 Inspection Alert: Indicates that a serious incident threatening life, health, or safety has occurred or is about to occur in the Level 1 monitoring area; requires responders to immediately go to the scene with rescue equipment to provide rescue or emergency intervention.

[0091] For example, the corresponding first preset abnormal events include: people swimming in the waters of the monitored area, people or animals falling into the waters of the monitored area, people or animals entering the core high-risk facility area (i.e., the hydrophilic danger zone and the hydrodynamic high-risk zone), and area fires.

[0092] Level 2 Inspection Alert: Indicates that an event with a risk level lower than that of a Level 1 Inspection Alert exists in the Level 1 monitoring area; requires responders to immediately proceed to the scene for handling.

[0093] For example, the corresponding second preset abnormal events include: people fishing on the shore, large foreign objects on the water surface, etc.

[0094] Level 3 patrol alert: Indicates an event requiring attention in the Level 2 monitoring area; requests response personnel to go to the site for verification and routine handling.

[0095] For example, the corresponding third preset abnormal events include: area intrusion, equipment intrusion risk, or equipment malfunction. Among them, equipment malfunction refers to performance failure caused by equipment aging, software errors, or non-human external environment; equipment intrusion risk refers to human damage, obstruction, or illegal operation of the equipment.

[0096] Furthermore, in the water patrol method based on multiple sensors provided in this embodiment, S205 may specifically include S205a, S205b or S205c.

[0097] S205a. If a first preset abnormal event is detected in the first-level monitoring area, a first-level patrol prompt will be triggered.

[0098] S205b: If a second preset abnormal event is detected in the primary monitoring area, a secondary inspection prompt will be triggered.

[0099] S205c: If a third preset abnormal event is detected in the secondary monitoring area, a third-level inspection prompt will be triggered.

[0100] Based on this solution, the intelligent patrol system can trigger different levels of patrol prompts according to the location and specific behavior of abnormal events. This can provide precise patrol guidance to patrol personnel, avoid the waste of time or rescue resources due to human error, and thus improve the reliability of the intelligent patrol system.

[0101] To facilitate understanding of this solution, the following scenario examples will all be illustrated using daytime as an example.

[0102] Scenario 1: Swimming or falling into the water.

[0103] The visible light feature extraction network identifies targets within the water surface mask in real time based on visible light data, and these targets possess preset morphological contour features. These preset morphological contour features can indicate at least part of the morphological contour of targets such as humans and animals. The infrared feature extraction network identifies, based on infrared data, the emergence of typical human or animal temperature contour hotspots within the water surface mask that strongly contrast with the water temperature, confirming the presence of a living organism. The echo feature extraction network detects the emergence of foreign object point cloud features within the water surface mask based on echo data. Furthermore, the intelligent patrol system, based on the weighted ratio of light and weather conditions, comprehensively determines the abnormal event as swimming or falling into the water, triggering a first-level patrol alert. The system then uses sensor locations to pinpoint the alarm area and uses sensors along the route for real-time tracking and positioning.

[0104] Scenario 2: Pre-set basic equipment is compromised.

[0105] The visible light feature extraction network identifies pre-defined morphological contour features in high-risk hydrodynamic zones based on visible light data; the infrared feature extraction network identifies newly formed typical human body temperature contour hotspots in high-risk hydrodynamic zones in real time based on infrared data, confirming the presence of living organisms; the echo feature extraction network detects newly formed foreign object point clouds in high-risk hydrodynamic zones based on echo data. The intelligent patrol system triggers a level-one patrol alert, locates the intrusion site of pre-defined infrastructure through sensor positioning, and tracks and locates the intruder's position in real time using sensors along the route.

[0106] Scenario 3: Area fire.

[0107] The visible light feature extraction network identifies a significant fire source within the monitored area based on visible light data, and the infrared feature extraction network identifies the outline and hot spot of the fire source based on infrared data. The monitoring system then determines that the area is on fire and detects newly formed foreign object point clouds based on echo data. The intelligent patrol system triggers a first-level patrol alert and locks the fire area through sensor location.

[0108] Scene 4: People fishing.

[0109] The visible light feature extraction network identifies preset morphological contour features in water areas or hydrophilic hazard zones in real time based on visible light data, and these preset morphological contour features match the contour features of rod fishing in the sample library; the infrared feature extraction network identifies newly formed and typical human body temperature contour hot spots in real time based on infrared data, confirming the presence of living organisms; the echo feature extraction network detects newly formed foreign object point clouds based on echo data; the intelligent patrol system determines that there are people fishing in the area, triggers a level two alarm, and locates the area where the fishing behavior is located through sensor positions.

[0110] Scene 5: Large foreign object on the water surface.

[0111] The visible light feature extraction network identifies targets in the water area with preset shape contour features based on visible light data in real time, and the area formed by the contour is larger than the preset area; the infrared feature extraction network does not identify hot spots where the target and the water temperature form a strong contrast based on infrared data; the echo feature extraction network detects point clouds of newly formed foreign objects on the water surface based on echo data. The intelligent patrol system determines that a large foreign object has appeared on the water surface, triggers a secondary patrol prompt, and locks the area where the foreign object is located through sensor positions, and relies on sensors along the line to track and locate it in real time.

[0112] Scenario 6: Human intrusion.

[0113] The visible light feature extraction network identifies the morphological contour features of a human body within the target based on visible light data, and the target is moving in real time. The infrared feature extraction network identifies newly formed and typical human body temperature contour hotspots based on infrared data, confirming the presence of a living organism. The echo feature extraction network identifies newly formed foreign object point clouds in the land area based on echo data, and the foreign object point clouds are in a moving state. The intelligent patrol system determines the abnormal event as human intrusion, triggers a three-level patrol alert, locates the alarm area based on sensor locations, and performs real-time tracking and positioning based on sensors along the route.

[0114] Scenario 7: Device intrusion.

[0115] The visible light feature extraction network identifies targets with preset morphological or contour features within 5 meters of infrastructure such as hydrological monitoring equipment and power supply boxes during non-predetermined equipment inspection periods based on visible light data. The infrared feature extraction network identifies newly formed and typical human body temperature contour hotspot features based on infrared data to confirm the presence of living organisms. The echo feature extraction network detects newly formed foreign object point clouds in land areas based on echo data, and these foreign object point clouds are in a moving state. If the intelligent patrol system determines that the equipment has been intruded, it will trigger a three-level patrol alert through sensor location, notifying staff to conduct patrols and remove the intruders.

[0116] Scenario 8: Monitoring equipment failure.

[0117] If the visible light feature extraction network detects persistent abnormal occlusion (such as leaves or soil) in the visible light data, it determines that the visible light sensor signal is lost and therefore the visible light sensor is faulty. If the infrared feature extraction network detects image anomalies in the infrared data (such as overexposure or complete darkness), it determines that the infrared sensor is faulty. If the echo feature extraction network detects interrupted echo data, it determines that the remote sensing equipment has experienced a communication interruption or malfunction. If any of the above conditions are met, the intelligent inspection system triggers a three-level inspection alert based on the sensor location, notifying personnel to conduct maintenance and repairs.

[0118] It should be noted that, to ensure that every triggered patrol alert is effectively verified and terminated, and to complete the closed loop of the entire process from automatic triggering of patrol alerts by the intelligent patrol system to manual on-site handling, the intelligent patrol system also provides a multi-layered and highly reliable patrol cancellation mechanism. This mechanism is activated after the patrol personnel arrive on site, and through three steps—identity authentication, multimodal on-site verification, and handling feedback—the patrol alert is finally cancelled after confirmation by the system.

[0119] 1) Mechanism trigger: Mobile terminal notifications and navigation.

[0120] Once the patrol alert from the monitoring center is triggered, in addition to audio and visual alerts, the intelligent patrol system will push detailed patrol alert information to the mobile terminals of the patrol personnel responsible for the area via a dedicated application or instant messaging mini-program.

[0121] The patrol alert information includes: patrol alert type, time of occurrence, precise location (which can be combined with electronic map navigation displayed on the mobile terminal), sensor snapshots / video clips, and the basis for the intelligent patrol system's judgment (e.g., "Infrared detected a human heat source, radar detected a moving target").

[0122] The mobile terminal interface provides a "one-click navigation" button, which can directly call the map application to guide patrol personnel to the scene quickly.

[0123] 2) On-site certification and verification mechanism.

[0124] After arriving at the location specified in the patrol prompt, patrol personnel need to prove to the intelligent patrol system that they have been on-site and completed the verification in the following ways.

[0125] Dual authentication of identity and location.

[0126] Location check-in: Patrol personnel proactively report their GPS (Global Positioning System) location information via mobile terminals. The intelligent patrol system compares this location with the location of any abnormal event, and only allows further operation if the location is within a set threshold range (e.g., within 20 meters).

[0127] Identity verification: Before operation, the inspectors must verify their identity through a mobile terminal, including by password login, fingerprint recognition or facial recognition, to ensure the operator's authority and traceability.

[0128] 3) Inspection prompts, decision-making, and feedback.

[0129] Based on the on-site evidence uploaded by the patrol personnel, the intelligent patrol system provides a manual confirmation method for lifting the restrictions (standard procedure).

[0130] After completing the on-site inspection, the patrol personnel selected the patrol notification and handling results on their mobile terminals.

[0131] Confirm the progress of the response; if the situation has been resolved: e.g., someone was indeed found to have fallen into the water and has been rescued, or the fire has been extinguished. A brief written description of the response measures is required.

[0132] False alarms in the system require optimization: If it is confirmed that the false alarm is caused by other factors, you can manually add a false alarm reason tag to optimize the algorithm.

[0133] Once the patrol alert is officially lifted in the intelligent patrol system, a complete handling record will be generated, including "time, location, person handling the situation, on-site evidence (video / images), and handling result".

[0134] Corresponding to the embodiments of the foregoing methods, this disclosure also provides embodiments of the apparatus and the computer equipment on which it is applied.

[0135] Embodiments of the disclosed device can be applied to computer equipment, such as servers or terminal devices. The device embodiments can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, a logically defined multi-sensor-based water patrol device is formed by the processor of its multi-sensor-based water patrol method loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 6 The diagram shown is a hardware structure diagram of a computer device used in the multi-sensor-based water patrol method of this disclosure, except for... Figure 6 In addition to the processor 610, memory 630, network interface 620, and non-volatile memory 640 shown, the server or electronic device in which the multi-sensor-based water patrol method is located in the embodiment may also include other hardware depending on the actual function of the computer device, which will not be described in detail here.

[0136] like Figure 7 As shown, Figure 7 This disclosure provides a multi-sensor-based water patrol device 700, comprising: a data acquisition module 701, an environmental perception module 702, a weight allocation module 703, an intelligent identification module 704, and a patrol prompting module 705. The data acquisition module 701 is used to acquire in real-time visible light data collected by a visible light sensor, infrared data collected by an infrared sensor, and echo data collected by an active sensor in the patrol area. The environmental perception module 702 is used to determine the real-time environmental status label of the patrol area based on the visible light data and echo data, whereby the environmental status includes light pattern and weather pattern. The weight allocation module 703 is used to dynamically allocate the weights of visible light data, infrared data, and echo data in the multimodal hazard identification network based on the identified real-time environmental status labels; the intelligent identification module 704 is used to perform weighted fusion analysis on multi-source data collected by multiple sensors based on the weights allocated to each type of data through the multimodal hazard identification network, and to identify whether there are preset abnormal events in the patrol area; the patrol prompt module 705 is used to trigger the corresponding level of patrol prompt according to the type and location of the preset abnormal event if a preset abnormal event is identified, and to mark the location of the abnormal event on the electronic map of the patrol area.

[0137] Optionally, the environmental perception module is specifically used to: calculate the global average brightness and image contrast of the image based on visible light data, and calculate the real-time effective detection distance of the active sensor and the point cloud density of the preset detection position based on echo data; the preset detection position is the position of the real-time effective detection distance at a preset percentage of the distance from the active sensor; determine the light pattern based on the global average brightness, and determine the weather pattern based on the image contrast, the real-time effective detection distance, and the point cloud density of the preset detection position, to obtain a label of the real-time environmental status of the patrol area.

[0138] Optionally, if the global average brightness is less than the brightness threshold, the light mode is nighttime; if the real-time effective detection distance is less than the distance threshold and the point cloud density is higher than the density threshold, the environment status label is "nighttime - rain / snow"; if the real-time effective detection distance is less than the distance threshold and the point cloud density is not higher than the density threshold, the environment status label is "nighttime - foggy"; if the real-time effective detection distance is greater than or equal to the distance threshold, the environment status label is "nighttime - clear".

[0139] Optionally, if the global average brightness is greater than or equal to the brightness threshold, the lighting mode is daytime; if the image contrast is greater than the first contrast threshold and the real-time effective detection distance is greater than or equal to the distance threshold, the environment status label is "daytime - sunny"; if the image contrast is less than the second contrast threshold, the real-time effective detection distance is less than the distance threshold, and the point cloud density is higher than the density threshold, the environment status label is "daytime - rain / snow"; if the first contrast threshold is greater than the second contrast threshold; if the image contrast is less than the second contrast threshold, the real-time effective detection distance is less than the distance threshold, and the point cloud density is not higher than the density threshold, the environment status label is "daytime - foggy"; if none of the above conditions are met, the environment status label is "daytime - cloudy".

[0140] Optionally, the intelligent recognition module is specifically used for: extracting detailed texture and color features from visible light data to obtain a first feature map; extracting thermal radiation contour features from infrared data to obtain a second feature map; parsing echo data to generate a point cloud containing the target's distance, velocity, and orientation information, and extracting spatial structure and motion features from the point cloud to obtain a third feature map; performing static weighted fusion of the first, second, and third feature maps according to the weights assigned by the environmental state labels; and outputting the recognition and location results of abnormal events based on the fused features.

[0141] Optionally, the multimodal hazard identification network includes: a visible light feature extraction network, an infrared feature extraction network, an echo feature extraction network, a feature fusion unit, and a detection head; the visible light feature extraction network is constructed from a target detection backbone network structure and is used to obtain a first feature map; the infrared feature extraction network is constructed from a lightweight convolutional network structure and is used to obtain a second feature map; the echo feature extraction network is constructed from sparse convolutional layers and is used to obtain a third feature map; the feature fusion unit is used to perform static weighted fusion of the first, second, and third feature maps according to the weights assigned by the environmental state labels; the detection head is used to output the identification and location results of abnormal events based on the fused features.

[0142] Optionally, the patrol notification module is specifically used to: trigger a level-one patrol notification if a first preset abnormal event is detected in the level-one monitoring area; trigger a level-two patrol notification if a second preset abnormal event is detected in the level-one monitoring area; and trigger a level-three patrol notification if a third preset abnormal event is detected in the level-two monitoring area. The safety risk in the level-one monitoring area is higher than that in the level-two monitoring area. The first preset abnormal event is an event that is currently threatening life and health; the second preset abnormal event is an event that may potentially threaten life and health; and the third preset abnormal event is an event that may potentially threaten the safety of water-related equipment. The level-one patrol notification instructs the user to immediately proceed to the scene with rescue equipment for rescue; the level-two patrol notification instructs the user to immediately proceed to the scene for handling; and the level-three patrol notification instructs the user to proceed to the scene for verification within a preset time period.

[0143] The multi-sensor-based water patrol device provided in this disclosure first utilizes multi-source sensors for monitoring. During monitoring, it first assesses the environmental state in real time based on visible light and infrared data. Then, it assigns fusion weights to data from different sources based on real-time environmental state labels. A multi-modal hazard identification network performs weighted fusion analysis on the multi-source data collected by the multiple sensors based on the assigned fusion weights for each data type, identifying whether preset abnormal events exist in the patrol area. Because the intelligent patrol system comprehensively applies data collected by multiple sensors, especially by adjusting the identification weights of each sensor type according to the environmental state before performing the identification task, favoring data that accurately expresses characteristics, the multi-modal hazard identification network combines visible light, infrared, and echo data for fusion judgment during target identification. This allows the intelligent patrol system to accurately identify abnormal events even at night and in extreme weather conditions. Firstly, the introduction of infrared and echo data enables 24 / 7, all-weather, and comprehensive precise monitoring, greatly improving patrol efficiency. Secondly, it significantly enhances the accuracy of identifying and locating potential hazards in waterways under poor lighting conditions, allowing for timely detection and alerts to abnormal events within the patrol area, and precise location of these events. This facilitates timely measures by security personnel to effectively prevent safety incidents, improving the reliability of the intelligent patrol system and increasing security personnel's trust in it. Thirdly, it reduces the high false alarm rate caused by technical defects, which leads to ineffective deployments and unnecessary emergency team deployments, equipment wear and tear, and subsequent handling costs associated with security personnel continuously dealing with false alarms.

[0144] Accordingly, this disclosure also provides a multi-sensor-based waterway patrol device, which includes a processor and a memory for storing processor-executable instructions. The processor is configured to: acquire in real-time visible light data collected by a visible light sensor, infrared data collected by an infrared sensor, and echo data collected by an active sensor in the patrol area; determine a label for the real-time environmental state of the patrol area based on the visible light data and echo data, the environmental state including light pattern and weather pattern; dynamically assign weights to the visible light data, infrared data, and echo data in a multimodal hazard identification network based on the identified real-time environmental state labels; perform weighted fusion analysis on the multi-source data collected by the multi-sensor network based on the weights assigned to each type of data to identify whether a preset abnormal event exists in the patrol area; if a preset abnormal event is identified, trigger a patrol alert of the corresponding level according to the type and location of the preset abnormal event, and mark the location of the abnormal event on an electronic map of the patrol area.

[0145] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps described in the above embodiments of the multi-sensor-based water patrol method.

[0146] This disclosure also provides a computer device, which includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the computer-readable instructions, when executed by the processor, implement the various steps as described in the above embodiments of the multi-sensor-based water patrol method.

[0147] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0148] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0149] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0150] Other embodiments of this disclosure will be readily apparent to those skilled in the art upon understanding and practicing this specification. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not claimed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0151] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0152] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A multi-sensor-based method for waterway patrol, characterized in that, The multi-sensor includes: a visible light sensor, an infrared sensor, and an active sensor; the method includes: Real-time acquisition of visible light data collected by visible light sensors, infrared data collected by infrared sensors, and echo data collected by active sensors in the patrol area; Based on visible light data and echo data, the real-time environmental status label of the patrol area is determined, including the light pattern and weather pattern. Based on the identified real-time environmental status labels, the weights of visible light data, infrared data, and echo data in the multimodal hazard identification network are dynamically assigned. The multimodal hazard identification network performs weighted fusion analysis on the multi-source data collected by the multi-sensor based on the weights assigned to each type of data to identify whether there are preset abnormal events in the patrol area. If a preset abnormal event is detected, a corresponding level of patrol prompt is triggered based on the type and location of the preset abnormal event, and the location of the abnormal event is marked on the electronic map of the patrol area.

2. The method according to claim 1, characterized in that, The label for determining the real-time environmental status of the patrol area based on visible light data and echo data includes: The global average brightness and image contrast of the image are calculated based on visible light data, and the real-time effective detection distance of the active sensor and the point cloud density at a preset detection position are calculated based on echo data; the preset detection position is the position at a preset percentage of the real-time effective detection distance from the active sensor. The lighting pattern is determined based on the global average brightness, and the weather pattern is determined based on the image contrast, the real-time effective detection distance, and the point cloud density at the preset detection location, thereby obtaining a label for the real-time environmental status of the patrol area.

3. The method according to claim 2, characterized in that, When the global average brightness is less than the brightness threshold, the light mode is nighttime. If the real-time effective detection distance is less than the distance threshold and the point cloud density is higher than the density threshold, then the environmental state label is "night - rain / snow". If the real-time effective detection distance is less than the distance threshold and the point cloud density is not higher than the density threshold, then the environmental state label is "night - foggy day"; If the real-time effective detection distance is greater than or equal to the distance threshold, then the environmental status label is "Night - Clear Day".

4. The method according to claim 2, characterized in that, When the global average brightness is greater than or equal to the brightness threshold, the lighting mode is daytime. If the image contrast is greater than the first contrast threshold and the real-time effective detection distance is greater than or equal to the distance threshold, then the environmental status label is "daytime - sunny". If the image contrast is less than the second contrast threshold, the real-time effective detection distance is less than the distance threshold, and the point cloud density is higher than the density threshold, then the environmental state label is "daytime - rain / snow"; the first contrast threshold is greater than the second contrast threshold. If the image contrast is less than the second contrast threshold, and the real-time effective detection distance is less than the distance threshold and the point cloud density is not higher than the density threshold, then the environmental state label is "daytime - foggy day". If none of the above conditions are met, the environmental status label will be "Daytime - Cloudy".

5. The method according to claim 1, characterized in that, The step of using the multimodal hazard identification network to perform weighted fusion analysis on the multi-source data collected by the multi-sensor network based on the weights assigned to each type of data, and identifying whether there are preset abnormal events in the patrol area, includes: The first feature map is obtained by extracting detailed texture and color features from visible light data; The second feature map is obtained by extracting thermal radiation contour features from infrared data; The echo data is analyzed to generate a point cloud containing the target's distance, velocity, and orientation information, and the spatial structure and motion features are extracted from the point cloud to obtain a third feature map. Based on the weights assigned to the environmental state labels, the first feature map, the second feature map, and the third feature map are subjected to static weighted feature fusion. The results of abnormal event identification and localization are output based on the fused features.

6. The method according to claim 5, characterized in that, The multimodal hazard identification network includes: a visible light feature extraction network, an infrared feature extraction network, an echo feature extraction network, a feature fusion unit, and a detection head; The visible light feature extraction network is constructed from a target detection backbone network structure and is used to obtain the first feature map; The infrared feature extraction network is constructed from a lightweight convolutional network structure and is used to obtain the second feature map; The echo feature extraction network is constructed from sparse convolutional layers and is used to obtain the third feature map; The feature fusion unit is used to perform static weighted feature fusion on the first feature map, the second feature map, and the third feature map according to the weights assigned by the environmental state labels. The detection head is used to output the identification and location results of abnormal events based on the fused features.

7. The method according to claim 1, characterized in that, The step of triggering corresponding levels of patrol alerts based on the type and location of preset abnormal events includes: If a first preset abnormal event is detected within the first-level monitoring area, a first-level patrol prompt will be triggered. If a second preset abnormal event is detected within the primary monitoring area, a secondary inspection prompt will be triggered. If a third preset abnormal event is detected in the secondary monitoring area, a third-level inspection prompt will be triggered. The safety accident risk in the first-level monitoring area is higher than that in the second-level monitoring area; the first preset abnormal event is an event that is threatening life and health safety, the second preset abnormal event is an event that may threaten life and health safety, and the third preset abnormal event is an event that may threaten the safety of water equipment; the first-level patrol prompts to carry rescue equipment and go to the scene immediately for rescue, the second-level patrol prompts to go to the scene immediately for handling, and the third-level patrol prompts to go to the scene for verification within a preset time.

8. A multi-sensor-based waterway patrol device, characterized in that, The multi-sensor system includes: a visible light sensor, an infrared sensor, and an active sensor; the water patrol device includes: a data acquisition module, an environmental perception module, a weight allocation module, an intelligent identification module, and a patrol prompt module. The data acquisition module is used to acquire visible light data collected by the visible light sensor, infrared data collected by the infrared sensor, and echo data collected by the active sensor in the patrol area in real time. The environmental perception module is used to determine the label of the real-time environmental status of the patrol area based on visible light data and echo data. The environmental status includes light pattern and weather pattern. The weight allocation module is used to dynamically allocate the weights of visible light data, infrared data, and echo data in the multimodal hazard identification network based on the identified real-time environmental status labels. The intelligent identification module is used to perform weighted fusion analysis on the multi-source data collected by the multi-sensor based on the weight assigned to each type of data through the multi-modal hazard identification network, and to identify whether there are preset abnormal events in the patrol area; The patrol prompt module is used to trigger a patrol prompt of the corresponding level according to the type and location of the preset abnormal event if a preset abnormal event is detected, and to mark the location of the abnormal event on the electronic map of the patrol area.

9. An electronic device, characterized in that, include: processor; And a memory storing computer-readable instructions that, when executed by the processor, implement the steps of the multi-sensor-based water patrol method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-sensor-based water patrol method according to any one of claims 1-7.

Images