A sound and vision dual-mode global falling-into-water intelligent detection and rescue system and method

Abstract

The application discloses a sound and vision dual-mode global falling-into-water intelligent detection and rescue system and method, and relates to the technical field of water area safety monitoring and emergency rescue. The system comprises a sound and vision dual-mode sensing unit, an intelligent fusion dispatching background, an unmanned aerial vehicle rescue unit and a remote command background unit. The sound and vision dual-mode sensing unit integrates a phased array acoustic microphone array and a dual-spectrum vision module, performs comprehensive judgment through acoustic early warning guidance and vision focusing by using a self-adaptive weighted fusion algorithm, and realizes full-coverage detection of vocal distress drowning and silent drowning by combining a silent drowning special recognition strategy. The system adopts an edge computing and back-end dispatching collaborative architecture, automatically completes danger determination without manual decision, verifies the target through airborne vision for the second time, and dynamically corrects the dropping coordinates by combining a dropping dynamics model and a drift prediction model. The application realizes all-weather, full-automatic and low-false-alarm water area falling-into-water detection and rapid closed-loop rescue.

Description

Technical Field

[0001] This invention relates to the fields of water safety monitoring, intelligent sensing, and unmanned aerial vehicle (UAV) automatic rescue technology, and particularly to the field of all-weather, unmanned automatic detection, precise positioning, and rapid closed-loop rescue technology applicable to various water areas such as urban rivers, landscape waterways, reservoirs, lakes, swimming pools, and construction site waterlogged areas. Specifically, it relates to an audio-visual dual-modal full-domain intelligent detection and rescue system and method for falling into water. Background Technology

[0002] Drowning accidents in water areas are characterized by their suddenness, short rescue window (generally considered to be 4 to 6 minutes after falling into the water as the golden rescue time), and high risk. With rapid urbanization and the increasing prevalence of water features, the population density in urban rivers, scenic lakes, and reservoirs is continuously increasing, leading to frequent drowning accidents and posing a serious threat to public safety. Existing water monitoring and rescue technologies are numerous, and based on their underlying system architecture, they mainly fall into the following categories: The first is a purely visual monitoring solution. Existing technologies, such as Chinese patent CN112382055A, disclose a drowning prevention system applicable to various water areas. This system uses cameras to collect images, identifies individuals in dangerous areas through image processing technology, and issues warnings via loudspeakers. However, purely visual monitoring relies on visible light imaging, which is prone to failure in scenarios such as nighttime, backlighting, rain, fog, strong light reflection, and wave interference. Furthermore, it struggles to identify silent drowning states—where the person is unable to call for help due to choking or unconsciousness—resulting in high rates of missed and false alarms. The second type is a single acoustic monitoring scheme. Some schemes use microphones to collect sound signals from the water area to determine drowning. However, single acoustic monitoring can only acquire sound signals, lacking spatial positioning capabilities and visual confirmation. It is easily affected by environmental noise, has insufficient reliability, and is difficult to achieve accurate positioning. The third type is a drone rescue scheme with manual decision-making. Existing technologies, such as Chinese patent CN104700576A, disclose a rapid water rescue system and method, which uses fixed-wing drones for patrol surveillance and multi-rotor drones for rescue. Chinese patent CN121564579A further discloses a drone-based intelligent identification and automatic throwing rescue method for people falling into the water, which uses drones to collect water area images to detect people who have fallen into the water. However, these schemes mostly rely on manual discovery, confirmation, decision-making, or operation, which takes a long time from discovery to takeoff, and may cause the golden rescue time to be missed.

[0003] In summary, existing technologies are insufficient to simultaneously meet the comprehensive requirements of all-weather operation, full coverage, low false alarm rate, automatic identification, automatic positioning, automatic dispatching, and automatic rescue. There is an urgent need for a highly reliable, fully automatic, and fast-response water-fall detection and rescue system. Summary of the Invention

[0004] This invention aims to overcome the shortcomings of existing technologies in drowning detection and rescue. This application provides an audio-visual dual-modal full-domain intelligent drowning detection and rescue system and method, which achieves at least one of the following objectives: 1. The audio-visual dual-modal sensing unit can be flexibly installed at any location around the water area, achieving 24-hour full-domain monitoring; 2. Automatic identification, judgment, and location of drowning events, without the need for manual confirmation or decision-making throughout the process; 3. The intelligent dispatch backend automatically triggers rescue commands, dispatching drones to take off automatically and autonomously; 4. The drone verifies the target a second time during flight, dynamically correcting the deployment coordinates to improve rescue accuracy; 5. A fully automatic unmanned closed loop of "detection and processing, identification and rescue" is formed, achieving rapid response, low false alarms, and high reliability; 6. It covers drowning with audible cries for help and silent drowning, adapting to various complex water environments; it retains a manual emergency intervention channel, allowing manual takeover of dispatch in emergencies, balancing efficiency and safety.

[0005] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0006] This invention provides an audio-visual dual-modal full-domain intelligent water-falling detection and rescue system, comprising: an audio-visual dual-modal sensing unit, an intelligent fusion scheduling backend, an unmanned aerial vehicle rescue unit, and a remote command backend unit;

[0007] The dual-mode acoustic and visual sensing unit integrates a phased array acoustic microphone array, a dual-spectrum vision module, an edge computing unit, and a communication module. It is used for global sound pickup, sound source localization, water-falling soundprint recognition, day and night image acquisition, human contour and heat source detection, event determination, and precise coordinate calculation. The edge computing unit performs adaptive weighted fusion decision-making based on acoustic and visual signals and calculates the fusion confidence level. :

[0008] ;in, For acoustic confidence level, For visual confidence, Spatial consistency characteristics for acoustic localization and visual detection. , For adaptive weights, This is the spatial consistency weighting coefficient;

[0009] when And duration At that time, it was determined to be a drowning emergency, and the precise coordinates were calculated and uploaded to the dispatch backend; The threshold for high-confidence decisions. This is the duration threshold;

[0010] The intelligent fusion scheduling backend is configured with an NPU to receive data from the audio-visual dual-mode perception unit, perform hazard classification and coordinate calibration, automatically match the nearest standby drone nest, and automatically issue rescue commands without human decision-making. The drone rescue unit includes a flight control module, an airborne vision module, and an emergency execution module, which are used to automatically travel to the target area, perform secondary identification and verification of the target that has fallen into the water, dynamically correct the deployment coordinates, hover, provide audible and visual warnings, and accurately deliver rescue supplies. The remote command backend unit is used for data monitoring, tracing, scheduling, and manual emergency takeover.

[0011] Preferably, the acoustic confidence level The calculation method is as follows: ;in, To obtain the similarity features of the sound signatures of falling water, the temporal spectrum was analyzed. The results were obtained by dynamic time warping (DTW) matching with a pre-trained water splash sound pattern template library. , The current frame's sound pressure level is measured in dB. As the environmental noise base; The time-domain waveform characteristics of splashing water / calling for help, when there is a duration in the signal envelope. s and pulse interval The value is 1 if there is a pulse train of s, otherwise it is 0;

[0012] The visual confidence The calculation method is as follows: ;in, , For thermal imaging confidence, The area of ​​the infrared connected region. The threshold for the area of ​​the human body heat source. Temperature is measured in degrees Celsius (°C). , To improve the confidence level of human detection based on YOLOv8, This refers to the bottom position of the detection frame.

[0013] Preferably, the adaptive weights and Dynamic adjustment based on real-time confidence variance for each modality:

[0014] , ;

[0015] in, The acoustic confidence level The statistical variance over the past N frames, The visual confidence level Statistical variance over the past N frames; in the formula and These represent the reciprocals of the statistical variances mentioned above, i.e. and ;

[0016] The spatial consistency feature The calculation method is as follows: ;in, The normalized projection distance on the image plane between the acoustically located target's orientation and the visually detected target's position. Spatial tolerance parameter; the high confidence decision threshold The duration threshold s.

[0017] Preferably, the edge computing unit is further provided with a low confidence threshold. ,when or When the acoustic warning is triggered, the system enters the acoustic warning trigger mode, driving the dual-spectrum vision module to focus on the sound source location for enhanced identification; when the acoustic warning is triggered but the vision does not confirm, and At this time, the system enters a continuous observation window, and the observation window duration is... s; within the observation window:

[0018] like Maintain a value of ≥0.7 for more than 5 seconds, but If the confidence level is consistently less than 0.5, a low confidence warning will be triggered and pushed to the remote command backend unit for manual assessment.

[0019] If the acoustic signal disappears within the observation window If the value is less than 0.7, the warning will be automatically cancelled and normal monitoring will resume.

[0020] If in the observation window If so, a fusion decision will be triggered immediately.

[0021] Preferably, the edge computing unit also executes a silent drowning-specific identification strategy and calculates the silent drowning confidence level. :

[0022]

[0023] in, , The angle of inclination of the human torso to the vertical direction; , The velocity of human body displacement on the water surface is expressed in m / s. , The frequency of water surface ripples over the chest region of the human body, measured in Hz. ;when And duration When the time reaches s, it is determined to be a silent drowning hazard, and the rescue process can be initiated directly without acoustic triggering.

[0024] Preferably, the airborne vision module of the UAV rescue unit performs dynamic tracking and deployment coordinate correction of the target in the water during flight, and the emergency execution module has a built-in deployment dynamics model and an optimal deployment point calculation model; the deployment dynamics model is a three-degree-of-freedom dynamic equation system of the lifebuoy space, with the center of mass coordinates as the basis. The system of equations for the state variables is as follows:

[0025] ;

[0026] ;

[0027] ;

[0028] Where m is the mass of the lifebuoy, and g is the acceleration due to gravity. air density, The drag coefficient, , , For the windward area in each direction, , , For velocity components, , The horizontal component of wind speed; the drift prediction equation for the person falling into the water is:

[0029] ;

[0030] in, The velocity vector of the water surface. The wind speed vector; the optimal throwing point ,in, For the lifebuoys released from the drop point The location where the person fell into the water. This is the predicted location of the person who fell into the water after drifting.

[0031] Preferably, the drone rescue unit also executes an emergency handling procedure for deployment failure, specifically including:

[0032] If the delivery mechanism malfunctions and no release signal is detected within 3 seconds after the delivery command is issued, the backup electromagnet will be activated to force release with 1.5 times the rated current; if this still fails, the mechanism will hover and wait, and the remote command center will be notified.

[0033] The lifebuoy was not inflated and no expansion was detected within 5 seconds of it falling into the water. A spare lifebuoy was deployed and ground rescue was notified simultaneously.

[0034] If the lifebuoy deviates from the target and the distance between the landing point and the person in the water is greater than 5m, immediately recalculate and correct the throwing point and deploy a spare lifebuoy;

[0035] If the person who fell into the water loses consciousness and is detected to be face down with no active grabbing for 10 seconds, switch to deploying a buoyancy ring that can automatically wrap around the neck.

[0036] This invention also provides a dual-modal audio-visual intelligent detection and rescue method for falling into water, based on the detection and rescue system described above, and specifically includes the following steps:

[0037] Step STP100: The acoustic and visual dual-mode sensing unit acquires acoustic signals and dual-spectral images in parallel 24 hours a day, and the edge computing unit analyzes them online in real time.

[0038] Step STP200: The edge computing unit extracts acoustic features and calculates... ,when or When the acoustic warning is triggered, the location of the sound source is output and the dual-spectrum vision module is driven to focus.

[0039] Step STP300: The dual-spectrum vision module focuses on the target water area and extracts data in parallel. and ,calculate ;

[0040] Step STP400: The edge computing unit performs an adaptive weighted fusion decision. ;when and The system immediately identifies a drowning incident and calculates and uploads precise coordinates; simultaneously, it performs silent drowning detection in parallel. and If the time is s, it is directly determined as a silent drowning hazard;

[0041] Step STP500: The intelligent integrated dispatch backend does not require manual decision-making, and automatically performs hazard classification, coordinate calibration, matching of the nearest drone nest, and issuance of rescue instructions.

[0042] Step STP600: The drone automatically takes off and heads towards the target area. During the journey, the onboard vision module performs a secondary verification, and the optimal drop point is calculated in real time by combining the drop dynamics model and the drift prediction model. Dynamically adjust the deployment coordinates;

[0043] Step STP700: After the drone arrives and hovers at RTK level (error ≤ 0.8), activate the audible and visual warnings, and proceed as follows. Accurately deploy life-saving equipment and transmit on-site video;

[0044] Step STP800: The drone automatically returns to its home base to recharge, simultaneously pushing early warning information, and the entire process is recorded with video and data. Throughout the process, when communication is interrupted, it automatically switches to a backup link and enters local autonomous mode. When equipment fails, it automatically executes a three-level degradation operation. When deployment is abnormal, it automatically executes an emergency handling procedure. In case of emergency, it supports remote command and manual emergency takeover from the backend.

[0045] Beneficial effects:

[0046] This invention establishes an all-weather, all-scenario water monitoring capability through the deep fusion of acoustic perception and dual-spectral vision. The acoustic channel is unaffected by environmental conditions such as lighting, rain, fog, and backlighting, and can independently trigger warnings and guide the visual system to focus precisely. The infrared thermal imaging channel effectively locates human heat sources at night and in low-visibility conditions. The visible light channel provides high-resolution human contour confirmation under good lighting conditions. The coordinated operation of these three components enables the system to effectively identify drowning in both audible distress calls and silent drowning scenarios, overcoming the inherent limitations of single-modal approaches under different environmental conditions. This significantly reduces the probability of missed and false alarms, achieving adaptive perception in complex water environments.

[0047] This invention utilizes an end-to-end fully automated closed-loop architecture to seamlessly connect the perception, judgment, scheduling, and execution stages, eliminating the time-consuming intermediate steps of manual discovery, confirmation, and decision-making found in traditional solutions. The system achieves real-time fusion judgment through front-end edge computing, automatic task allocation and command issuance without human intervention through an intelligent scheduling backend, and dynamic target tracking and deployment coordinate correction through autonomous drone flight and secondary verification. The entire process, from incident occurrence to rescue response, is highly automated, significantly shortening emergency response time, especially providing a rapid response channel for silent emergencies such as silent drowning, effectively solving the problem of missing the golden rescue window.

[0048] This invention ensures high reliability across the entire chain and emergency response capabilities in extreme scenarios through system-level fault-tolerant design and multi-dimensional redundancy mechanisms. A three-level fault detection and three-level degradation operation mechanism ensures that the system maintains minimum monitoring and rescue functions under different fault levels, including single sensor degradation, complete modal failure, and even edge computing unit failure. A multi-link redundant communication scheme allows the system to enter local autonomous mode when communication is interrupted, independently making decisions and triggering the nearest drone to take off. A deployment dynamics model and emergency handling procedures for deployment failures ensure the accuracy and reliability of the rescue execution phase. A manual emergency takeover channel in the remote command backend provides a safety net for extremely complex scenarios. These mechanisms collectively ensure the system's continuous availability under all-weather, all-region, and all-fault modes, while the full-process data storage function provides complete evidence for post-event traceability and responsibility determination. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a system block diagram of the present invention.

[0051] In the diagram: 100 - Dual-mode audio-visual sensing unit; 110 - Phased array acoustic microphone array; 120 - Dual-spectrum vision module; 130 - Edge computing unit; 140 - Communication module; 150 - Power supply module; 200 - Intelligent fusion scheduling backend; 300 - UAV rescue unit; 400 - Remote command backend unit. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0053] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0054] Example 1:

[0055] This embodiment provides an audio-visual dual-modal full-domain intelligent water-falling detection and rescue system. (See also...) Figure 1 As shown, it includes: a dual-mode audio-visual perception unit 100, an intelligent fusion scheduling backend 200, a drone rescue unit 300, and a remote command backend unit 400.

[0056] Dual-mode audio-visual sensing units 100 are evenly deployed along the perimeter of the waterway, spaced 300 to 500 meters apart, with denser deployments at bends, under bridges, and in areas with high pedestrian traffic. Each sensing unit is an independent node, connected to the intelligent fusion scheduling backend 200 via a communication network. Unmanned aerial vehicle (UAV) automated nests are deployed every 1 to 2 kilometers along the waterway, each nest equipped with 1 to 2 standby industrial UAVs, in a 24-hour ready-to-trigger state. A remote command backend unit 400 is deployed in the emergency command center for overall monitoring and emergency takeover. The dual-mode audio-visual sensing unit 100 includes:

[0057] The phased array acoustic microphone array 110 consists of 16 MEMS microphones arranged in a uniform circular array with a diameter of 20cm. Each microphone channel includes a preamplifier circuit. It features a 20dB gain and a 24-bit ADC with a sampling rate of 48kHz. The array can achieve omnidirectional sound pickup with a horizontal range of 180° and a vertical range of 90°. Sound source localization employs the Generalized Cross-Correlation-Phase Transform (GCC-PHAT) algorithm, combined with a far-field array signal model, achieving an angular accuracy of ≤±1.5°. A windproof shield and a waterproof and acoustically permeable membrane are installed in front of the array, providing IP67 protection. An adaptive filter chip performs real-time multi-channel adaptive noise cancellation, using ambient wind noise and water flow noise as reference noise sources, achieving a noise reduction of ≥25dB.

[0058] The dual-spectrum vision module 120 includes a visible light camera and an infrared thermal imaging camera, both coaxially mounted on a two-axis gimbal. The gimbal has a horizontal rotation range of 360° and a pitch range of -15° to 90°. The visible light camera uses a 1 / 1.8-inch CMOS sensor, 4 megapixels, outputs 1080P / 25fps video stream, has a minimum illumination of 0.001 lux, supports a wide dynamic range (WDR) of 120dB, and has an effective detection distance of 0 to 150m. The infrared thermal imaging camera uses an uncooled vanadium oxide focal plane array, has a resolution of 640×512, a temperature measurement range of -20℃ to 150℃, a temperature measurement accuracy of ±0.5℃, and an effective detection distance of 0 to 150m.

[0059] The edge computing unit 130 adopts an ARM architecture AISoC, with a built-in NPU computing power of 6 TOPS, and is equipped with 4GB LPDDR4 memory and 32GB eMMC storage. It comes pre-installed with a lightweight Linux real-time operating system, achieving AI inference latency ≤50ms. The following algorithm models are deployed within the edge computing unit 130:

[0060] Voiceprint recognition model: A 1D-CNN-based classifier for water-falling voiceprints. The input is a 256×128 time-spectrum image after short-time Fourier transform, and the output is the similarity of the water-falling voiceprints. Sound pressure level estimation module: Real-time calculation of ambient noise floor That is, the moving average over the past 60 seconds; and the sound pressure level of the current frame. Time-domain waveform detection module: detects pulse train characteristics in the signal envelope. Target detection model: A lightweight human detector based on YOLOv8n, taking a 640×640 visible light image as input and outputting human bounding boxes and confidence scores. Infrared heat source detection module: Based on adaptive threshold connected region analysis, outputs heat source temperature. ,area and confidence level Pose estimation model: A lightweight pose estimator based on HRNet extracts 17 key points of the human body and calculates the tilt angle. And facial orientation; Optical flow motion analysis module: Based on Farneback dense optical flow method, calculate human motion velocity. Motion pattern entropy; Respiratory rate estimation module: Based on Fourier transform analysis of water surface ripple frequency in the human chest region. Fusion Decision Module: Performs adaptive weighted fusion decision based on audio and visual signals, and outputs the fusion confidence score. And the verdict.

[0061] The communication module 140 adopts a multi-link redundancy design, specifically including: Link 1: a 5G NR module, supporting SA / NSA networking, with a theoretical downlink speed of 2Gbps; Backup Link 1: a LoRa module, operating at a frequency of 470MHz, with a transmit power of 20dBm and a coverage radius of 2 to 5km; Backup Link 2: a UAV data transmission radio module, operating at a frequency of 900MHz, with a transmit power of 30dBm and a point-to-point communication coverage radius of 10km. The communication module 140 sends heartbeat packets to the intelligent fusion scheduling backend 200 every 500ms to monitor the link status in real time. It also includes a power supply module 150; the power supply module 150 adopts a dual power supply scheme of solar power and AC mains power. The solar panel has a power of 100W, equipped with a 12V / 100Ah lithium iron phosphate battery, which can provide continuous power for 72 hours on cloudy or rainy days. The AC mains adapter input is AC100-240V and outputs DC12V / 5A. The peak power consumption of the entire unit is 15W, and the standby power consumption is 5W. It has an IP67 protection rating, an operating temperature range of -10℃ to 60℃, and a built-in surge protection module and heater.

[0062] Example 2:

[0063] This embodiment provides a preferred implementation of the audio-visual adaptive weighted fusion decision algorithm based on embodiment 1. The specific implementation steps are as follows:

[0064] Step 1: Acoustic Feature Extraction. After the raw signal acquired by the acoustic microphone array 110 is synchronously sampled through 16 channels, the edge computing unit 130 performs the following processing:

[0065] First, a time-frequency transformation is performed. For each frame of signal, in this embodiment, a short-time Fourier transform is performed with a frame length of 512 sampling points, a frame shift of 256 sampling points, and a Hanning window applied to obtain the time spectrum. ;

[0066] Then, voiceprint similarity calculation is performed. Dynamic time-warping matching is performed with a pre-trained water-falling sound template library. The template library contains over 5000 real and simulated water-falling sound samples, covering different distances, water environments, and water-falling methods, including active jumping, slipping, and silent sinking. Dynamic time warping outputs the cumulative distance under the optimal alignment path, which is converted into a similarity score. .

[0067] Next, sound pressure level anomaly detection is performed, and the sound pressure level of the current frame is calculated. ,in, Pa. Environmental noise floor This is a moving average over the past 60 seconds. Sound pressure level anomaly characteristics. .

[0068] Next, time-domain waveform detection is performed, and pulse detection is performed on the signal envelope. When a duration exists... s and pulse interval When the pulse train of s, Otherwise, it is 0. This feature is used to identify temporal patterns of splashing motions and cries for help.

[0069] Finally, acoustic confidence level calculation is performed: .

[0070] Step 2: Visual feature extraction. After the dual-spectrum visual module 120 focuses on the target water area, the edge computing unit 130 executes the process in parallel. This specifically includes infrared heat source detection: after non-uniformity correction and adaptive threshold segmentation of the infrared image, connected regions with temperatures ranging from 32℃ to 37℃ are extracted. The area of ​​the connected regions is then calculated. And adaptively adjust the area threshold according to the distance between the sensing unit and the target. Infrared heat source characteristics: ,in, The thermal imaging confidence score, influenced by factors such as ambient temperature, humidity, and target distance, is output in real-time by the infrared camera's built-in quality assessment module. Visible light human detection: A modified YOLOv8n model is input to a visible light image, and the output is a human detection bounding box. and detection confidence Determine the bottom of the detection box. Whether the location is in a water surface area or near a water area is determined by a pre-defined water area semantic segmentation mask. Visible light human body contour features: Then, visual confidence is calculated using the following formula: To achieve; among which, , For thermal imaging confidence, The area of ​​the infrared connected region. The threshold for the area of ​​the human body heat source. Temperature is measured in °C.

[0071] Step 3: Spatial consistency calculation; the acoustic localization module outputs the azimuth angle of the target sound source. After coordinate transformation and projection onto the image plane, the acoustic projection points are obtained. The visual inspection module outputs the center position of the human detection box. Calculate the normalized Euclidean distance between the two on the image plane:

[0072] Where W and H are the width and height of the image, respectively, in pixels. Spatial consistency feature: ;in, This is a spatial tolerance parameter, corresponding to a deviation tolerance of approximately 10% of the image size on the image plane. When the acoustic localization and visual detection spatial location are consistent, →1; the larger the deviation, the better. The smaller it is.

[0073] Step 4: Adaptive weight calculation. The ambient light sensor collects the ambient illuminance E (lux) in real time, and the wind speed sensor collects the wind speed. (m / s), environmental parameters are collected by temperature and humidity sensors. The edge computing unit dynamically adjusts according to the following rules. and .in,

[0074] When daytime sunlight is good (E≥100 lux) and there is no rain or fog. , ;

[0075] When nighttime light intensity is low (E < 10 lux) or during rainy or foggy weather... , ;

[0076] In the transition scenario 10≤E<100lux, , .

[0077] In this example, regarding the adjustment of confidence variance: statistics were collected over the past N=30 frames, which is approximately 1.2 seconds. and variance and When a modality has a large variance and its output is unstable, its weight should be reduced. , ,in, and ;

[0078] Step 5: Fusion decision, calculated using the following fusion confidence formula:

[0079] Spatial consistency weight coefficient Empirical values ​​enable spatial consistency to play a supplementary corrective role in the fusion process.

[0080] The logic behind the judgment is as follows:

[0081] when And duration When s, it is determined to be a drowning emergency, and the rescue procedure is immediately triggered; when 0.5≤ <0.75 or When the value is ≥0.7, it is determined to be an acoustic warning trigger, driving the vision module to focus and enhance recognition; when When the value is less than 0.5, the warning will be lifted and full-area monitoring will continue.

[0082] Step 6: Observe the window mechanism; when the acoustic warning is triggered... ≥0.70, but visual confirmation not yet available. When the value is less than 0.50, the system does not immediately cancel the warning, but instead enters a continuous observation window for a specified duration. s.

[0083] like Maintain a value greater than or equal to 0.7 for more than 5 seconds, but If the value is consistently less than 0.5, a "low confidence warning" is triggered and sent to the remote command center for manual assessment; if the acoustic signal disappears within the observation window... If the value is less than 0.70, the warning will be automatically canceled and normal monitoring will resume; if it is within the observation window... If the value is ≥0.50, immediately exit the observation window and trigger the fusion decision.

[0084] Example 3:

[0085] Based on any of the above embodiments, this embodiment provides a preferred identification strategy for silent drowning. Specifically, silent drowning refers to the drowning state in which the person who has fallen into the water is unable to call for help due to choking, unconsciousness, or other reasons. Typical characteristics of a silent drowning victim include: remaining still on the water surface, floating face down or upright, no active swimming movements, and extremely low or absent breathing rate.

[0086] The silent drowning detection and rescue system provided in this embodiment has a dedicated identification strategy that does not rely on acoustic signals at all, but is based entirely on visual features for identification, and includes the following three stages:

[0087] Phase 1: Human target detection and tracking runs continuously. Edge computing unit 130 detects human targets in the water area based on an improved YOLOv8n model and outputs detection boxes. With confidence level The DeepSORT algorithm is used for multi-target tracking, assigning a unique ID to each target and recording its trajectory. .

[0088] The second stage involves posture and motion feature extraction. The system updates every second and includes posture estimation, motion feature calculation, and respiratory feature calculation. Posture estimation uses a lightweight HRNet model to extract 17 key points of the human body, including left and right shoulders, left and right hips, head, left and right wrists, and left and right ankles, calculating the following posture features: body tilt angle. The angle between the line connecting the midpoint of the shoulder to the midpoint of the hip and the vertical direction. A normal swimmer's body tilt angle is usually less than 15°, while a silent drowning victim, due to loss of balance, often has a tilt angle greater than 30°. Facial orientation is determined by the relationship between key head points, such as the left and right eyes, nose, and ears, and the torso; a face-down orientation is a typical characteristic of silent drowning.

[0089] The motion characteristic calculation includes water surface displacement velocity. The unit is m / s. Normal swimmers typically have a speed greater than 0.5 m / s, while the speed of a silent drowning victim approaches 0. The calculation of motion pattern entropy involves calculating the entropy of changes in motion direction over the past N frames. Normal swimmers have high motion pattern entropy, while silent drowning victims have low entropy, either stationary or drifting with the current. The analysis also includes Fourier transform analysis of the water surface ripple frequency in the chest region of the human body. The unit is Hz. The normal respiratory rate for a person is 0.2 to 0.33 Hz, or 12 to 20 breaths per minute. (This refers to the respiratory rate during silent drowning.) A state that approaches 0 or is extremely irregular.

[0090] Phase Three: Comprehensive Assessment of Silent Drowning. First, define the confidence level for silent drowning.

[0091]

[0092] The calculation of each component is as follows:

[0093] First, posture scoring , . The angle of inclination of the human torso to the vertical direction is considered. When the angle of inclination is greater than 30° or the face is facing down, this item scores higher.

[0094] Secondly, the athletic score , The value represents the human body's displacement velocity on the water surface, in m / s. When the displacement velocity is less than 0.3 m / s and the motion mode entropy is low, the score for this item approaches 1.

[0095] Furthermore, there is the respiratory score. , The frequency of water surface fluctuations in the chest area of ​​the human body is measured in Hz. When the respiratory rate is below 0.1Hz or irregular, the score for this item approaches 1.

[0096] Finally, thermal feature scoring Infrared thermal imaging provides auxiliary confirmation.

[0097] when And duration When the time reaches s, it is determined to be a silent drowning hazard, and the rescue process can be initiated directly without acoustic triggering.

[0098]

[0099] The test was conducted in a controlled aquatic environment, using a human model to simulate silent drowning. The sensing unit was deployed 20 to 50 meters above the water surface and 30 to 100 meters from the test point. Response time was started from the moment the target entered the water (or entered the monitoring field of view) until the system issued a silent drowning warning. Under the same test conditions, the silent drowning recognition accuracy of the pure vision solution (without dual-spectral fusion) was 78.5%, while this system achieved 93.3%, an improvement of 14.8 percentage points.

[0100] Example 4:

[0101] This embodiment, based on any of the above embodiments, elaborates on a preferred embodiment regarding the specific structure and deployment dynamics model of the drone rescue unit 300, specifically including...

[0102] The drone platform utilizes a hexacopter industrial drone with a wheelbase of 1200mm, a maximum takeoff weight of 25kg, and a maximum payload of 5kg. It is wind resistant to levels 6-7, has an IP54 protection rating, and operates in temperatures ranging from -10℃ to 60℃. It is equipped with an RTK differential positioning module (hovering accuracy ≤0.8m), a binocular visual obstacle avoidance system (detection distance 0.5-20m), and a 4G / 5G image transmission module (unlimited transmission distance, dependent on cellular network coverage).

[0103] The airborne vision module includes: a forward-facing dual-light camera with 2 megapixels of visible light and infrared thermal imaging, used for target search and secondary verification during flight; a downward-facing single-light camera with 4 megapixels of visible light, used for precise target positioning and monitoring of the status of those who have fallen into the water before deployment; and a laser ranging module used to measure the distance between the UAV and the water surface / target with an accuracy of ±0.1m.

[0104] The emergency execution module includes an electromagnetic delivery mechanism: dual-channel independent control, capable of carrying 2 to 3 rescue devices, with a single device release response time ≤0.5s; an automatically inflatable life ring, folded dimensions 20cm×15cm×10cm, automatically inflating upon contact with water, with buoyancy ≥150N; and optionally equipped with a buoyancy ring: automatically wrapping around the neck of a person falling into the water, suitable for scenarios where the person has lost consciousness; and an audible and visual warning device: a high-brightness LED strobe light with a visibility distance ≥5km + a high-power speaker (100dB@1m).

[0105] The deployment dynamics model shows that the motion of the lifebuoy after release from the drone is influenced by gravity, air resistance, and wind. Using the lifebuoy's center of mass position (x, y, z) as the state variables, a system of three-degree-of-freedom dynamic equations is established:

[0106] ;

[0107] ;

[0108] ;

[0109] The optimal values ​​for each parameter and their data sources are shown in the table below:

[0110]

[0111] Considering that a person falling into the water will drift due to the influence of water flow and wind, drift prediction is accomplished using the following equation: ;

[0112] in, The water surface velocity vector is estimated in real time by the UAV's onboard vision using the optical flow method. The wind speed vector is given, and the coefficient 0.03 represents the drag coefficient of the wind on the drifting human body, calibrated through actual measurement. The optimal throwing point is calculated by combining the lifebuoy's descent trajectory with the predicted drift of the person in the water. ,in, To the candidate throwing point The released lifebuoy's descent time The theoretical landing position was obtained by numerical integration of the above three-degree-of-freedom dynamic equations. Time elapsed for the person who fell into the water Add rafting time The predicted position is obtained by the drift prediction model; The descent time is determined by the drop height H and the vertical motion equation of the lifebuoy, and assuming a simplified estimate that neglects air resistance. . The estimated time from when the lifebuoy hits the water to when it is touched by the person in the water is taken as 3 to 5 seconds. Specifically, the solution process can use the existing Monte Carlo method: generate 50 to 100 random perturbation points around the candidate throwing point to simulate the landing point of the lifebuoy and the drift trajectory of the person in the water under different wind fields and water flow conditions, and select the throwing point that minimizes the expected distance between the lifebuoy and the person in the water. This part is existing technology and can be easily completed by those skilled in the art, so it will not be described in detail in this embodiment.

[0113] The drone continuously performs the following dynamic correction cycle during flight: The first aspect is hovering positioning correction, with the onboard vision module updating the location of the person who fell into the water every 200ms. The flight control module dynamically adjusts the hovering point based on the real-time position; the second aspect is the updating of environmental parameters: the airborne anemometer and visual optical flow method recalculate every 500ms. and The process involves four steps: first, resolving for the optimal drop point; second, selecting the drop timing: triggering the drop when the actual drone position deviates from the optimal drop point by less than 0.3m; and third, post-drop verification: after the lifebuoy hits the water, the drone continuously tracks the lifebuoy and the person in the water. If the distance between them is less than 2m and lasts for more than 5 seconds, the drop is considered successful; if the distance is greater than 5m, the drop is considered failed, triggering a re-drop. The emergency handling procedure for drop failures during drone drop operations is as follows:

[0114] Scenario 1: If no release signal is detected within 3 seconds after the release command is issued, it is determined that the release mechanism is stuck. The backup electromagnet will be activated immediately to force release with 1.5 times the rated current. If it still fails, it will hover and wait for further instructions and notify the remote command backend.

[0115] Scenario 2: If no expansion is detected within 5 seconds of falling into the water, it is determined that the lifebuoy is not inflated. In this case, a spare lifebuoy should be deployed immediately, and ground rescue forces should be notified simultaneously.

[0116] Scenario 3: When the distance between the point where the lifebuoy lands and the person in the water is greater than 5m, it is determined to be a deviation from the target, and the throwing point needs to be recalculated and corrected immediately, and a spare lifebuoy needs to be deployed.

[0117] Scenario 4: If the person is detected to be face down and without any active grabbing action for 10 seconds, it is determined that the person has lost consciousness. In this case, it is necessary to switch to deploying a buoyancy ring that can automatically wrap around the neck.

[0118] Example 5:

[0119] To improve system reliability, this embodiment adds a three-level fault detection and three-level degradation operation mechanism based on any of the above embodiments. The first level is a hardware self-test, with a detection cycle of 1-3 seconds in this example. Specifically, it includes detecting the bias voltage and sensitivity of each MEMS microphone. If the bias voltage deviates from the rated value by ±15% or the sensitivity drops by more than 3dB, the microphone channel is determined to be faulty, marked, and removed from the array, completing the sensor array detection. It also detects the image frame rate and exposure status. If the frame rate is lower than 50% of the nominal value or the image is completely black / white, and the pixel average deviates from the normal range, it is determined to be a camera fault. The CPU / GPU / NPU utilization and temperature are detected. If the utilization rate remains at 100% for more than 30 seconds or the temperature is higher than 85℃, it is determined to be an overload fault, triggering frequency reduction protection. Finally, 4G / 5G signal strength and packet loss rate are detected. If the signal strength is lower than -110dBm or the packet loss rate is higher than 15%, it is determined to be communication degradation, triggering link switching. The second level of testing is a functional self-test, with a testing cycle set to 10-15 seconds. A 1kHz, 60dB standard test tone is injected through a built-in miniature speaker to test whether the microphone array can correctly identify the frequency and positioning direction. If the identification deviation exceeds ±5% or the positioning deviation exceeds ±3°, it is considered an acoustic functional malfunction. Target detection is performed on preset reference objects, namely a standard color chart and a heat source calibration plate installed within the field of view of the sensing unit, to verify whether the visible light and infrared channels can correctly identify and output detection results. The built-in test dataset, containing 10 known positive water-fall samples and 10 known interference negative samples, is run to verify whether the recognition accuracy of the current algorithm model is ≥90%. If it is below the threshold, the model is automatically rolled back to the previous stable version. The third level of testing involves a system-level health assessment. This cycle can be longer, such as 30 seconds or longer. Those skilled in the art can freely choose an appropriate cycle based on the actual application scenario, hardware operating conditions, stability, etc. A system health score is calculated by comprehensively considering the hardware status and functional self-test results. ,in, The health coefficient for each submodule is set as follows: 1 for normal, 0 for fault, and a value between 0.3 and 0.9 for degradation. The system alarm is triggered and reported to the remote command backend.

[0120] The three-level degradation operation mode effectively ensures that the system maintains a minimum level of monitoring and rescue capabilities under any fault conditions, preventing complete system failure due to a single point of failure. The specific mode settings are as follows: Slight degradation of a single sensor, such as a decrease in the sensitivity of a single microphone or slight contamination of a visible light lens, is classified as Level 1 (mild). In this mode, the acoustic / visual dual-modal operation continues, while the degraded sensor is downgraded; a high threshold is used for fusion-based decision-making. The threshold value was increased from 0.75 to 0.82, and the judgment threshold was set at... ;

[0121] Level 2 (moderate) indicates a complete modal failure, either due to the acoustic array or infrared / visible light failure. The operating mode switches to single-modal independent operation; decisions are based solely on available modalities; the warning push frequency is doubled; and the single-modal threshold is [value missing]. .

[0122] The trigger condition for Level 3 moderate degradation is the failure of the edge computing unit or the simultaneous failure of the audio and visual dual modes. In this case, it is necessary to switch to pass-through mode to transmit the raw sensing data directly to the cloud, where the backup AI model makes the judgment; only the data collection and forwarding functions are retained locally.

[0123] Example 6:

[0124] This embodiment, based on any of the above embodiments, provides a preferred multi-link redundant communication and local autonomy scheme. Firstly, the communication module 140 adopts a three-link redundancy design. The main link uses a 4G / 5G cellular network for daily data uploads, video stream transmission, and command issuance. Backup link 1 uses LoRa long-range low-power communication with a coverage radius of 2 to 5 km, used for transmitting critical commands and alarm information when the main link is interrupted. Backup link 2 uses a drone data radio for point-to-point communication with a coverage radius of 10 km, used for direct communication between drones and the issuance of emergency commands. The communication module 140 sends heartbeat packets to the intelligent fusion scheduling backend 200 every 500 ms, containing the device ID, timestamp, and link quality. If no ACK confirmation signal is received from the backend for three consecutive times, or if the local pending data buffer queue depth exceeds 100 records, the main communication link is determined to be interrupted. During communication interruptions, the dual-mode audio-visual sensing unit automatically executes the following local autonomous strategies: the edge computing unit 130 independently completes all audio-visual fusion decisions and hazard assessments, without relying on cloud computing power; all real-time detection data and event records are stored in local eMMC storage (32GB), with a cache capacity supporting at least 72 hours of continuous data storage. A first-in-first-out (FIFO) strategy is adopted to ensure that the latest data is retained first; when a water-related hazard is detected, a forced takeoff command is issued directly to the nearest unmanned aerial vehicle (UAV) nest via a LoRa backup link or a data transmission radio backup link, bypassing the main server. The command includes: hazard GPS coordinates, hazard confidence level, and timestamp; when the main communication link is restored, all locally cached data is automatically retransmitted to the intelligent fusion scheduling backend in chronological order to ensure data integrity. The retransmission uses a breakpoint resumption mechanism to support reliable transmission under unstable network conditions.

[0125] Those skilled in the art should understand that, without departing from the concept of the present invention, several alternative solutions can be made based on the above embodiments: (1) The acoustic microphone array is not limited to MEMS microphones, but can also be a condenser microphone or a piezoelectric microphone; the array shape is not limited to a circle, but can also be a linear array, a rectangular array or a random array. (2) The dual-spectrum vision module is not limited to a combination of visible light and infrared, but can also be a combination of visible light, near-infrared or visible light and multispectral. (3) The UAV is not limited to a multi-rotor, but can also be a vertical take-off and landing fixed-wing UAV or a tiltrotor UAV. (4) The rescue equipment is not limited to an automatic inflatable lifebuoy, but can also be a life-saving float, a life-saving net, an inflatable buoyancy chamber, etc. (5) The communication method is not limited to 4G / 5G, LoRa, data radio, but can also be Wi-Fi, ZigBee, satellite communication, etc. (6) The deployment method of the sensing unit is not limited to fixed installation, but can also be mobile deployment, such as mounted on patrol boats, unmanned surface vessels, etc.

[0126] Example 7:

[0127] This embodiment provides a dual-modal audio-visual intelligent water-falling detection and rescue method. It employs a detection and rescue system comprised of hardware and software from any of the above embodiments. Taking a specific water-falling incident as an example, the complete rescue method of this system is illustrated. Scenario setting: A city river, 11:00 PM, ambient light 5 lux, light breeze, wind speed 3 m / s. A person accidentally falls into the water from the bank, making splashing sounds and calling for help, while struggling in the water.

[0128] The entire detection and rescue operation includes the following steps:

[0129] Step STP100 involves a dual-mode acoustic and visual sensing unit that acquires acoustic signals and dual-spectral images in parallel 24 hours a day, while an edge computing unit performs real-time online analysis. The dual-mode acoustic and visual sensing unit 100, deployed on the riverbank approximately 50 meters from the point of impact with the water, operates 24 hours a day. An acoustic microphone array 110 continuously acquires ambient sounds, and a dual-spectral visual module 120 continuously acquires images of the water area. An edge computing unit 130 analyzes the data in real time.

[0130] Step STP200: The edge computing unit extracts acoustic features and calculates... ,when or The system triggers an acoustic warning, outputs the location of the sound source, and drives the dual-spectrum vision module to focus; specifically, the edge computing unit 130 extracts the water-pumping pulse train from the acoustic signal. s, s, Voiceprint similarity The sound pressure level is abnormal and exceeds the template matching threshold. If the noise level is more than 15 dB higher than the ambient noise level, calculate the acoustic confidence level. ,because If the sound source is detected, an acoustic warning will be triggered, and the sound source positioning module will output an azimuth angle of 25° horizontally and -8° vertically, driving the gimbal to rotate to the target water area.

[0131] Step STP300: After the dual-spectrum vision module 120 focuses on the target water area, infrared thermal imaging detects a human heat source at 32.5℃. , , The saturation level exceeded 1.0. Human silhouettes were detected using a starlight-level sensor under low-light conditions at night. , Calculate visual confidence .

[0132] Step STP400, in a low-light environment at night, with an ambient illuminance of 5 lux as the baseline weight. , Within the past 30 frames Small variance , The variance is slightly larger. Calculate adaptive weights , Normalized distance between the acoustic projection point and the center of the visual inspection box. , Calculate the fusion confidence score. . If the duration exceeds 1.5 seconds, it is determined to be a drowning emergency. The edge computing unit calculates the precise GPS coordinates with an accuracy of ±2m and uploads them to the intelligent fusion scheduling backend.

[0133] Step STP500: After receiving the emergency information, the intelligent fusion scheduling backend 200 automatically executes the following without manual confirmation: emergency classification, of which Level 1 is the highest priority, coordinate calibration, matching the nearest (1.2km) and best-condition drone nest, and automatically issuing rescue instructions, including target GPS coordinates, emergency confidence level, and timestamp.

[0134] In step STP600, after receiving the command, the drone pod completes self-check, unlocking, and takeoff within 15 seconds. It then proceeds to the target area at a speed of 15 m / s along the planned route. During flight, the onboard vision module performs secondary identification of the target area, confirming that the target is still near its original location, dynamically locking onto the target's position, and calculating the optimal drop point in real time using the drop dynamics model and drift prediction model.

[0135] Step STP700: After the drone arrives at the target area, it hovers precisely at a height of approximately 5 meters above the person in the water using RTK. It activates audible and visual warnings, such as strobe lights and a loudspeaker broadcasting "Do not panic, rescue is on its way," etc. Of course, any method that is conspicuous and effective is acceptable; the method of warning is not limited and can be customized by those skilled in the art. An automatically inflatable lifebuoy is deployed at the optimal drop point. The lifebuoy automatically inflates within 3 seconds of hitting the water, approximately 1.5 meters from the person in the water. The person grabs the lifebuoy and awaits further rescue. Simultaneously, the drone transmits real-time video of the scene to the remote command center via 5G.

[0136] Step STP800: After deployment, the drone automatically returns to its home base and connects to the charging dock. The system simultaneously pushes early warning information to the emergency management platform, the river chief's office, and patrol teams. The entire process is automatically recorded with video and data. From the moment the drone falls into the water to the deployment of the lifebuoy, the entire process takes less than 2 minutes.

[0137] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A dual-modal (audio-visual) intelligent water-falling detection and rescue system, characterized in that, include: Dual-mode audio-visual perception unit, intelligent fusion dispatch backend, drone rescue unit and remote command backend; The sensing unit integrates a phased array acoustic microphone array, a dual-spectrum vision module, an edge computing unit, and a communication module, and is used for global sound pickup, sound source localization, water splash sound pattern recognition, day and night image acquisition, human body contour and heat source detection, event judgment, and coordinate calculation. Its edge computing unit performs an acoustic-visual adaptive weighted fusion decision, calculating a fusion confidence score by weighting acoustic confidence, visual confidence, and spatial consistency features. The weights of the acoustic and visual confidence scores are dynamically adjusted according to the environment and the stability of their respective confidence scores. The spatial consistency features characterize the spatial consistency between acoustic localization and visual detection. When the fusion confidence score is greater than a high confidence threshold and the duration reaches a preset threshold, it is determined to be a water-falling hazard and the precise coordinates are uploaded. The intelligent fusion scheduling backend is configured with an NPU, which is used to receive data, perform hazard classification and coordinate calibration, automatically match the nearest standby drone nest, and automatically issue rescue instructions without human decision-making. The drone rescue unit includes a flight control module, an airborne vision module, and an emergency execution module, which are used to automatically travel to the target area, perform secondary verification of the water-fallen target, dynamically correct the delivery coordinates, hover, provide audible and visual warnings, and accurately deliver rescue supplies. The remote command back-end unit is used for data monitoring, tracing, scheduling, and manual emergency takeover.

2. The system according to claim 1, characterized in that, The edge computing unit calculates the acoustic confidence score as follows: First, it performs time-frequency transformation on the acquired acoustic signal to obtain the time spectrum; second, it performs dynamic time warping matching on the time spectrum and a pre-trained water-falling sound pattern template library to obtain water-falling sound pattern similarity features; simultaneously, it calculates the excess of the current frame's sound pressure level relative to the environmental noise floor and limits its amplitude to obtain sound pressure level anomaly features; furthermore, it detects whether there are pulse trains with specific durations and intervals in the signal envelope, and if so, it determines that they possess time-domain waveform features; finally, it performs a weighted summation of the water-falling sound pattern similarity features, the sound pressure level anomaly features, and the time-domain waveform features according to a preset first weight combination to obtain the acoustic confidence score. The visual confidence score is calculated as follows: First, visual features are extracted from the infrared thermal imaging channel and the visible light imaging channel, respectively. The visual features extracted from the infrared channel are the product of the thermal imaging confidence score, the ratio of the infrared connected region area to the human body heat source area threshold, and the determination result that the detected temperature is within the human body temperature range. The visual features extracted from the visible light channel are the product of the human body detection confidence score based on the improved YOLOv8 and the determination result that the bottom of the detection frame is located in the water surface area or near the edge of the water area. Finally, the visual features from the infrared channel and the visual features from the visible light channel are weighted and summed according to a preset second weight combination to obtain the visual confidence score.

3. The system according to claim 1, characterized in that, The adaptive weights are dynamically adjusted based on the proportional relationship between the reciprocal of the statistical variance of acoustic confidence over the past multiple frames and the reciprocal of the statistical variance of visual confidence over the past multiple frames. The spatial consistency feature is a Gaussian kernel function value calculated based on the normalized projection distance between the acoustic positioning target's orientation and the visual detection target's position on the image plane. This value is used to measure the degree of spatial consistency between the two. The high confidence threshold is 0.75, and the duration threshold is 1.5 seconds.

4. The system according to claim 1, characterized in that, The edge computing unit is also equipped with a low confidence threshold. When the fusion confidence is between the low confidence threshold and the high confidence threshold or the acoustic confidence is higher than the preset acoustic warning threshold, the system enters the acoustic warning trigger mode and drives the dual-spectrum vision module to focus on the sound source location for enhanced recognition. When an acoustic warning is triggered but visual confirmation is not available, the system enters a continuous observation window; Within the observation window: if the acoustic confidence level remains higher than the acoustic warning threshold for more than a preset duration while the visual confidence level remains lower than the preset visual confirmation threshold, a low confidence warning is triggered and pushed to the remote command backend unit for manual assistance in judgment. If the acoustic confidence level drops below the acoustic warning threshold within the observation window, the warning will be automatically revoked and normal monitoring will resume. If the visual confidence level reaches the visual confirmation threshold within the observation window, a fusion decision is immediately triggered.

5. The system according to claim 1, characterized in that, The edge computing unit also executes a silent drowning-specific identification strategy to calculate the silent drowning confidence score. This score is obtained by a weighted sum of four factors: posture score, motion score, breathing score, and thermal feature score. The posture score is determined based on the torso tilt angle and facial orientation. The motion score is calculated based on the body's water surface displacement velocity. The breathing score is calculated based on the water surface fluctuation frequency in the chest region. The thermal feature score is obtained by multiplying the thermal imaging confidence score by the determination that the detected temperature falls within the body's temperature range. When the confidence level of the silent drowning is greater than the preset silent drowning judgment threshold and the duration reaches the preset silent drowning duration threshold, it is judged as a silent drowning hazard, and the rescue process can be directly initiated without acoustic triggering.

6. The system according to claim 1, characterized in that, The airborne vision module of the UAV rescue unit performs dynamic tracking and deployment coordinate correction of the target that has fallen into the water during flight, and the emergency execution module has a built-in deployment dynamics model and optimal deployment point calculation model. The deployment dynamics model is a set of three-degree-of-freedom dynamic equations for the lifebuoy space. The equations of motion are established by taking the three directional components of the center of mass coordinate as state variables and comprehensively considering factors such as the mass of the lifebuoy, gravitational acceleration, air density, drag coefficient, windward area in all directions, velocity components in all directions, and horizontal component of wind speed. The system also includes a drift prediction model for people who have fallen into the water. Through integral calculations, it comprehensively considers the dragging effects of water surface velocity vector and wind speed vector to predict the future position of people who have fallen into the water. The optimal throwing point is determined as follows: among the candidate throwing points, find the point that minimizes the error between the water entry position of the lifebuoy released from that point after the fall time and the future position of the person in the water predicted by the drift prediction model, and take that point as the optimal throwing point.

7. The system according to claim 6, characterized in that, The drone rescue unit also executes an emergency handling procedure for deployment failures, specifically including: Deployment mechanism stuck: If no release signal is detected within the predetermined time after the deployment command is issued, the backup electromagnet is activated to increase the current and force release; if it still fails, it hovers and waits for command and notifies the remote command center. Uninflated lifebuoy: If no expansion is detected within the predetermined time after the lifebuoy falls into the water, deploy a spare lifebuoy and simultaneously notify ground rescue. Deviation from target: The distance between the lifebuoy's landing point and the person in the water is greater than the predetermined value. Immediately recalculate and correct the throwing point and deploy a spare lifebuoy. If the person falls into the water and loses consciousness: If the person is face down and does not actively grab onto anything for a predetermined time, switch to deploying a buoyancy ring that can automatically wrap around the neck.

8. A dual-modal (audio-visual) intelligent detection and rescue method for all-domain water falls, employing the system described in any one of claims 1 to 7, characterized in that, Includes the following steps: Step STP100: The acoustic and visual dual-mode sensing unit acquires acoustic signals and dual-spectral images in parallel around the clock, and the edge computing unit analyzes them online in real time. In step STP200, the edge computing unit extracts acoustic features and calculates acoustic confidence. When the acoustic confidence is higher than the preset acoustic warning threshold or the fusion confidence is higher than the low confidence threshold, an acoustic warning is triggered, the sound source location is output, and the dual-spectrum vision module is driven to focus. Step STP300: The dual-spectrum vision module focuses on the target water area, extracts infrared and visible light visual features in parallel, and calculates visual confidence. In step STP400, the edge computing unit performs adaptive weighted fusion decision and calculates the fusion confidence. When the fusion confidence is greater than the high confidence decision threshold and the duration reaches the preset duration threshold, it is determined to be a drowning hazard, and the accurate coordinates are calculated and uploaded. At the same time, silent drowning recognition is performed in parallel. When the silent drowning confidence is greater than the preset silent drowning judgment threshold and the duration reaches the preset silent drowning duration threshold, it is directly determined to be a silent drowning hazard. Step STP500: The intelligent integrated dispatch backend does not require manual decision-making, and automatically performs hazard classification, coordinate calibration, matching of the nearest drone nest, and issuance of rescue instructions. In step STP600, the drone automatically takes off and heads to the target area. During the journey, the onboard vision module performs a second verification and calculates the optimal drop point in real time by combining the drop dynamics model and the drift prediction model, and dynamically corrects the drop coordinates. Step STP700: After the drone arrives, it hovers in RTK mode, activates the audible and visual warnings, accurately deploys the life-saving equipment at the optimal drop point, and transmits the on-site video back. Step STP800: The drone automatically returns to home and charges, simultaneously pushing out early warning information, and the entire process is recorded with video and data. Throughout the entire process, when communication is interrupted, the system automatically switches to a backup link and enters local autonomous mode; when equipment fails, it automatically performs a three-level degradation operation; when deployment is abnormal, it automatically executes an emergency handling procedure; and in emergency situations, it supports remote command and manual emergency takeover from the back office.

Images