A River Drowning Monitoring Method Based on Lossless Downsampling Network and Multimodal Semantic Disambiguation

Abstract

This invention relates to a river drowning monitoring method based on lossless downsampling networks and multimodal semantic disambiguation, belonging to the field of computer vision. The method includes: collecting heterogeneous data and performing enhanced preprocessing before inputting it into a target detection network to detect geometric detection confidence scores; using these scores for initial target screening; calculating the dense light flow within a continuous time window for suspected targets and extracting the light flow motion entropy based on the dense light flow field; selecting keyframe images and preceding / following temporal feature maps based on the motion entropy; performing feature-semantic mapping based on a cross-modal attention mechanism using a large visual model in the cloud to generate textual semantic results; performing Bayesian weighted fusion of the geometric detection confidence scores detected by the target detection network and the textual semantic results output by the large visual model; and finally triggering a terminal audio-visual alarm. This invention overcomes the bottleneck of traditional visual algorithms lacking behavioral semantic cognition and significantly reduces the false drowning alarm rate.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method for monitoring drowning in rivers based on lossless downsampling networks and multimodal semantic disambiguation. Background Technology

[0002] Drowning accidents pose a significant threat to public safety, especially in unstructured waterways such as rivers and lakes in the wild, where the environment is complex and difficult to manage, and the golden rescue time is extremely short. Traditional drowning monitoring mainly relies on manual patrols, which suffers from high labor costs and delays due to visual fatigue. With the development of artificial intelligence technology, target detection algorithms based on computer vision are gradually becoming the mainstream method for water safety.

[0003] Chinese patent CN120108002A discloses a "drowning detection method based on human skeletal key points," which constructs an improved SSM-YOLO11-pose network to extract 17 skeletal key points of the human body and combines the ByteTrack tracking algorithm with a dual-stream spatiotemporal graph convolutional network (ST-GCN) to classify the swimmer's action sequence. This method avoids traditional physical threshold judgment and uses a deep learning model to achieve refined recognition of complex limb movements. However, when applying this method to wild rivers or waters with complex lighting, the following technical bottlenecks remain: 1. Skeletal point-based behavior recognition is overly dependent on feature integrity and has poor robustness in occluded scenarios. The core criterion of this technical approach relies entirely on the accurate regression of human skeletal key points. In turbid waters or when most of the drowning person's body is submerged (only the head and part of the arms are exposed), the detection model is prone to key point loss, drift, or false detection. If the input skeletal coordinate sequence contains noise, the backend ST-GCN behavior recognition model will make misjudgments due to incorrect data sources. In short, this method struggles to handle partially obscured drowning scenarios where limbs are not clearly visible. 2. It lacks environmental context semantics, making it difficult to distinguish highly dynamic non-drowning behaviors. While the ST-GCN model can capture the spatiotemporal motion features of limbs, its essence remains focused on the "geometric motion patterns of the skeleton." In summer water play scenarios, normal roughhousing, rapid swimming, and other behaviors exhibit high-frequency, large-amplitude skeletal movements highly similar to those of drowning struggles. This system cannot perceive high-dimensional semantic information such as surrounding interactions (e.g., companions / lifesaving equipment) and water flow, leading to a high false alarm rate in "false drowning" scenarios. 3. The multi-stage serial computing architecture struggles to achieve high-concurrency monitoring at the edge. This scheme employs a full-link serial processing mode of "detection + tracking + GCN recognition," requiring independent GCN inference calculations for each target in each frame. In the field and densely populated areas, as the number of targets increases, the computing load grows linearly, which can easily exceed the computing power limit of edge computing nodes, resulting in a decrease in system frame rate and an increase in latency, making it impossible to meet the real-time monitoring needs of multiple video streams.

[0004] Chinese patent CN120747854A discloses a "Real-time Drowning Detection Method Based on Multi-Module Integration." This solution integrates RFAConv (Dynamic Receptive Field Attention), C2PSA-SEAM (Channel Spatial Attention), and BiFPN (Bidirectional Feature Pyramid) modules into a lightweight YOLO model, and employs Depthwise Separable Convolution (DWConv) and model quantization techniques, significantly reducing computational complexity. It has made significant progress in solving water surface reflection interference and enabling real-time deployment on embedded edge devices (such as Raspberry Pi). However, when applied to wide rivers or complex human interaction scenarios in the wild, this technical approach still faces the following insurmountable technical bottlenecks: 1. Lack of deep behavioral semantic cognition, making it difficult to distinguish highly dynamic non-drowning behaviors. Although this technical approach enhances the extraction ability of human geometric features through attention mechanisms, its essence remains "closed-set detection based on visual texture." The model can only recognize the spatial relationship between "human body" and "water area," and cannot understand complex human interaction intentions from a cognitive perspective. In open water during summer, normal playful roughhousing (high-frequency body movements) or underwater breath-holding (long periods of invisibility) visually resemble the struggle of drowning. The lack of high-dimensional semantic understanding capabilities, similar to a large visual model (VLM), prevents the integration of contextual information such as interactions with surrounding companions for disambiguation, leading to a high false positive rate in densely populated areas. 2. Lightweight convolution and lossy downsampling result in the loss of features from small targets. To achieve extreme lightweighting (e.g., parameter compression to 1.8M), this scheme extensively employs depthwise separable convolution (DWConv) and stride downsampling strategies. While improving inference speed, for small targets with scarce texture information (e.g., a head barely visible above the water at the far end of a wide river), this physical pixel thinning leads to irreversible loss of fine-grained features. Even with subsequent feature fusion using BiFPN, it is difficult to recover the lost spatial information at the lower levels, easily resulting in missed detections of distant targets. 3. Lack of a "perception-cognition" collaborative computational scheduling mechanism. This solution employs a single edge-end inference model, with the cloud merely acting as a receiver of alarm data, failing to fully utilize the powerful computing capabilities of the cloud / server for "secondary semantic verification." In environments with weak network coverage or complex lighting conditions, relying solely on the limited computing power of the lightweight edge model makes it difficult to run complex cognitive algorithms to filter out long-tail false alarms. Furthermore, the lack of an event-driven (e.g., intensity of action) cloud-edge collaboration mechanism prevents the achievement of high-precision semantic-level monitoring while maintaining real-time performance.

[0005] Chinese patent CN117132924B discloses a "Drowning Detection Method Based on ATT-YOLO-V7 Network." This method constructs a three-stage architecture of "feature extraction-attention-prediction" in the detection network and designs an attention module containing multiple random fusion branches. The aim is to enhance the model's ability to extract features of swimmers' movement states (swimming, standing, drowning) in indoor swimming pool environments, thereby improving detection accuracy. However, when this approach is applied to unstructured scenarios such as wild rivers and natural water bodies, it still faces the following problems: 1. The discrimination logic based on "static classification" lacks dynamic behavioral semantic cognition. This approach treats "drowning" as a static classification label alongside "swimming" and "standing," and the model classifies based solely on the visual features of a single frame image. However, drowning is essentially a dynamic process of "temporal evolution" (such as from struggling to sinking). In open water, normal playful or diving actions are highly similar to drowning postures in a single frame. 1. Existing technologies lack analysis of "action continuity" and semantic understanding of "environmental context" (such as whether there is human interaction), making it easy to misjudge highly dynamic non-drowning behaviors as dangerous situations. 2. Multi-branch parallel architecture is difficult to adapt to efficient inference at the edge. This solution adopts a non-standard network architecture containing a large number of parallel basic units and random fusion branches. Although this "wide and shallow" multi-branch design theoretically increases feature richness, it severely disrupts the memory access optimization mechanism of modern NPUs (Neural Processing Units) for "regular convolutional structures," resulting in a significant increase in inference latency on edge computing devices (such as embedded AI modules), making it difficult to meet the needs of high frame rate real-time monitoring. 3. Lack of feature preservation mechanism for distant small targets. The field of view of monitoring rivers in the wild is wide, and the target of a person falling into the water is often extremely small. Existing technologies do not have a special lossless downsampling design for "small targets." During the convolution and downsampling process, the feature extraction module is prone to losing key pixel information of small targets, resulting in missed detections in long-distance monitoring. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a river drowning monitoring method based on lossless downsampling network and multimodal semantic disambiguation, thus solving the deficiencies of the prior art.

[0007] The objective of this invention is achieved through the following technical solution: a river drowning monitoring method based on lossless downsampling networks and multimodal semantic disambiguation, the method comprising:

[0008] Step 1: Collect heterogeneous data and perform enhanced preprocessing before inputting it into the target detection network deployed at the edge to detect geometric detection confidence. The geometric detection confidence is then used for initial target screening.

[0009] Step 2: For the suspected targets in the initial screening, calculate the dense light flow within a continuous time window, extract the light flow motion entropy based on the dense light flow field, and select key frame images and preceding and following time sequence feature maps based on the motion entropy.

[0010] Step 3: Upload the keyframe images and contextual information to the cloud. The cloud uses a large visual model to perform feature-semantic mapping based on a cross-model attention mechanism to generate textual semantic results.

[0011] Step 4: Perform Bayesian weighted fusion of the geometric detection confidence score detected by the target detection network and the textual semantic results output by the visual big model. If the final risk index after fusion exceeds the alarm threshold, generate alarm information containing the target location and on-site environment description, and trigger the terminal's audible and visual alarm.

[0012] The object detection network utilizes spatial-to-depth convolution for lossless downsampling. Let the input feature map be F, and its size be... Spatial to depthwise convolution generates four sub-feature maps through slicing operations. The four sub-feature maps are concatenated along the channel dimension to output the feature map. The dimension becomes While reducing the resolution, all pixel information is retained. At the same time, dynamic receptive field attention is combined to suppress background noise. S is the spatial size and C1 is the number of input channels.

[0013] Let the input feature be F, and the receptive field space feature weights be... for The final output includes the target's location coordinates, category confidence score, and water area segmentation mask. If a target is detected within the water area mask and its confidence score exceeds a preset threshold, it is marked as a preliminary suspected target. Indicates global average pooling. This represents the learnable weight matrix. This represents the temperature coefficient.

[0014] The step of extracting optical flow motion entropy from a dense optical flow field includes: calculating the motion vectors of pixels within the target region. and motion amplitude Discretize the motion amplitude into K intervals and statistically analyze the probability distribution of each interval. The time window length is set to T, and the motion characteristics of the initially screened suspected targets in consecutive video frames are continuously updated and analyzed in chronological order. That is, as the video frame sequence progresses, the current frame and its adjacent previous frames are repeatedly selected. If a frame is used as an analysis window to analyze initially screened potential targets, then the motion entropy within that time window is... Only when the entropy of motion Exceeding the set struggle threshold When an action within a given time window is deemed a high-risk abnormal behavior, the current keyframe image and its preceding and following temporal feature maps are captured. and These represent the optical flow displacement of a pixel in the horizontal and vertical directions between two adjacent frames, respectively.

[0015] Step three specifically includes the following:

[0016] Visual feature encoding and alignment: The uploaded keyframe image is segmented into fixed-size image blocks using a pre-trained visual encoder, and each image block is encoded into a high-dimensional visual feature vector sequence. The visual feature vectors are then mapped to the semantic embedding space of the large visual model through a multilayer perceptron or Q-Former projection layer, transforming them into a visual token sequence that the large visual model can understand.

[0017] Structured prompt word vector injection: The pre-defined structured prompt words are templated into a four-segment structure: task description segment, analysis dimension segment, constraint field segment, and output format segment. The task description segment clarifies the task objective of the current multimodal semantic reasoning; the analysis dimension segment limits the behavioral features and environmental interaction dimensions that the visual model needs to focus on during the reasoning process; the constraint field injects domain prior constraint information related to the drowning monitoring scenario; the output format segment further structures and limits the expression form of the visual model's generated results. Finally, the vectors are converted into text embedding vectors by a text segmenter and form a system prompt word token sequence. The visual token sequence generated by the visual encoder is used as a prefix and concatenated with the previous system prompt word sequence in the sequence dimension to construct a multimodal mixed input sequence, which can be represented as [Visual_Tokens, System_Prompt_Tokens], for subsequent cross-modal semantic reasoning and textual semantic generation. Here, Visual_Tokens is the visual token sequence, and System_Prompt_Tokens is the system prompt word token sequence.

[0018] Autoregressive semantic reasoning and generation: The mixed input sequence is input into the decoder of the visual big model. The visual big model uses a multi-head self-attention mechanism to dynamically calculate the attention weight between the current text instruction and the local features of the image when reasoning each text output token. When the visual big model generates an environmental interaction description, the attention is spread to the water ripple texture and floating object area around the human body. Finally, the visual big model generates textual semantic results that conform to the JSON format constraints word by word in an autoregressive manner.

[0019] The structured prompts are constructed using a template-based approach, consisting of four parts: task description, analysis dimensions, constraint fields, and output format. Physical constraints, physiological constraints, positional constraints, and format constraints are mapped to their respective fields.

[0020] The physical constraints play an attention-guiding role: by using fluid dynamics principles as prior knowledge, the self-attention mechanism of the large visual model is guided to give higher attention weight to the water surface texture region when processing image features, thereby achieving concentric diffusion and laminar flow in the feature-level region.

[0021] The physiological constraints serve as a logical filter: human kinematic instructions force the visual big model to execute the thought chain logic during the reasoning stage, comparing the relationship between the frequency of limb movements and the displacement of the center of mass, thereby filtering out false positive samples with high frequency and large displacement at the semantic level.

[0022] The location constraint plays a role in search and rescue positioning: the visual landmark association instruction is used as a spatial reference constraint to force the visual large model to identify highly recognizable salient location reference points in the image and generate a semantic location description of the target relative to these salient location reference points, thereby providing rescuers with intuitive and rapid guidance information and assisting in outputting pixel coordinates to achieve accurate positioning.

[0023] The format constraints serve a standardization function: by using the JSON syntax structure, the decoder of the large visual model is forced to follow the set syntax tree when generating text, ensuring that the output results are directly parsed by the edge end, and the output standard is JSON format, which includes risk level, behavior description and environmental attribution fields.

[0024] The method also includes constructing a multi-source hybrid and feature-separated dataset to train a large visual model, and sampling a combination of basic and advanced augmentation strategies during the training phase.

[0025] The construction of the multi-source hybrid and feature separation dataset includes:

[0026] Full-scene interaction data: Real-world scene data containing complete interactions between people and the aquatic environment is collected from publicly available video media based on an adaptive interval frame extraction algorithm. This data is used to train a large visual model to judge the behavioral logic of drowning and swimming.

[0027] Independent human feature data: Introducing publicly available human detection datasets to supplement human geometric features in complex poses, forcing the large visual model to focus on learning the structural information of the human body itself rather than relying on the background environment, thereby improving the ability of the large visual model to recognize people in complex backgrounds;

[0028] Independent water environment data: Public water semantic segmentation datasets are introduced to train the powerful segmentation branch of the visual large model, enhance the regression accuracy of the visual large model for water body boundaries under different water quality and lighting conditions, and ensure the accuracy of dynamic water mask generation.

[0029] The present invention has the following advantages:

[0030] 1. This method overcomes the bottleneck of traditional visual algorithms lacking behavioral semantic cognition, significantly reducing the false alarm rate of "false drowning." It introduces a Visual Large Model (VLM) as a cloud-based cognitive hub, constructing structured prompt word vectors containing action sequence and environmental interaction information, thus endowing the system with human-like deep semantic understanding capabilities. In highly dynamic interactive scenarios with dense crowds, this method can accurately filter non-critical abnormal behaviors, effectively solving the "crying wolf" false alarm problem of traditional solutions.

[0031] 2. This paper addresses the issue of feature loss for distant, small targets in a wide water environment, significantly improving the detection recall rate. An improved SPD-YOLO11-RiverGuard network (target detection network) was constructed, replacing the traditional stride convolution with an SPD-Conv (spatial-to-depth convolution) module, achieving lossless preservation of full-link features. Furthermore, by combining RFAConv (dynamic receptive field attention) and BiFPN (bidirectional feature pyramid), background noise interference from water surface shimmering and complex shorelines was effectively suppressed, significantly enhancing the system's perception robustness in complex hydrological environments in the field.

[0032] 3. It achieves extreme optimization of edge computing power and network bandwidth, balancing real-time monitoring with accuracy. A dynamic scheduling architecture based on heterogeneous computing power collaboration (QCS6490 + server / cloud API) is constructed, and a keyframe optimization mechanism based on "optical flow motion entropy" is proposed. High-performance cloud-based semantic inference and high-definition image transmission are only triggered when abnormally drastic target behavior is detected. This "edge-based initial screening - cloud-based final judgment" mechanism reduces invalid data transmission by over 90% while ensuring rapid danger warnings even in weak network environments.

[0033] 4. It possesses a flexible dual-mode inference architecture and non-contact, all-domain perception capabilities, enhancing the system's applicability. It allows for non-cooperative, 24 / 7 proactive monitoring without requiring the monitored object to wear any auxiliary devices. Simultaneously, the cloud-based cognitive service unit employs a dual-mode hybrid architecture of "local privacy mode (Ollama)" and "online augmented mode (cloud API)," meeting the stringent requirements for localized closed-loop data processing in privacy-sensitive areas such as home swimming pools and private garden water features. It also leverages the massive public cloud model to handle long-tail, complex scenarios in open water bodies (such as wild rivers), demonstrating significant engineering application value and promising prospects for widespread adoption. Attached Figure Description

[0034] Figure 1 This is a schematic flowchart of the method of the present invention;

[0035] Figure 2 This is a schematic diagram of the spatial-to-depth convolution structure of the present invention;

[0036] Figure 3 This is a schematic diagram illustrating the principle of multimodal semantic mapping based on structured constraint instructions in this invention.

[0037] Figure 4 This is a schematic diagram of the system structure. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application provided below with reference to the accompanying drawings is not intended to limit the scope of protection of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. The present invention will be further described below with reference to the accompanying drawings.

[0039] like Figure 1 As shown, one embodiment of the present invention relates to a river drowning monitoring method based on lossless downsampling networks and multimodal semantic disambiguation, which specifically includes the following:

[0040] S1. Heterogeneous Data Acquisition and Enhancement Processing:

[0041] Real-time video streams are acquired using heterogeneous computing terminals deployed at the edge of water areas; adaptive image tiling and water semantic constraints are applied to the video frames to generate a tensor to be detected that enhances the features of small targets and the texture of the water surface.

[0042] S2, edge-side full-volume non-destructive initial screening:

[0043] The preprocessed image is input into the SPD-YOLO11-RiverGuard object detection network deployed on the edge side; the object detection network uses SPD-Conv (spatial-to-depth convolution) for lossless downsampling, assuming the input feature map is... The size is SPD-Conv generates four sub-feature maps through slicing operations. S is the spatial dimension, C1 is the number of input channels, and its mapping formula is as follows:

[0044] ,

[0045] Furthermore, such as Figure 2 As shown, SPD-Conv achieves lossless downsampling by transferring information from the spatial dimension to the depth (channel) dimension, including the following:

[0046] Left side (input end): Displays a size of The original feature map. To visually illustrate the slicing logic, the image uses pixel blocks of four different textures (or grayscale / colors) (A, B, C, and D in the image) arranged alternately, representing the original image's features. The starting position of the sliding window pixels.

[0047] Intermediate (Slicing Process): This section demonstrates the slicing operation of the SPD (Space-to-Depth) layer. The original large image is broken down into four smaller images of varying sizes. The sub-feature maps are shown. As can be seen, each sub-feature map is re-aggregated from pixels with the same texture (or color) in the original image. This process completely preserves all pixel information of the original image without the information loss that occurs in traditional pooling operations.

[0048] The right side (output end) demonstrates the channel concatenation operation, which concatenates the four sub-feature maps along the channel dimension to output the feature map. The dimension becomes This method retains all pixel information while reducing resolution. Simultaneously, it incorporates RFAConv (Dynamic Receptive Field Attention) to suppress background noise.

[0049] Assume the input features are Sensing Wild Space Feature Weights The calculation formula is defined as follows:

[0050] ,

[0051] in, Indicates global average pooling. The weight matrix is ​​a learnable matrix. This represents the temperature coefficient. The final output includes the target's location coordinates, target confidence (bbox_conf), category confidence (cls_conf), and the water area segmentation mask. If a target is detected within the water area mask and its confidence exceeds a preset threshold, it is marked as a "preliminary suspected target." The geometric detection confidence (geo_conf) is obtained by weighted fusion of the target confidence (bbox_conf) output by the detection head and the human bounding box-water area mask overlap ratio (IoU) (e.g., geo_conf = bbox_conf * IoU), and is used for subsequent decision fusion.

[0052] The final output feature map has a 50% reduction in spatial resolution but a 4x increase in channel dimension, thus reducing computational cost while ensuring that pixel-level features of small objects at a distance in the river channel are not missed.

[0053] Furthermore, this invention makes the following structural improvements to the SPD-YOLO11-RiverGuard target detection network for river environments:

[0054] Backbone Reconstruction: SPD-Conv replaces the strided convolutional or pooling layers with a stride of 2 in the original network. SPD-Conv compresses the feature map size by stacking feature slices from the spatial dimension to the channel dimension, while retaining all pixel-level fine-grained information, thus solving the problem of feature loss for distant, small targets.

[0055] Specifically, for the input feature map X, its size is Where S is the spatial dimension and C1 is the number of input channels. SPD-Conv first performs a spatial slicing operation, decomposing the input feature map into four sub-feature maps according to the parity of pixel positions. The first sub-feature map extracts pixels in all even rows and even columns, the second sub-feature map extracts pixels in even rows and odd columns, the third sub-feature map extracts pixels in odd rows and even columns, and the fourth sub-feature map extracts pixels in odd rows and odd columns. The spatial dimension of each sub-feature map is... The number of channels remains at C1.

[0056] Subsequently, the above four sub-feature maps are concatenated along the channel dimension to obtain a size of [size missing]. The output feature map shows that although the spatial resolution is reduced by 50%, all pixel information of the original input is completely preserved in the channel dimension without any information loss. To control the expansion of the number of channels, a 1×1 convolutional layer is applied after the concatenation operation to compress the number of channels to the target output number. .

[0057] Taking the second layer of the backbone network as an example, the input feature map size is 320×320×64. After passing through SPD-Conv, it is first sliced ​​into four 160×160×64 sub-feature maps, then concatenated into a 160×160×256 feature map, and finally compressed to 160×160×128 through 1×1 convolution. The entire process reduces the spatial resolution by half, but retains all pixel-level fine-grained information, making it particularly suitable for detecting small targets at a distance.

[0058] Neck network (Neek) optimization: Introduce BiFPN (Bidirectional Feature Pyramid Network) structure to establish a bidirectional feature fusion pathway from bottom to top and from top to bottom, and assign learnable weights to feature maps of different scales to enhance the scale adaptability of the object detection network to targets at different distances.

[0059] Specifically, the backbone network outputs three feature maps at different scales, denoted as P3 (80×80×256), P4 (40×40×512), and P5 (20×20×1024). In the top-down path, P5 is first upsampled by a factor of 2 to obtain a 40×40×512 feature map, which is then weighted and fused with P4 to obtain the intermediate feature map. During fusion, learnable weights w1 and w2 are assigned to the two input features. These weights are processed using an exponential function and normalization to ensure that the sum of the weights is 1. The fused features are then processed through a 3×3 convolutional layer for feature extraction.

[0060] Similarly, P4_td is upsampled by a factor of 2 to obtain an 80×80×256 feature map, which is then weighted and fused with P3 to obtain the output feature P3_out. In the bottom-up path, P3_out is downsampled by a factor of 2 (using a 3×3 convolution with a stride of 2) to obtain a 40×40×256 feature map, which is then weighted and fused with P4_in and P4_td to obtain the output feature P4_out. Likewise, P4_out is downsampled and fused with P5_in and P5_td to obtain the output feature P5_out.

[0061] Through this bidirectional feature fusion mechanism, shallow features can not only obtain deep semantic information, but deep features can also obtain shallow detailed information, enabling the network to have stronger detection capabilities for targets of different scales. Compared with traditional FPN, BiFPN improves the average accuracy of small target detection tasks by about 3-5 percentage points.

[0062] Head Enhancement: An RFAConv (Receptive Field Attention Convolution) module is introduced in front of the head to dynamically adjust the receptive field shape and weight of the convolution kernel through a reparameterization mechanism, so as to suppress environmental noise interference caused by water surface shimmering and complex shoreline background.

[0063] Specifically, for the input feature map F (with dimensions H×W×C), global context information is first extracted through global average pooling, compressing the spatial information of each channel into a scalar value, resulting in a 1×1×C feature vector. This feature vector is then transformed through a fully connected layer and a learnable weight matrix to obtain preliminary attention weights. To control the smoothness of the attention distribution, a temperature coefficient τ is introduced to scale the weights, which are then normalized using the Softmax function to obtain the final spatial attention weights A. rf Its size is 1×1×C. Next, the input feature map F and the attention weights A are... rf Channel-by-channel multiplication yields a weighted feature map F_weighted. This operation essentially recalibrates the features of different channels, enhancing the feature responses of important channels and suppressing the features of less important channels. Finally, the weighted feature map F_weighted is input into subsequent convolutional layers for feature extraction.

[0064] In river drowning detection scenarios, RFAConv learns to focus strongly on the "human-water interface area" while paying less attention to "pure water background" or "fixed objects on the bank," thus effectively filtering background noise and improving target detection accuracy. Experiments show that the false alarm rate is reduced by approximately 15-20% after introducing the RFAConv module.

[0065] S3. Keyframe image selection based on motion entropy:

[0066] For initially screened suspected targets, the dense optical flow field within a continuous time window is calculated, and the optical flow motion entropy is extracted accordingly. First, the motion vectors of pixels within the target region are calculated. and motion amplitude Discretize the motion amplitude as Given several intervals, calculate the probability distribution for each interval. The time window length is set to T (corresponding to N consecutive frames), and the motion characteristics of the initially screened suspected targets in consecutive video frames are continuously updated and analyzed in chronological order. That is, as the video frame sequence progresses, the current frame and its adjacent previous frames are repeatedly selected. The frame is used as an analysis window to analyze the initial screening of suspected targets, and the motion entropy within this time window is... The calculation is as follows:

[0067] ,

[0068] Only when the motion entropy value Exceeding the set struggle threshold When the action within the time window is determined to be a high-risk abnormal behavior, the current high-resolution keyframe image and the preceding and following time sequence feature maps are captured.

[0069] Furthermore, the keyframe image optimization mechanism based on optical flow motion entropy specifically includes:

[0070] Optical flow field calculation: The dense optical flow between two consecutive image frames is calculated using the RAFT (Recursive All-Pair Transform) algorithm. First, features are extracted from the two input images to obtain a multi-scale feature pyramid. Then, the correspondence between the features of the two frames is established through correlation calculation, constructing a 4-dimensional correlation volume. Next, the optical flow estimation is updated iteratively in a recursive manner. In each iteration, the corresponding feature is queried from the correlation volume based on the current optical flow and then input into a gated recurrent unit (GRU) to update the optical flow. After multiple iterations, a refined optical flow field is obtained, where each similarity corresponds to a two-dimensional motion vector. , and These represent the optical flow displacement (in pixels) of a pixel along the horizontal (x-axis) and vertical (y-axis) directions between two adjacent frames.

[0071] In river drowning monitoring scenarios, optical flow fields can clearly depict the movement patterns of a drowning person's limbs. During normal swimming, the optical flow field exhibits a regular, periodic pattern, such as the symmetrical optical flow generated by alternating arm strokes. However, during drowning struggles, the optical flow field exhibits a disordered, high-frequency, and large-amplitude chaotic pattern, with ineffective flailing of limbs resulting in erratic movement directions.

[0072] Definition and calculation of motion entropy: First, for suspected targets detected at the edge, extract the optical flow sub-region within its detection box. Assuming the detection box coordinates are (x, y, w, h), the optical flow of the corresponding region is cropped from the full-image optical flow field to obtain an optical flow sub-map of size h×w×2.

[0073] Next, the motion amplitude of each pixel, i.e., the magnitude of the optical flow vector, is calculated. For pixel (i, j), the formula for calculating its motion amplitude M(i,j) is: sqrt represents the square root, thus obtaining a motion amplitude diagram of h×w.

[0074] Then, the motion amplitude is discretized. First, the maximum value M_max and the minimum value M_min of the entire amplitude map are counted. Then, the interval [M_min, M_max] is evenly divided into K sub-intervals (K=10 in this embodiment). The number of pixels in each sub-interval is counted to obtain the histogram distribution hist[i], i=1,2,...,K.

[0075] Next, the histogram is normalized into a probability distribution. The probability for the i-th interval is... , where h × w is the total number of pixels, ensuring that the sum of all probabilities is 1.

[0076] Finally, the motion entropy is calculated according to the definition of Shannon entropy. The range of values ​​for the motion entropy H is: .when When the value is close to 0, it indicates that the motion amplitude is highly concentrated in a certain range, and the motion pattern is simple. near When the amplitude of motion is evenly distributed across all intervals, the motion pattern is chaotic and disordered.

[0077] For normal swimmers, their limb movements are regular and coordinated, and the optical flow amplitude generated by the arm and leg strokes is relatively uniform, with motion entropy values ​​typically between 1.5 and 2.5. For stationary or slowly moving targets, the optical flow amplitude is close to zero and highly concentrated, with motion entropy values ​​typically less than 1.0. Therefore, this invention sets the motion entropy threshold δ to 3.5. When the average motion entropy of the detected target exceeds 3.5 within a continuous time window, it is judged as high-risk abnormal behavior, triggering cloud-based VLM semantic inference. To avoid misjudgments caused by single-frame noise, a sliding window smoothing mechanism is adopted, with the window size set to 10 frames (approximately 0.5 seconds). An alarm is only triggered when the average motion entropy within the window continuously exceeds the threshold. In addition, the variance of the motion entropy within the window is calculated to determine the degree of motion fluctuation. Drowning struggles are usually accompanied by high entropy values ​​and high variance, while normal swimming, although occasionally exhibiting high instantaneous entropy values, has relatively small variance. By comprehensively considering the average entropy value and variance, the accuracy of the judgment can be further improved.

[0078] For drowning victims struggling, their limbs exhibit disordered, high-frequency, and large-amplitude thrashing movements, their heads repeatedly submerge and surface, and their body posture is unstable, resulting in a highly discrete distribution of optical flow amplitude. Some areas move violently (such as the arms thrashing in the water), while others move slowly (such as the sinking torso). The motion entropy value is usually between 3.5 and 4.5.

[0079] Keyframe Image Optimization Strategy: When a high-risk target is detected, the most representative keyframes need to be selected from consecutive video frames and uploaded to the cloud for VLM inference. Due to limited network bandwidth and cloud computing power, not all frames can be uploaded; therefore, a reasonable keyframe optimization strategy needs to be designed. Thus, this invention employs a keyframe selection method based on motion entropy scoring. For each frame within the sliding window, its comprehensive score is calculated. ,in The average motion entropy within the window. Let ε be the variance of the motion entropy, and let ε be a small smoothing term.

[0080] Among them, the higher the average entropy value, the more chaotic the motion; the larger the variance, the more intense the motion fluctuation. Combining the two can more accurately identify the critical moments of drowning struggle. The scores of all frames within the sliding window are calculated, and the top-5 frames with the highest scores are selected as keyframes. These, along with the temporal context of the two frames before and after them, are uploaded to the cloud, for a total of 5×5=25 frames.

[0081] This strategy ensures the representativeness of uploaded frames while controlling the amount of data transmitted, thus reducing network bandwidth consumption and cloud computing burden while maintaining the accuracy of semantic analysis. Compared to full upload, this strategy can reduce data transmission by more than 90% while maintaining a similar detection accuracy.

[0082] S4, Cloud-Edge Collaboration and Multimodal Semantic Mapping:

[0083] like Figure 3 As shown, the edge side uploads keyframe images and preceding and following temporal feature maps to the cloud, and simultaneously uploads contextual information (such as timestamps, detection confidence, target location, environmental labels, etc.) to the cloud-based cognitive service center to assist in the semantic reasoning of the Visual Model (VLM). The cloud uses the Visual Model (VLM) to perform feature-semantic mapping based on a cross-modal attention mechanism. The specific process includes the following three sub-steps:

[0084] Visual Encoding & Alignment: First, a pre-trained visual encoder is used to segment the uploaded high-resolution keyframe image into fixed-size image patches, and each patch is encoded into a sequence of high-dimensional visual feature vectors. Then, a multilayer perceptron (MLP) or Q-Former projection layer (which performs cross-modal alignment through learnable query vectors, resulting in stronger semantic fusion capabilities) maps these visual feature vectors to the text embedding space of the Visual Model (VLM), transforming them into a sequence of "visual tokens" that the VLM can understand (visual tokens are serialized vector units that map visual features to the embedding space of a large visual model, used for subsequent semantic inference).

[0085] Prompt Injection: Pre-defined structured prompts are presented in a templated four-segment structure: task description, analysis dimensions, constraint fields, and output format. The task description clarifies the current multimodal semantic reasoning objective; the analysis dimensions define the behavioral features and environmental interactions that the model needs to focus on during reasoning; the constraint fields inject domain-specific prior constraints related to the drowning monitoring scenario (including physical, physiological, positional, and output format constraints); and the output format further structures and defines the expression of the model's generated results—finally, these are converted into text embeddings via a tokenizer, forming a system prompt token sequence (System_Prompt_Tokens). The visual token sequence (Visual_Tokens) generated by the visual encoder is used as a prefix and concatenated with the system prompt word token sequence mentioned above in the sequence dimension to construct a multimodal mixed input sequence, which can be represented as [Visual_Tokens, System_Prompt_Tokens], and is used for subsequent cross-modal semantic reasoning and textual semantic generation.

[0086] Autoregressive Reasoning: The mixed input sequence is fed into the decoder of the Visual Large Model (VLM). The model utilizes a multi-head self-attention mechanism to dynamically calculate the attention weights between the current text instruction and the local region features of the image when reasoning for each text output token.

[0087] When the model generates descriptions of "environmental interactions," its attention expands to the water ripple texture and floating object areas surrounding the human body. Ultimately, the model uses an autoregressive approach to generate textual semantic results (containing fields such as risk_level and action_desc) that conform to JSON format constraints, thereby achieving a precise mapping from pixel-level visual information to high-level semantic logic.

[0088] Furthermore, by injecting domain knowledge constraints, the problem of severe "illusion" in general-purpose visual models under specific scenarios is solved: the prompt word vector adopts a template-based assembly method, consisting of four parts: task description, analysis dimension, constraint field, and output format. Physical constraints, physiological constraints, positional constraints, and format constraints are mapped to their corresponding fields. This construction method is consistent with the design principle of the structured prompt words discussed later, ensuring that the prompt words are semantically and format-controllable and parsable. The construction of the structured prompt word vector (Prompt) includes the following constraint instructions:

[0089] The attention-guiding role of physical constraints: Fluid dynamics principles serve as prior knowledge, guiding the VLM's self-attention mechanism to assign higher attention weights to water surface texture areas (such as ripples and foam) when processing image features, thereby distinguishing between concentric circle diffusion (falling water features) and laminar flow (normal water flow) at the feature level.

[0090] The logical filtering effect of physiological constraints: Human kinematic instructions force the visual big model to execute the chain-of-thought logic during the reasoning stage, comparing the relationship between the frequency of limb movements and the displacement of the center of mass, thereby filtering out false positive samples of "high frequency and large displacement" (swimming) at the semantic level.

[0091] The role of location constraints in search and rescue: The "visual landmark association" command serves as a spatial reference constraint, forcing the model to identify highly recognizable salient location reference points in the image (such as buoys, bridge piers, and warning signs) and generate semantic location descriptions of the target relative to these reference points (such as "located about 3 meters to the left of the red buoy"). This provides rescuers with intuitive and rapid guidance information and assists the system in achieving accurate positioning through pixel coordinates output.

[0092] The standardization effect of format constraints: By using JSON syntax structure to constrain the model's decoder, it is forced to follow a specific syntax tree when generating text, ensuring that the output can be directly parsed by edge programs. "The output standard is JSON format results, which must include the risk_level, action_desc, and environment_context fields."

[0093] S5. Decision Fusion and Hierarchical Early Warning:

[0094] The geometric detection confidence score geo_conf (overlap ratio between river instance segmentation and human target detection box) detected by the target detection network on the edge side is combined with the semantic risk level risk_level and its confidence score vlm_conf of the textual semantic result output by the cloud VLM. If the final risk index after fusion exceeds the alarm threshold, an alarm message containing the target location and on-site environment description is generated, and the terminal's audible and visual alarm is triggered.

[0095] Furthermore, the method also includes constructing a multi-source hybrid and feature-separated dataset to train a large visual model, wherein constructing the multi-source hybrid and feature-separated dataset includes:

[0096] Full-scene interaction data: Real-world scene data containing complete interactions between people and the aquatic environment is collected from publicly available video media based on an adaptive interval frame extraction algorithm. This data is used to train a large visual model to judge the behavioral logic of drowning and swimming.

[0097] Independent human feature data: Introduce public human detection datasets (pure human samples) to supplement human geometric features in complex poses, forcing the large visual model to focus on learning the structural information of the human body itself rather than relying on the background environment, thereby improving the ability of the large visual model to recognize people in complex backgrounds.

[0098] Independent water environment data: An open water semantic segmentation dataset (pure water surface samples) is introduced to train the powerful segmentation branch of the visual large model, enhance the regression accuracy of the visual large model for water body boundaries under different water quality and lighting conditions, and ensure the accuracy of dynamic water mask generation.

[0099] Furthermore, for unstructured environments in the wild, a combined strategy of basic augmentation and advanced augmentation is adopted during the training phase of the large visual model, specifically including the following:

[0100] Basic luminosity and geometric enhancement: Random HSV color space perturbation and affine transformation are applied to the training images to simulate the changes in river lighting throughout the day and different monitoring perspectives, thereby reducing the sensitivity of the large visual model to color and angle.

[0101] Advanced Hybrid Augmentation: The Mosaic augmentation strategy is adopted, randomly selecting 4 images for stitching and rearranging, which significantly increases the number of small objects in a single batch of training; the MixUp augmentation strategy is adopted, which linearly weights and stacks the images and their labels to smooth the model decision boundary and prevent overfitting on a small number of drowning postures.

[0102] Enhanced environmental adaptability: Gaussian blur and motion blur processing with random kernel size are applied to the image to simulate the motion blur produced by low-end monitoring equipment when capturing high-speed struggling targets, ensuring the stability of feature extraction under blurred image quality.

[0103] like Figure 4 As shown, another embodiment of the present invention relates to a river drowning monitoring system based on lossless downsampling network and multimodal semantic disambiguation, which includes: an edge flow control and acquisition unit and an edge intelligent inference unit on the edge side, and a cloud cognitive service unit;

[0104] Edge Flow Control and Acquisition Unit: Employs a general-purpose embedded microcontroller to control the high-definition camera for video stream acquisition, perform image segmentation and grayscale conversion preprocessing, and connect to external environmental sensor data reading and control of audible and visual alarms;

[0105] Edge intelligent inference unit: used to deploy target detection network, detect geometric detection confidence through target detection network, perform initial target screening based on geometric detection confidence, and calculate dense light flow within a continuous time window for suspected targets in the initial screening, extract optical flow motion entropy based on dense light flow field, and select key frame images and preceding and following temporal feature maps based on motion entropy;

[0106] The cloud-based cognitive service unit receives keyframe images and their associated temporal feature maps uploaded by the edge intelligent inference unit. The cloud utilizes a large visual model to perform feature-semantic mapping based on a cross-model attention mechanism to generate textual semantic results. It also performs Bayesian weighted fusion of the geometric detection confidence scores detected by the object detection network with the textual semantic results output by the large visual model. If the final risk index after fusion exceeds the alarm threshold, it generates alarm information containing the target location and description of the scene environment, and triggers the terminal's audible and visual alarm. When the final risk index exceeds the alarm threshold, it automatically triggers the email notification service to send alarm emails to preset management personnel.

[0107] Furthermore, the email content included:

[0108] Keyframe images for drowning detection (standard target bounding boxes), semantic summaries of visual large models (e.g., the target is struggling violently in the water, and its head is submerged multiple times), descriptions of drowning person characteristics (e.g., wearing a red shirt and dark shorts), target location information (e.g., located in the shallow area in the lower left corner of the image, pixel coordinates [x,y,w,h]), key environmental features of the surrounding area (focusing on highly recognizable representative landmarks or hydrological conditions that help rescuers quickly locate the target, such as being about 5 meters away from the red buoy, located in a backwater area, and surrounded by significant white floating objects), the ratio of human-water overlap (IoU value), and confidence scores.

[0109] Under normal circumstances, the system only runs the pixel difference algorithm of the edge flow control unit for motion detection. The edge intelligent inference unit is woken up to load the target detection network only when a dynamic target is detected entering the preset water area. The communication module is woken up to establish a cloud connection only when the edge intelligent inference unit determines that the motion entropy is abnormal, thereby achieving the optimal energy efficiency solution in the field independent deployment environment.

[0110] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and improvements, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for monitoring drowning in rivers based on lossless downsampling networks and multimodal semantic disambiguation, characterized in that: The method includes: Step 1: Collect heterogeneous data and perform enhanced preprocessing before inputting it into the target detection network deployed at the edge to detect geometric detection confidence. The geometric detection confidence is then used for initial target screening. Step 2: For the suspected targets in the initial screening, calculate the dense light flow within a continuous time window, extract the light flow motion entropy based on the dense light flow field, and select key frame images and preceding and following time sequence feature maps based on the motion entropy. Step 3: Upload the keyframe images and contextual information to the cloud. The cloud uses a large visual model to perform feature-semantic mapping based on a cross-model attention mechanism to generate textual semantic results. Step 4: Perform Bayesian weighted fusion between the geometric detection confidence score detected by the SPD-YOLO11-RiverGuard target detection network and the textual semantic results output by the visual large model. If the final risk index after fusion exceeds the alarm threshold, generate alarm information containing the target location and on-site environment description, and trigger the terminal's audible and visual alarm. The object detection network utilizes spatial-to-depth convolution for lossless downsampling. Let the input feature map be F, and its size be... Spatial to depthwise convolution generates four sub-feature maps through slicing operations. The four sub-feature maps are concatenated along the channel dimension to output the feature map. The dimension becomes While reducing the resolution, all pixel information is retained. At the same time, dynamic receptive field attention is combined to suppress background noise. S is the spatial size and C1 is the number of input channels. Let the input feature be F, and the receptive field space feature weights be... for The final output includes the target's location coordinates, category confidence score, and water area segmentation mask. If a target is detected within the water area mask and its confidence score exceeds a preset threshold, it is marked as a preliminary suspected target. Indicates global average pooling. This represents the learnable weight matrix. Indicates the temperature coefficient; The step of extracting optical flow motion entropy from a dense optical flow field includes: calculating the motion vectors of pixels within the target region. and motion amplitude Discretize the motion amplitude into K intervals and statistically analyze the probability distribution of each interval. The time window length is set to T, and the motion characteristics of the initially screened suspected targets in consecutive video frames are continuously updated and analyzed in chronological order. That is, as the video frame sequence progresses, the current frame and its adjacent previous frames are repeatedly selected. If a frame is used as an analysis window to analyze initially screened potential targets, then the motion entropy within that time window is... Only when the entropy of motion Exceeding the set struggle threshold When an action within a given time window is deemed a high-risk abnormal behavior, the current keyframe image and its preceding and following temporal feature maps are captured. and These represent the optical flow displacement of a pixel in the horizontal and vertical directions between two adjacent frames, respectively.

2. The method for river drowning monitoring based on lossless downsampling network and multimodal semantic disambiguation according to claim 1, characterized in that: Step three specifically includes the following: Visual feature encoding and alignment: The uploaded keyframe image is segmented into fixed-size image blocks using a pre-trained visual encoder, and each image block is encoded into a high-dimensional visual feature vector sequence. The visual feature vectors are then mapped to the semantic embedding space of the large visual model through a multilayer perceptron or Q-Former projection layer, transforming them into a visual token sequence that the large visual model can understand. Structured prompt word vector injection: The pre-defined structured prompt words are templated into a four-segment structure: task description segment, analysis dimension segment, constraint field segment, and output format segment. The task description segment clarifies the task objective of the current multimodal semantic reasoning; the analysis dimension segment limits the behavioral features and environmental interaction dimensions that the visual model needs to focus on during the reasoning process; the constraint field injects domain prior constraint information related to the drowning monitoring scenario; the output format segment further structures and limits the expression form of the visual model's generated results. Finally, the vectors are converted into text embedding vectors by a text segmenter and form a system prompt word token sequence. The visual token sequence generated by the visual encoder is used as a prefix and concatenated with the previous system prompt word sequence in the sequence dimension to construct a multimodal mixed input sequence, which can be represented as [Visual_Tokens, System_Prompt_Tokens], for subsequent cross-modal semantic reasoning and textual semantic generation. Here, Visual_Tokens is the visual token sequence, and System_Prompt_Tokens is the system prompt word token sequence. Autoregressive semantic reasoning and generation: The mixed input sequence is input into the decoder of the visual big model. The visual big model uses a multi-head self-attention mechanism to dynamically calculate the attention weight between the current text instruction and the local features of the image when reasoning each text output token. When the visual big model generates an environmental interaction description, the attention is spread to the water ripple texture and floating object area around the human body. Finally, the visual big model generates textual semantic results that conform to the JSON format constraints word by word in an autoregressive manner.

3. The method for river drowning monitoring based on lossless downsampling network and multimodal semantic disambiguation according to claim 2, characterized in that: The structured prompts are constructed using a template-based approach, consisting of four parts: task description, analysis dimensions, constraint fields, and output format. Physical constraints, physiological constraints, positional constraints, and format constraints are mapped to their respective fields.

4. The method for river drowning monitoring based on lossless downsampling network and multimodal semantic disambiguation according to claim 3, characterized in that: The physical constraints play an attention-guiding role: by using fluid dynamics principles as prior knowledge, the self-attention mechanism of the large visual model is guided to give higher attention weight to the water surface texture region when processing image features, thereby achieving concentric diffusion and laminar flow in the feature-level region. The physiological constraints serve as a logical filter: human kinematic instructions force the visual big model to execute the thought chain logic during the reasoning stage, comparing the relationship between the frequency of limb movements and the displacement of the center of mass, thereby filtering out false positive samples with high frequency and large displacement at the semantic level. The location constraint plays a role in search and rescue positioning: the visual landmark association instruction is used as a spatial reference constraint to force the visual large model to identify highly recognizable salient location reference points in the image and generate a semantic location description of the target relative to these salient location reference points, thereby providing rescuers with intuitive and rapid guidance information and assisting in outputting pixel coordinates to achieve accurate positioning. The format constraints serve a standardization function: by using the JSON syntax structure, the decoder of the large visual model is forced to follow the set syntax tree when generating text, ensuring that the output results are directly parsed by the edge end, and the output standard is JSON format, which includes risk level, behavior description and environmental attribution fields.

5. The method for river drowning monitoring based on lossless downsampling network and multimodal semantic disambiguation according to claim 1, characterized in that: The method also includes constructing a multi-source hybrid and feature-separated dataset to train a large visual model, and sampling a combination of basic and advanced augmentation strategies during the training phase.

6. The method for river drowning monitoring based on lossless downsampling network and multimodal semantic disambiguation according to claim 5, characterized in that: The construction of the multi-source hybrid and feature separation dataset includes: Full-scene interaction data: Real-world scene data containing complete interactions between people and the aquatic environment is collected from publicly available video media based on an adaptive interval frame extraction algorithm. This data is used to train a large visual model to judge the behavioral logic of drowning and swimming. Independent human feature data: Introducing publicly available human detection datasets to supplement human geometric features in complex poses, forcing the large visual model to focus on learning the structural information of the human body itself rather than relying on the background environment, thereby improving the ability of the large visual model to recognize people in complex backgrounds; Independent water environment data: Public water semantic segmentation datasets are introduced to train the powerful segmentation branch of the visual large model, enhance the regression accuracy of the visual large model for water body boundaries under different water quality and lighting conditions, and ensure the accuracy of dynamic water mask generation.

Images