A drowning visual recognition method integrating infrared temperature difference features

Abstract

The present invention discloses a method for visually identifying drowning by integrating infrared temperature difference features, comprising the following steps: S1, image acquisition and feature extraction; S2, edge feature map extraction; S3, shape feature map extraction; S4, color feature map extraction; S5, feature map fusion and segmentation; S6, drowning person instance identification and alarm triggering. The present invention combines infrared thermal imaging with visible light images and utilizes multi-feature fusion technology to improve the accuracy and real-time performance of drowning detection, effectively identifying drowning persons in low-light environments in particular; the system rapidly transmits drowning person information through intelligent alarms and precise positioning, shortening rescue response time and improving rescue efficiency; in addition, it uses advanced neuron impulse response networks and convolutional neural networks to more accurately extract features of drowning persons in complex backgrounds, making it applicable to a variety of water environments and providing innovative solutions for drowning accident prevention.

Description

Technical Field

[0001] The present invention relates to the field of drowning monitoring systems, and in particular to a drowning visual recognition method integrating infrared temperature difference features. Background Art

[0002] In recent years, drowning incidents have occurred frequently in waters, especially in open waters, beaches, lakes and other places. Drowning accidents have become a major issue affecting people's life safety. Traditional drowning monitoring methods mostly rely on manual patrols, shore surveillance cameras or fixed water surface monitoring systems. These traditional methods have obvious defects in many aspects: traditional monitoring systems often require manual intervention or manual analysis to detect abnormal situations, and the response speed is slow, resulting in poor rescue timeliness in drowning incidents; fixed surveillance cameras or manual patrols cannot cover large areas of water, especially in vast lakes, sea areas and other areas, there are blind spots or it is impossible to monitor all areas in real time; due to the complex environment, factors such as fluctuations and floating objects on the water surface can easily lead to false alarms, increasing the difficulty of processing monitoring results.

[0003] To address these issues, drones, as a flexible and efficient monitoring tool, are increasingly being used in drowning monitoring. Drones can cruise quickly, collecting real-time image and video data within the water, significantly improving monitoring coverage and response speed. Especially with the support of multi-sensor fusion technology, drones can provide more accurate data support in diverse environmental conditions.

[0004] Currently, drone-based drowning monitoring mostly relies on visible light images, infrared thermal imaging, or other single sensor monitoring. However, single sensors have certain limitations in complex environments:

[0005] While visible light images are clear, their effectiveness is significantly reduced in low light conditions or with strong light reflections, particularly at night or in low-light environments, where they are unable to effectively capture target information. Infrared thermal imaging can be effective in low-light or nighttime conditions, but its low resolution makes it difficult to accurately identify smaller targets or complex backgrounds on the water surface. It is also susceptible to interference from factors such as ambient temperature and wind and waves, resulting in reduced recognition accuracy. Therefore, this paper combines the maneuverability of drones with the advantages of multiple sensor images to propose a new method for efficiently detecting drowning victims. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention provides a drowning visual recognition method that integrates infrared temperature difference features, which solves the problem that the existing technical solutions cannot achieve efficient, accurate and real-time drowning detection in complex water environments.

[0007] To achieve the above objectives, the following technical solutions can be used:

[0008] The present invention provides a drowning visual recognition method integrating infrared temperature difference features, comprising the following steps:

[0009] S1. Image acquisition and feature extraction: When the drone is cruising, it is equipped with a visible light camera and infrared thermal imaging equipment to collect visible light images and infrared thermal images within the cruising range in real time;

[0010] S2. Edge feature map extraction: Preprocess the infrared thermal image and construct a neural impulse response network to extract edge information with significant changes in thermal radiation, highlighting areas with significant changes in thermal intensity, thereby helping to detect the outline of the drowning person and important structures;

[0011] S3. Shape feature map extraction: By performing multi-scale rectangular feature extraction and dynamic weighted screening on the input image, a strong classifier is constructed by combining feature combinations to further analyze the shape of objects in the image and effectively identify the body shape of the drowning person;

[0012] S4, color feature map extraction: multi-layer encoding of visible light images through convolutional neural networks to extract color and texture features, enhancing target detection in low-light environments;

[0013] S5. Feature map fusion and segmentation: The edge, shape, and color feature maps extracted in steps S2-S4 are fed into the deep learning model to generate a unified instance feature map. Graph segmentation technology is used to accurately extract the outline of the drowning person, and the heat map and visible light information are combined to lock the target.

[0014] S6. Drowning person instance identification and alarm triggering: Real-time alarm is issued for detected drowning person instances, and the alarm mechanism is triggered by the drowning person's location and status, and the drowning person's location and detection confidence are transmitted to the ground control station.

[0015] As a further solution of the present invention, the specific method steps of step S2 are as follows:

[0016] S21, performing normalization processing on the input infrared thermal image, scaling the pixel values ​​in the image to the range of [0, 1], and eliminating the difference by adjusting the ratio of the minimum and maximum pixel values;

[0017] S22. Construct a neuron impulse response network and extract edge features using changes in neuron membrane potential and neighborhood coupling. The dynamic equation of the neuron is:

[0018]

[0019] Among them, v(x,y,t) represents the membrane potential of the neuron at position (x,y) at time step t, v thresholdis the threshold of the neuron, c(x,y) is the coupling strength of the neuron, I'(x,y) is the intensity value of the input image, N(x,y) is the neighborhood pixel around the position (x,y), and λ is the attenuation coefficient;

[0020] S23, when the neuron's membrane potential v(x,y,t) reaches the threshold v threshold When , the neuron will generate an impulse response, indicating that the feature of the position is activated. The pulse generation rule is:

[0021]

[0022] Among them, δ(x, y, t) represents the impulse response. If the membrane potential exceeds the threshold, a pulse signal is output;

[0023] S24. Use the impulse response graph to simulate the coupling between neurons. By calculating the cumulative output of the impulse, the area with drastic changes in thermal intensity in the image is highlighted, thereby generating an edge response graph E(x,y). Where T is the total number of time steps of pulse propagation, E(x,y) represents the edge strength of the image at position (x,y);

[0024] S25. De-noise and smooth the preliminary edge information image obtained by accumulating the impulse responses to generate a high-quality edge feature map F(x, y). The specific steps are as follows:

[0025] By setting the edge strength threshold T edge , remove the response intensity below the threshold T edge The pulse points are then smoothed to remove isolated pixels to ensure edge continuity. The edge feature map F(x,y) is expressed as:

[0026] The final output edge feature map Fedge(x,y) clearly shows the area where the thermal intensity changes dramatically in the image, which is used for subsequent drowning person detection and target recognition.

[0027] As a further solution of the present invention, the specific method steps of step S3 are as follows:

[0028] S31. Standardize and denoise the infrared thermal image to ensure input stability. The specific steps are as follows:

[0029] S311, normalize the pixel values ​​to the interval [0,1], the normalized image of the original image I(x,y) is

[0030] Among them, min(I) and max(I) are the minimum and maximum pixel values ​​in the image respectively;

[0031] S312, then apply Gaussian filtering to reduce image noise, and the smoothed image I smooth (x,y) is: Where G(x,y,σ) is a filter with a standard deviation of σ;

[0032] S32. By calculating the brightness difference of rectangular areas of different scales and directions, extract local brightness features and construct feature vectors. The specific steps are as follows:

[0033] S321, define multiple rectangular areas R1, R2, ..., R near the pixel point (x, y) m Covering different sizes, directions and positions;

[0034] S322, calculate the rectangular area pair (R i , R j )’s brightness difference:

[0035] where f ij (x,y) represents the position

[0036] The eigenvalue of (x,y) is obtained by the rectangular area pair (R i , R j ) is calculated by the brightness difference

[0037] The result,describes the local brightness differences of the image;

[0038] S323, according to all rectangular eigenvalues ​​f ij , generate a feature vector F(x,y) at each pixel position, expressed as: F(x,y)=[f 11 (x,y),f 12 (x,y),…,f mn (x,y)];

[0039] S33. Introduce a dynamic weighting mechanism to assign different weights to each rectangular feature. By iteratively optimizing the feature weights, the detection accuracy is improved. The specific steps are as follows:

[0040] S331. Initialize the weights of all features to be the same. For the kth feature, the initial weight is:

[0041]

[0042] S332, in each round of iteration, select the feature combination with the minimum error and update its weight. In the tth round, select feature f ij As the main feature, its classification error is ε ij , then the updated weight is:

[0043] in, is the adjustment coefficient, h ij Based on the feature f ij The weak classifier output, l() represents the indicator function, which is used to determine whether the classification is correct;

[0044] S333. After updating the weights of all features, perform normalization:

[0045] S34. Combining weighted features into weak classifiers and generating strong classifiers through weighted integration. The specific steps are as follows:

[0046] S341, for each feature f ij Construct weak classifier h ij (x,y), the output is 1 or -1, indicating whether the position matches the target shape. The weak classifier is defined as:

[0047] Among them, θ ij is the feature f ij The classification threshold;

[0048] S342. Combine all weak classifiers into a strong classifier H(x,y) by weighted summation, which is defined as: Among them, α ij is the weight of the weak classifier, sign() represents the sign function, output 1 means the shape feature is detected, output -1 means it is not detected;

[0049] S35. Use the strong classifier H(x,y) to detect each pixel and output the shape feature map F shape (x,y), the specific steps are:

[0050] S351. First, use the strong classifier H(x,y) to determine whether each pixel (x,y) belongs to the target shape area:

[0051] S352. Then perform morphological processing on the output shape feature map, including dilation and erosion operations, to eliminate isolated points and smooth boundaries to obtain the final target detection result.

[0052] As a further solution of the present invention, the specific method steps of step S4 are as follows:

[0053] S41, normalizing and border-expanding the input visible light image to ensure image consistency and integrity;

[0054] S42. Extract the color and texture features of the image layer by layer through convolution and pooling operations to generate a multi-level feature representation. The specific encoding process is as follows:

[0055] S421, each convolution layer uses convolution kernel K to extract local features of the image, and the convolution kernel size of the lth layer is k l ×k l , the convolution kernel weight is W l , bias is b l , the output of the l-th layer convolution is:

[0056] Among them, σ is a nonlinear activation function, I in (x,y) is the input feature map;

[0057] S422, apply pooling operation after convolution to reduce the spatial dimension, and set the pooling window size to p×

[0058] p, the feature map after pooling is: P l (x,y)=max u,v∈p×p F l (x+u,y+v);

[0059] S423, through multiple convolution and pooling operations, gradually build a multi-layer encoding structure to generate image feature maps of different scales and depths;

[0060] S43. Enhance the context information of color feature extraction through skip connection and context fusion. The specific steps are as follows:

[0061] S431, introduce a skip connection between the encoding layer and the decoding layer to retain the shallow feature information. The output of the skip connection is: C l (x,y)=P l (x,y)+D l (x,y), where P l (x,y) and D l (x, y) represents the feature map of the lth layer in the encoding stage and the decoding stage respectively. The skip connection ensures that the high-resolution color information is not lost;

[0062] S432. Concatenate the feature maps of different layers in the encoding stage to form a context feature representation:

[0063] C context (x,y)=concat{C1(x,y),C2(x,y),...,C l (x,y)};

[0064] S44, gradually upsampling and convolution to restore image resolution and extract complete color features. The specific steps are as follows:

[0065] S441, each layer restores the low-resolution feature map to high resolution by upsampling, set upsampling

[0066] The factor is s, and the upsampled output of the lth layer is:

[0067] S442. After each upsampling, convolution is applied to extract details. The output of the decoded convolution layer is:

[0068] Among them, W d and b d are the weights and biases of the convolution kernels in the decoding layer, k d is the size of the decoding convolution kernel, σ is the activation function;

[0069] S45. Generate color feature map F by upsampling and decoding convolution color (x,y), and use the softmax function to enhance the color contrast information, After generating the feature map, the edges are smoothed through a refinement operation to preserve the color information of the target object in preparation for multimodal fusion.

[0070] As a further solution of the present invention, the specific method steps of step S5 are as follows:

[0071] S51. Linear normalize the edge, shape, and color feature maps separately to ensure that the data range is consistent and avoid weight deviation due to numerical differences. The specific formula is as follows:

[0072]

[0073] Among them, F' edge (x,y), F' shape (x,y), F' color (x, y) are normalized edge, shape, and color feature maps respectively;

[0074] S52, introduce feature weight coefficients α, β, and γ corresponding to edge, shape, and color feature maps respectively. The weight coefficients are determined by optimizing the training data and satisfy α+β+γ=1, where α, β, γ∈[0,1];

[0075] S53, perform weighted fusion on the normalized edge, shape and color feature maps according to the weight coefficient to generate a fused feature map F fuse (x,y),

[0076] F fuse (x,y)=α·F e ' dge (x,y)+β·F s ' hape (x,y)+γ·F c ' olor (x,y);

[0077] S54, use high-pass filter to fusion feature map Ffuse (x, y) is convolved to obtain the enhanced feature map F enhance (x,y);

[0078] S55. Model the image as a graph G = (V, E), where V is the set of pixels, E is the edge between pixels, and the weight of each edge w(i, j) is based on the feature similarity and spatial proximity between pixels, and is defined as:

[0079] Among them, σ and d are parameters that control feature similarity and spatial distance; the minimum cut of the graph G is calculated by the minimum cut maximum flow theory to obtain the segmentation result F graph (x,y),

[0080]

[0081] S56, according to the graph cut segmentation result F graph (x,y) is further processed to extract clear instances of drowning people. The specific steps are as follows:

[0082] S561, apply dilation and erosion operations to fill the holes in the segmentation results, remove noise points, and enhance the connectivity of the target area. The processing formula is: in,

[0083] represents the expansion operation, represents the erosion operation, B and C are structural elements respectively;

[0084] S562. Perform connected domain analysis on the results after morphological processing to identify and extract independent targets

[0085] If N connected domains are detected, each connected domain R i The pixel set is:

[0086] R i ={(x,y)|F morph (x,y)=1and(x,y)∈R i};

[0087] S563, for the connected domain R i Perform size and shape verification, exclude false detection and non-target areas, and set the area threshold A min and Amax , only keep those that meet condition A min ≤|R i |≤ Amax The connected domain of is a valid drowning person instance;

[0088] S57. Based on the degree of matching between the detection results and the actual situation, dynamically optimize the weight coefficients α, β, and γ of feature fusion to improve the accuracy of feature weighted fusion. The specific adjustment formula is:

[0089] α (t+1) =α (t) +Δα,β (t+1) =β (t) +Δβ,γ (t+1) =γ (t) +Δγ;

[0090] Among them, Δα, Δβ and Δγ are adjustment increments.

[0091] Compared with the prior art, the present invention has the following beneficial effects:

[0092] 1. By combining infrared thermal imaging and visible light images, the present invention can obtain more comprehensive information about drowning incidents. Especially in low-light or nighttime environments, the thermal radiation information of infrared images can help identify drowning people, improve the detection rate of drowning incidents, and reduce the risk of failing to detect drowning people in time due to environmental influences.

[0093] 2. The present invention integrates the three features of edge, shape and color, not only utilizing the change of thermal radiation, but also combining the shape and color information of the object, effectively avoiding the misjudgment that may be caused by a single feature, and further improving the accuracy and robustness of recognition.

[0094] 3. The present invention adopts advanced image processing methods such as neural impulse response network and convolutional neural network to effectively improve the feature extraction accuracy of the image, especially in complex backgrounds, which can better extract the outline and body information of the drowning person and overcome the interference of complex background.

[0095] 4. This invention uses real-time alarms and precise positioning to quickly transmit the location and status of drowning people to ground control centers, significantly shortening rescue response time. This is of great significance for improving the efficiency of drowning rescue and reducing injuries and deaths.

[0096] 5. By combining drone cruising with infrared thermal imaging technology, the present invention can be widely used in various high-incidence drowning accident scenarios such as seas, rivers, and lakes, providing an innovative solution for water safety monitoring and drowning accident prevention. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 is a flow chart of the overall method steps of the present invention;

[0098] Figure 2 This is a flow chart of the method steps of step S2 of the present invention;

[0099] Figure 3This is a flowchart of the method steps of step S3 of the present invention;

[0100] Figure 4 This is a flow chart of the method steps of step S4 of the present invention;

[0101] Figure 5 This is a flow chart of the method steps of step S5 of the present invention. DETAILED DESCRIPTION

[0102] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0103] Example 1:

[0104] like Figure 1 As shown, an embodiment of the present invention provides a drowning visual recognition method integrating infrared temperature difference features, comprising the following steps:

[0105] S1. Image acquisition and feature extraction: When the drone is cruising, it is equipped with a visible light camera and infrared thermal imaging equipment to collect visible light images and infrared thermal images within the cruising range in real time;

[0106] S2. Edge feature map extraction: Preprocess the infrared thermal image and construct a neural impulse response network to extract edge information with significant changes in thermal radiation, highlighting areas with significant changes in thermal intensity, thereby helping to detect the outline of the drowning person and important structures;

[0107] S3. Shape feature map extraction: By performing multi-scale rectangular feature extraction and dynamic weighted screening on the input image, a strong classifier is constructed by combining feature combinations to further analyze the shape of objects in the image and effectively identify the body shape of the drowning person;

[0108] S4, color feature map extraction: multi-layer encoding of visible light images through convolutional neural networks to extract color and texture features, enhancing target detection in low-light environments;

[0109] S5. Feature map fusion and segmentation: The edge feature map, shape feature map, and color feature map extracted from steps S2-S4 are input into the deep learning model to generate a unified instance feature map. The outline of the drowning person is accurately extracted using graph segmentation technology, and the target is locked by combining the heat map and visible light information.

[0110] S6. Drowning person instance identification and alarm triggering: Real-time alarm is issued for detected drowning person instances, and the alarm mechanism is triggered by the drowning person's location and status, and the drowning person's location and detection confidence are transmitted to the ground control station.

[0111] Specifically, such as Figure 2 As shown, the specific method steps of step S2 are as follows:

[0112] S21. First, the input infrared thermal image is standardized. Infrared thermal images are generally obtained directly by thermal sensors, where the value of each pixel represents the thermal radiation intensity at that location. Infrared thermal images are usually grayscale images. Therefore, before further processing, the image is normalized to ensure that its pixel values ​​are within a certain range. Each pixel value is scaled to the range of [0,1] to represent the relative value of the thermal intensity. Assume that the input infrared thermal image is, where represents the pixel coordinates of the image and the image size is. The image after normalization can be expressed as:

[0113] Among them, I min and I max and are the minimum and maximum pixel values ​​in the image, respectively;

[0114] S22. Construct a neuron impulse response network and extract edge features using changes in neuron membrane potential and neighborhood coupling relationships. The output of each neuron is determined by its input image value, local coupling strength, and global threshold. Let v(x,y,t) represent the membrane potential of the neuron at position (x,y) at time step t, and v threshold is the threshold of the neuron, c(x,y) is the coupling strength of the neuron, and I'(x,y) is the intensity value of the input image. Then the dynamic equation of the neuron can be expressed as:

[0115]

[0116] Where N(x,y) is the neighborhood pixels around the position (x,y), and λ is the attenuation coefficient;

[0117] S23, when the neuron's membrane potential v(x,y,t) reaches the threshold v threshold When , the neuron will generate an impulse response, indicating that the feature of the position is activated. The pulse generation rule is:

[0118]

[0119] Among them, δ(x, y, t) represents the impulse response. If the membrane potential exceeds the threshold, a pulse signal is output;

[0120] S24. Use the impulse response graph to simulate the coupling between neurons. By calculating the cumulative output of the impulse, the area with drastic changes in thermal intensity in the image is highlighted, thereby generating an edge response graph E(x,y). Where T is the total time steps of pulse propagation, and E(x,y) represents the edge strength of the image at position (x,y). The higher the strength value, the more likely the position is an edge position.

[0121] S25. De-noise and smooth the preliminary edge information image obtained by accumulating the impulse responses to generate a high-quality edge feature map F(x, y). The specific steps are as follows:

[0122] Denoising: By setting the threshold T of edge strength edge , remove the response intensity below the threshold T edge Pulse points to remove possible noise;

[0123] Edge smoothing: Smoothing is performed to remove isolated pixels and ensure edge continuity. The edge feature map F(x,y) is expressed as:

[0124] Where T edge Is the threshold of edge strength. If the edge strength exceeds the threshold, the position is considered to be an edge. The final output edge feature map F edge (x,y) clearly shows the area with drastic changes in thermal intensity in the image, which is used for subsequent drowning person detection and target recognition.

[0125] Specifically, such as Figure 3 As shown, the specific method steps of step S3 are as follows:

[0126] S31. Standardize and denoise the infrared thermal image to ensure input stability. The specific steps are as follows:

[0127] S311, normalization processing: normalize the pixel values ​​to the interval [0,1], the normalized image of the original image I(x,y) is I norm (x,y), then Among them, min(I) and max(I) are the minimum and maximum pixel values ​​in the image respectively;

[0128] S312, noise removal: Apply Gaussian filtering to reduce image noise, and the smoothed image I smooth (x,y) is: Where G(x,y,σ) is a filter with a standard deviation of σ;

[0129] S32. Extracting rectangular features of different scales and directions from the preprocessed image, extracting local brightness features and constructing feature vectors. The specific steps are as follows:

[0130] S321, define rectangular areas: define several rectangular areas around each pixel point (x, y), namely R1, R2, ..., R m ,These rectangular regions vary in size, location, and orientation to capture a variety of shape features;

[0131] S322, brightness difference calculation: For each rectangular area pair (R i ,R j ), calculate the brightness difference:

[0132] where f ij (x,y) represents the position

[0133] The eigenvalue of (x,y) is obtained by the rectangular area pair (R i , R j ) to describe the local brightness differences of the image. The model can effectively capture the different morphological and structural information of the image, providing a basis for subsequent classification and detection tasks.

[0134] S323, feature vector construction: At each pixel position, according to all rectangular features f ij Generate the feature vector F(x,y), that is: F(x,y)=[f 11 (x,y),f 12 (x,y),…,f mn (x,y)]; eigenvector

[0135] The dimension of F(x,y) is related to the number of selected rectangular feature combinations.

[0136] S33. To improve detection accuracy, a dynamic weighting mechanism is introduced to assign different weights to each rectangular feature and gradually optimize the weights through multiple iterations. The specific steps are as follows:

[0137] S331. Initial weight allocation: Set the initial weights of all features to be the same, that is, for the kth feature, the initial weight is:

[0138] S332, Error calculation and weight adjustment: In each round of iteration, select the feature combination with the smallest error and adjust its weight. Suppose that feature f is selected in round t. ij As the main feature, its classification error is ε ij , then the corresponding weight update is: in,

[0139] is the adjustment coefficient, h ij Based on the feature f ijThe weak classifier output, l() represents the indicator function, which is used to determine whether the classification is correct;

[0140] S333. After updating the weights of all features, perform normalization:

[0141] Through this dynamic weight update, the importance of features gradually focuses on features with better performance.

[0142] S34. Combine multiple weighted features into a series of weak classifiers, and generate the final strong classifier through weighted integration. The specific steps are as follows:

[0143] S341, weak classifier construction: for each feature f ij Construct weak classifier h ij (x,y), the output is 1 or -1, indicating whether the position matches the target shape. The weak classifier is defined as:

[0144] Among them, θ ij is the feature f ij The classification threshold;

[0145] S342, strong classifier integration: All weak classifiers are combined into a strong classifier H(x,y) by weighted summation, which is defined as: Among them, α ij is the weight of the weak classifier, sign() represents the sign function, output 1 means the shape feature is detected, output -1 means it is not detected;

[0146] S35. Use the strong classifier H(x,y) to detect each pixel and output the shape feature map F shape (x,y), the specific steps are:

[0147] S351, feature map generation: First use the strong classifier H(x,y) to determine whether each pixel (x,y) belongs to the target shape area:

[0148] S352, post-processing: In order to improve the continuity of the feature map, morphological operations are performed on the output shape feature map, including dilation and erosion operations, to eliminate isolated points and smooth the boundaries.

[0149] Specifically, such as Figure 4 As shown, the specific method steps of step S4 are as follows:

[0150] S41, normalizing and border-expanding the input visible light image to ensure image consistency and integrity;

[0151] Normalization: Map each pixel value of the visible light image to the interval [0,1] to ensure that pixels of different intensities can be processed in the same numerical range. Let the input image be I RGB (x,y), its normalized image I norm (x,y) is represented as:

[0152]

[0153] Among them, min(I RGB ) and max(I RGB ) are the minimum and maximum pixel values ​​of the image, respectively.

[0154] Boundary expansion: In order to avoid edge information loss during convolution operations, symmetric padding is used to expand the image boundary.

[0155] S42. In the encoding stage, the input image is passed through a series of convolution and pooling operations layer by layer to generate image feature representations of different scales. The feature maps of each layer gradually focus on different color information and texture information from shallow to deep layers. Assuming the input image size is H×W, the specific encoding process is as follows:

[0156] S421, convolution layer: Each convolution layer uses convolution kernel K to extract local features of the image, and the size of the convolution kernel in the lth layer is k l ×k l , the convolution kernel weight is W l , bias is b l , the output of the l-th layer convolution is:

[0157] Among them, σ is a nonlinear activation function (such as ReLU), I in (x,y) is the input feature map;

[0158] S422, Pooling layer: After each convolution layer, pooling operation is applied to reduce the spatial dimension of the feature map and improve computational efficiency. Assuming the pooling window size is p×p, the feature map after pooling is:

[0159] P l (x,y)=max u,v∈p×p F l (x+u,y+v); The pooling operation reduces the spatial resolution of the feature map layer by layer and retains the main activation value of each area.

[0160] S423, multi-layer coding: Repeat convolution and pooling operations to build a multi-layer coding structure. Let the number of coding layers be L, then the final generated coding feature map is represented as F L (x,y), the feature map contains the color and texture information of the original image at different scales and depths;

[0161] S43. In order to enhance the context information in color feature extraction, an aggregation layer is designed to aggregate the multi-scale color features into one representation through feature fusion. The specific steps are as follows:

[0162] S431, skip connection: Introduce a skip connection between the encoding layer and the decoding layer to retain shallow feature information. The output of the skip connection is: C l (x,y)=P l (x,y)+D l (x,y), where P l (x,y) and D l (x, y) represents the feature map of the lth layer in the encoding stage and the decoding stage respectively. The skip connection ensures that the high-resolution color information is not lost;

[0163] S432, context fusion: splicing the feature maps of different layers in the encoding stage to form a context feature representation: C context (x,y)=concat{C1(x,y),C2(x,y),...,C l (x,y)};

[0164] S44, by gradually upsampling the feature map to restore it to the resolution of the original image, thereby extracting complete color feature information. The decoding layer corresponds to the encoding layer and combines the context information to reconstruct the high-resolution color feature map. The specific steps are as follows:

[0165] S441, upsampling operation: Each layer restores the low-resolution feature map to high resolution by upsampling. Assume that the upsampling factor is s, and the upsampling output of the lth layer is:

[0166] S442, decoding convolution layer: After each upsampling, convolution is applied to extract details. The output of the decoding convolution layer is:

[0167] Among them, W d and b d are the weights and biases of the convolution kernels in the decoding layer, k d is the size of the decoding convolution kernel, σ is the activation function;

[0168] S45, after layer-by-layer upsampling and decoding convolution operations, the final output is the color feature map F color (x,y), this feature map retains the color information of the input image and is suitable for subsequent feature fusion, including:

[0169] Feature map generation: The generation formula of the color feature map is:

[0170] Among them, the softmax function normalizes the output into a probability distribution to enhance the color contrast information of the image.

[0171] Post-processing: After the color feature map is generated, a refinement operation is applied to make the feature edges smoother. The final generated color feature map can retain the color information of the target object and effectively support subsequent multimodal fusion processing.

[0172] Specifically, such as Figure 5 As shown, the specific method steps of step S5 are as follows:

[0173] In the above steps, the edge feature maps F of the infrared thermal image are extracted respectively. edge (x,y), shape feature map F of infrared thermal image shape (x, y) and the color feature map F of the visible light image color (x,y). This step aims to effectively combine the above three feature maps through the feature weighted fusion method, and further segment the fused features to accurately extract the drowning person instance.

[0174] S51. Linear normalize the edge, shape, and color feature maps separately to ensure that the data range is consistent and avoid weight deviation due to numerical differences. The specific formula is as follows:

[0175]

[0176] Among them, F' edge (x,y), F' shape (x,y), F' color (x, y) are normalized edge, shape, and color feature maps respectively;

[0177] S52. In order to reasonably reflect the importance of each feature map in the fusion process, feature weight coefficients α, β, and γ are introduced to correspond to edge, shape, and color feature maps respectively. The weight coefficients are determined by optimizing the training data and satisfy α+β+γ=1, where α, β, γ∈[0,1];

[0178] S53, perform weighted fusion on the normalized edge, shape and color feature maps according to the weight coefficient to generate a fused feature map F fuse (x,y),

[0179] F fuse (x,y)=α·F e ' dge (x,y)+β·F s ' hape (x,y)+γ·F c ' olor (x,y), where F fuse(x,y) integrates the feature information of edge, shape and color, providing a rich and coordinated feature basis for subsequent segmentation.

[0180] S54, use high-pass filter H(x,y) to fusion feature map F fuse (x, y) is convolved to obtain the enhanced feature map F enhance (x,y), F enhance (x,y)=F fuse (x,y)*H(x,y), where * represents the convolution operation;

[0181] S55. First, the image is modeled as a graph G = (V, E), where V is a set of pixels, E is the edge between pixels, and the weight of each edge w(i, j) is based on the feature similarity and spatial proximity between pixels, and is defined as:

[0182] Among them, σ and d are parameters that control feature similarity and spatial distance; the minimum cut of the graph G is calculated by the minimum cut maximum flow theory to obtain the segmentation result F graph (x,y),

[0183]

[0184] S56, the graph cut segmentation result F obtained by the segmentation algorithm graph (x,y) needs to be further processed to extract clear instances of drowning people. The specific steps are as follows:

[0185] S561, morphological processing: Apply dilation and erosion operations to fill holes in the segmentation results, remove isolated noise points, and enhance the connectivity of the target area. The processing formula is:

[0186] in, represents the expansion operation, represents the erosion operation, B and C are structural elements respectively;

[0187] S562, Connected Domain Analysis: Perform connected domain analysis on the results after morphological processing to identify and extract independent target areas. If N connected domains are detected, each connected domain R i The pixel set is:

[0188] R i ={(x,y)|F morph (x,y)=1and(x,y)∈R i};

[0189] S563, target verification: for the connected domain R i Perform size and shape verification, exclude false detection and non-target areas, and set the area threshold Amin and Amax , only keep those that meet condition A min ≤|R i |≤ Amax The connected domain of is taken as a valid drowning person instance, where |R i | is the connected domain R i The area (number of pixels), A min and Amax Set according to the actual size range of the target instance.

[0190] S57. To improve the adaptability of the system in different environments, dynamically adjust the feature weight coefficients α, β, and γ to improve the accuracy of feature weighted fusion. Set up a feedback mechanism to adjust the weight coefficients by real-time monitoring the degree of match between the detection results and the actual situation. The specific adjustment formula is as follows:

[0191] α (t+1) =α (t) +Δα,β (t+1) =β (t) +Δβ,γ (t+1) =γ (t) +Δγ;

[0192] Among them, Δα, Δβ and Δγ are adjustment increments, which are dynamically adjusted according to the detection accuracy and false detection rate to ensure the stability and accuracy of the fusion method under different water areas and lighting conditions. The adjustment principle must meet α (t+1) +β (t+1) +γ (t+1) =1,α (t+1) ,β (t+1) ,γ (t+1) ∈[0,1],

[0193] Dynamic adjustments are made based on the detection accuracy and false detection rate to ensure the stability and accuracy of the fusion method under different water and lighting conditions.

[0194] Through the above steps, the present invention realizes the effective fusion and segmentation of multiple feature maps, significantly improves the accuracy and real-time performance of drowning detection, overcomes the shortcomings of traditional monitoring methods such as slow response speed and limited coverage, and gives full play to the advantages of drone maneuverability and multi-sensor image processing.

[0195] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A drowning visual recognition method integrating infrared temperature difference features, characterized in that: The following steps are involved: S1. Image acquisition and feature extraction: When the drone is cruising, it is equipped with a visible light camera and infrared thermal imaging equipment to collect visible light images and infrared thermal images within the cruising range in real time; S2. Edge feature map extraction: Preprocess the infrared thermal image and construct a neural impulse response network to extract edge information with significant changes in thermal radiation, highlighting areas with significant changes in thermal intensity, thereby helping to detect the outline of the drowning person and important structures; S3. Shape feature map extraction: By performing multi-scale rectangular feature extraction and dynamic weighted screening on the input image, a strong classifier is constructed by combining feature combinations to further analyze the shape of objects in the image and effectively identify the body shape of the drowning person; S4, color feature map extraction: multi-layer encoding of visible light images through convolutional neural networks to extract color and texture features, enhancing target detection in low-light environments; S5. Feature map fusion and segmentation: The edge, shape, and color feature maps extracted in steps S2-S4 are fed into the deep learning model to generate a unified instance feature map. Graph segmentation technology is used to accurately extract the outline of the drowning person, and the heat map and visible light information are combined to lock the target. S6. Drowning person instance identification and alarm triggering: Real-time alarm is issued for detected drowning person instances, and the alarm mechanism is triggered by the drowning person's location and status, and the drowning person's location and detection confidence are transmitted to the ground control station.

2. The drowning visual recognition method integrating infrared temperature difference features according to claim 1 is characterized in that: The specific method steps of step S2 are as follows: S21, performing normalization processing on the input infrared thermal image, scaling the pixel values ​​in the image to the range of [0, 1], and eliminating differences by adjusting the ratio of the minimum and maximum pixel values; S22. Construct a neuron impulse response network and extract edge features using changes in neuron membrane potential and neighborhood coupling. The dynamic equation of the neuron is: Among them, v(x,y,t) represents the membrane potential of the neuron at position (x,y) at time step t, v threshold is the threshold of the neuron, c(x,y) is the coupling strength of the neuron, I'(x,y) is the intensity value of the input image, N(x,y) is the neighborhood pixel around the position (x,y), and λ is the attenuation coefficient; S23, when the neuron's membrane potential v(x,y,t) reaches the threshold v threshold When , the neuron will generate an impulse response, indicating that the feature of the position is activated. The pulse generation rule is: Among them, δ(x, y, t) represents the impulse response. If the membrane potential exceeds the threshold, a pulse signal is output; S24. Use the impulse response graph to simulate the coupling between neurons. By calculating the cumulative output of the impulse, the area with drastic changes in thermal intensity in the image is highlighted, thereby generating an edge response graph E(x,y). Where T is the total number of time steps of pulse propagation, E(x,y) represents the edge strength of the image at position (x,y); S25. De-noise and smooth the preliminary edge information image obtained by accumulating the impulse responses to generate a high-quality edge feature map F(x, y). The specific steps are as follows: By setting the edge strength threshold T edge , remove the response intensity below the threshold T edge The pulse points are then smoothed to remove isolated pixels to ensure edge continuity. The edge feature map F(x,y) is expressed as: The final output edge feature map F edge (x,y) clearly shows the area with drastic changes in thermal intensity in the image, which is used for subsequent drowning person detection and target recognition.

3. The drowning visual recognition method integrating infrared temperature difference features according to claim 1 is characterized in that: The specific method steps of step S3 are as follows: S31. Standardize and denoise the infrared thermal image to ensure input stability. The specific steps are as follows: S311, normalize the pixel values ​​to the interval [0,1], and the normalized image of the original image I(x,y) is I norm (x,y), then Among them, min(I) and max(I) are the minimum and maximum pixel values ​​in the image respectively; S312, then apply Gaussian filtering to reduce image noise, and the smoothed image I smooth (x,y) is: Where G(x,y,σ) is a filter with a standard deviation of σ; S32. By calculating the brightness difference of rectangular areas of different scales and directions, extract local brightness features and construct feature vectors. The specific steps are as follows: S321, define multiple rectangular areas R1, R2, ..., R near the pixel point (x, y) m , covering different sizes, orientations and positions; S322, calculate the rectangular area pair (R i , R j )’s brightness difference: where f ij (x,y) represents the position The eigenvalue of (x,y) is obtained by the rectangular area pair (R i , R j ) is calculated by the brightness difference The result,describes the local brightness differences of the image; S323, according to all rectangular eigenvalues ​​f ij , generate a feature vector F(x,y) at each pixel position, Expressed as: F(x,y)=[f 11 (x,y),f 12 (x,y),…,f mn (x,y)]; S33. Introduce a dynamic weighting mechanism to assign different weights to each rectangular feature. By iteratively optimizing the feature weights, the detection accuracy is improved. The specific steps are as follows: S331. Initialize the weights of all features to be the same. For the kth feature, the initial weight is: S332, in each round of iteration, select the feature combination with the minimum error and update its weight. Select feature f in round ij As the main feature, its classification error is ε ij , then the updated weight is: in, is the adjustment coefficient, h ij Based on the feature f ij The weak classifier output, l() represents the indicator function, which is used to determine whether the classification is correct; S333. After updating the weights of all features, perform normalization: S34. Combining weighted features into weak classifiers and generating strong classifiers through weighted integration. The specific steps are as follows: S341, for each feature f ij Construct weak classifier h ij (x,y), the output is 1 or -1, indicating whether the position matches the target shape. The weak classifier is defined as: Among them, θ ij is the feature f ij The classification threshold; S342. Combine all weak classifiers into a strong classifier H(x,y) by weighted summation, which is defined as: Among them, α ij is the weight of the weak classifier, sign() represents the sign function, output 1 means the shape feature is detected, output -1 means it is not detected; S35. Use the strong classifier H(x,y) to detect each pixel and output the shape feature map F shape (x,y), the specific steps are: S351. First, use the strong classifier H(x,y) to determine whether each pixel (x,y) belongs to the target shape area: S352. Then perform morphological processing on the output shape feature map, including dilation and erosion operations, to eliminate isolated points and smooth boundaries to obtain the final target detection result.

4. The drowning visual recognition method integrating infrared temperature difference features according to claim 1 is characterized in that: The specific method steps of step S4 are as follows: S41, normalizing and expanding the boundaries of the input visible light image to ensure image consistency and integrity; S42. Extract the color and texture features of the image layer by layer through convolution and pooling operations to generate a multi-level feature representation. The specific encoding process is as follows: S421, each convolution layer uses convolution kernel K to extract local features of the image, and the convolution kernel size of the lth layer is k l ×k l , the convolution kernel weight is W l , bias is b l , the output of the l-th layer convolution is: Among them, σ is a nonlinear activation function, I in (x,y) is the input feature map; S422, apply pooling operation after convolution to reduce the spatial dimension. Set the pooling window size to p×p, and the feature map after pooling is: P l (x,y)=max u,v∈p×p F l (x+u,y+v); S423, through multiple convolution and pooling operations, gradually build a multi-layer encoding structure to generate image feature maps of different scales and depths; S43. Enhance the context information of color feature extraction through skip connection and context fusion. The specific steps are as follows: S431, introduce a skip connection between the encoding layer and the decoding layer to retain the shallow feature information. The output of the skip connection is: C l (x,y)=P l (x,y)+D l (x,y), where P l (x,y) and D l (x, y) represents the feature map of the lth layer in the encoding stage and the decoding stage respectively. The skip connection ensures that the high-resolution color information is not lost; S432. Concatenate the feature maps of different layers in the encoding stage to form a context feature representation: C context (x,y)=concat{C1(x,y),C2(x,y),...,C l (x,y)}; S44, gradually upsampling and convolution to restore image resolution and extract complete color features. The specific steps are as follows: S441, each layer restores the low-resolution feature map to high resolution by upsampling. Let the upsampling factor be s, and the upsampling output of the lth layer is: S442. After each upsampling, convolution is applied to extract details. The output of the decoded convolution layer is: Among them, W d and b d are the weights and biases of the convolution kernels in the decoding layer, k d is the size of the decoding convolution kernel, σ is the activation function; S45. Generate color feature map F by upsampling and decoding convolution color (x,y), and use the softmax function to enhance the color contrast information, After generating the feature map, the edges are smoothed through a refinement operation to preserve the color information of the target object in preparation for multimodal fusion.

5. The drowning visual recognition method integrating infrared temperature difference features according to claim 1 is characterized in that: The specific method steps of step S5 are as follows: S51. Linear normalize the edge, shape, and color feature maps separately to ensure that the data range is consistent and avoid weight deviation due to numerical differences. The specific formula is as follows: Among them, F' edge (x,y), F' shape (x,y), F' color (x, y) are normalized edge, shape, and color feature maps respectively; S52, introduce feature weight coefficients α, β, and γ corresponding to edge, shape, and color feature maps respectively. The weight coefficients are determined by optimizing the training data and satisfy α+β+γ=1, where α, β, γ∈[0,1]; S53, perform weighted fusion on the normalized edge, shape and color feature maps according to the weight coefficient to generate a fused feature map F fuse (x,y), F fuse (x,y)=α·F e ′ dge (x,y)+β·F s ′ hape (x,y)+γ·F c ′ olor (x,y); S54, use high-pass filter to fusion feature map F fuse (x, y) is convolved to obtain the enhanced feature map F enhance (x,y); S55. Model the image as a graph G = (V, E), where V is the set of pixels, E is the edge between pixels, and the weight of each edge w(i, j) is based on the feature similarity and spatial proximity between pixels, and is defined as: Among them, σ and d are parameters that control feature similarity and spatial distance; the minimum cut of the graph G is calculated by the minimum cut maximum flow theory to obtain the segmentation result F graph (x,y), 6. The drowning visual recognition method integrating infrared temperature difference features according to claim 5 is characterized in that: After step S55, step S56 is further included, wherein the graph segmentation result F is divided into graph (x,y) is further processed to extract clear instances of drowning people. The specific steps are as follows: S561, apply dilation and erosion operations to fill the holes in the segmentation results, remove noise points, and enhance the connectivity of the target area. The processing formula is: in, represents the expansion operation, represents the erosion operation, B and C are structural elements respectively; S562, the result after morphological processing F morph (x, y) performs connected domain analysis to identify and extract independent target areas. If N connected domains are detected, each connected domain R i The pixel set is: R i = {(x, y)|F morph (x, y) = 1 and (x, y) ∈ R i}; S563, for the connected domain R i Perform size and shape verification, exclude false detection and non-target areas, and set the area threshold A min and Amax , only keep those that meet condition A min ≤|R i |≤ Amax The connected domain of is taken as a valid drowning person instance.

7. The drowning visual recognition method integrating infrared temperature difference features according to claim 6 is characterized in that: After step S56, the method further includes step S57, dynamically optimizing the weight coefficients α, β, and γ of feature fusion according to the degree of matching between the detection result and the actual situation to improve the accuracy of feature weighted fusion. The specific adjustment formula is: a (t+1) =a (t) +Dα,β (t+1) =b (t) +Δβ,γ (t+1) =c (t) +Δγ; where Δα, Δβ, Δγ are adjustment increments, and α (t+1) + β (t+1) + γ (t+1) = 1, and α (t+1) , β (t+1) , γ (t+1) ∈ [0, 1].

Images