Drowning visual identification method fusing infrared temperature difference characteristics

Abstract

The invention discloses a drowning visual identification method fusing infrared temperature difference features. The method comprises the following steps: S1, image acquisition and feature extraction; s2, extracting an edge feature map; s3, extracting a shape feature map; s4, extracting a color feature map; s5, feature map fusion and segmentation; and S6, identifying a drowning person instance and triggering an alarm. According to the invention, infrared thermal imaging and visible light images are combined, the accuracy and real-time performance of drowning detection are improved by using a multi-feature fusion technology, and particularly, a drowning person is effectively identified in a low-light environment; the system rapidly transmits drowning person information through intelligent alarm and accurate positioning, shortens rescue response time, and improves rescue efficiency. Besides, by adopting an advanced neuron impulse response network and a convolutional neural network, features of drowners are extracted more accurately in a complex background, the method is suitable for various water area environments, and an innovative solution is provided for drowning accident prevention.

Description

Technical Field

[0001] The present invention relates to the field of drowning monitoring systems, and in particular to a drowning visual recognition method integrating infrared temperature difference features. Background Art

[0002] In recent years, drowning incidents have occurred frequently in waters, especially in open waters, beaches, lakes and other places. Drowning accidents have become a major issue affecting people's life safety. Traditional drowning monitoring methods mostly rely on manual patrols, shore monitoring cameras or fixed water surface monitoring systems. These traditional methods have obvious defects in many aspects: traditional monitoring systems often require manual intervention or manual analysis to detect abnormal situations, and the response speed is slow, resulting in poor rescue timeliness for drowning incidents; fixed monitoring cameras or manual patrols cannot cover a large area of ​​water, especially in vast lakes, seas and other areas, there are blind spots or it is impossible to monitor all areas in real time; due to the complex environment, factors such as fluctuations and floating objects on the water surface can easily lead to false alarms, which increases the difficulty of processing monitoring results.

[0003] In order to solve these problems, drones, as a flexible and efficient monitoring tool, are gradually being used in the field of drowning monitoring. Drones can cruise quickly and collect image and video data in real time within the water area, greatly improving the coverage and response speed of monitoring. Especially with the technical support of multi-sensor fusion, drones can provide more accurate data support under different environmental conditions.

[0004] Currently, drone-based drowning monitoring mostly relies on monitoring by visible light images, infrared thermal imaging or other single sensors. However, a single sensor has certain limitations in complex environments:

[0005] Although visible light images are clear, their effectiveness is greatly reduced in the case of insufficient light or strong light reflection, especially at night or in low-light environments, where visible light images cannot effectively obtain target information; infrared thermal imaging images can work in low-light or night conditions, but their resolution is low and they cannot accurately identify smaller targets or complex backgrounds on the water surface. They are easily disturbed by factors such as ambient temperature and wind and waves, resulting in reduced recognition accuracy. Therefore, the present invention combines the maneuverability of drones and the advantages of multiple sensor images to propose a new method that can efficiently detect drowning people. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a drowning visual recognition method integrating infrared temperature difference features, which solves the problem that the prior art solutions cannot achieve efficient, accurate and real-time drowning detection in complex water environments.

[0007] To achieve the above purpose, the following technical solutions can be used:

[0008] The present invention provides a drowning visual recognition method integrating infrared temperature difference features, comprising the following steps:

[0009] S1. Image acquisition and feature extraction: When the drone is cruising, it is equipped with a visible light camera and infrared thermal imaging equipment to collect visible light images and infrared thermal images within the cruising range in real time;

[0010] S2. Edge feature map extraction: Preprocess the infrared thermal image and construct a neural impulse response network to extract edge information with significant changes in thermal radiation and highlight areas with significant changes in thermal intensity, thereby helping to detect the outline and important structures of the drowning person;

[0011] S3, shape feature map extraction: by performing multi-scale rectangular feature extraction and dynamic weighted screening on the input image, a strong classifier is constructed by combining feature combinations to further analyze the shape of objects in the image and effectively identify the body shape of the drowning person;

[0012] S4, color feature map extraction: multi-layer encoding of visible light images is performed through convolutional neural networks to extract color and texture features to enhance target detection in low-light environments;

[0013] S5, feature map fusion and segmentation: The edge, shape and color feature maps extracted in steps S2-S4 are input into the deep learning model for fusion to generate a unified instance feature map, and the outline of the drowning person is accurately extracted using the graph segmentation technology, and the target is locked by combining the heat map and visible light information;

[0014] S6. Drowning person instance identification and alarm triggering: Real-time alarm is issued for detected drowning person instances, and the alarm mechanism is triggered by the drowning person's location and status, and the drowning person's location and detection confidence are transmitted to the ground control station.

[0015] As a further solution of the present invention, the specific method steps of step S2 are as follows:

[0016] S21, standardize the input infrared thermal image, scale the pixel values ​​in the image to the range of [0, 1], and eliminate the difference by adjusting the ratio of the minimum and maximum pixel values;

[0017] S22. Construct a neuron impulse response network and extract edge features using changes in neuron membrane potential and neighborhood coupling. The dynamic equation of the neuron is:

[0018]

[0019] Where v(x,y,t) represents the membrane potential of the neuron at position (x,y) at time step t, v thresholdis the threshold of the neuron, c(x,y) is the coupling strength of the neuron, I'(x,y) is the intensity value of the input image, N(x,y) is the neighborhood pixels around the position (x,y), and λ is the attenuation coefficient;

[0020] S23, when the neuron's membrane potential v(x,y,t) reaches the threshold v threshold When , the neuron will generate an impulse response, indicating that the feature of the position is activated, and the pulse generation rule is:

[0021]

[0022] Among them, δ(x,y,t) represents the pulse response. If the membrane potential exceeds the threshold, a pulse signal is output;

[0023] S24. The pulse response graph is used to simulate the coupling effect between neurons. By calculating the cumulative output of the pulse, the area where the thermal intensity changes dramatically in the image is highlighted, thereby generating an edge response graph E(x,y). Where T is the total time steps of pulse propagation, and E(x,y) represents the edge strength of the image at position (x,y);

[0024] S25, denoising and smoothing the preliminary edge information image obtained by accumulating the impulse response to generate a high-quality edge feature map F(x, y), the specific steps are as follows:

[0025] By setting the edge strength threshold T edge , remove the response intensity below the threshold T edge The pulse points are smoothed to remove isolated pixels to ensure edge continuity. The edge feature map F(x,y) is expressed as:

[0026] The final output edge feature map Fedge(x,y) clearly shows the area where the thermal intensity changes dramatically in the image, which is used for subsequent drowning person detection and target recognition.

[0027] As a further solution of the present invention, the specific method steps of step S3 are as follows:

[0028] S31. Standardize and denoise the infrared thermal image to ensure input stability. The specific steps are as follows:

[0029] S311, normalize the pixel values ​​to the interval [0,1], and the normalized image of the original image I(x,y) is

[0030] Among them, min(I) and max(I) are the minimum and maximum pixel values ​​in the image respectively;

[0031] S312, apply Gaussian filtering to reduce image noise, then the smoothed image I smooth (x,y) is: Where G(x,y,σ) is a filter with a standard deviation of σ;

[0032] S32, by calculating the brightness difference of rectangular areas of different scales and directions, extracting local brightness features and constructing feature vectors, the specific steps are as follows:

[0033] S321, define multiple rectangular areas R1, R2, ..., R near the pixel point (x, y) m Covering different sizes, directions and locations;

[0034] S322, calculate the rectangular area pair (R i , R j )’s brightness difference:

[0035] where f ij (x,y) represents the position

[0036] The eigenvalues ​​of (x,y) are obtained by the rectangular area pair (R i , R j ) is calculated by the brightness difference

[0037] As a result, the local brightness differences of the image are described;

[0038] S323, according to all rectangular eigenvalues ​​f ij , generate a feature vector F(x,y) at each pixel position, expressed as: F(x,y)=[f 11 (x,y),f 12 (x,y),…,f mn (x,y)];

[0039] S33, introduce a dynamic weighting mechanism to assign different weights to each rectangular feature, and improve detection accuracy by iteratively optimizing feature weights. The specific steps are as follows:

[0040] S331. Initialize the weights of all features to be the same. For the kth feature, the initial weight is:

[0041]

[0042] S332. In each round of iteration, select the feature combination with the smallest error and update its weight. In the tth round, select feature f ij As the main feature, its classification error is ε ij , then the updated weight is:

[0043] in, is the adjustment coefficient, h ij Based on the feature f ij The weak classifier output, l() represents the indicator function, which is used to determine whether the classification is correct;

[0044] S333. After updating the weights of all features, normalization is performed:

[0045] S34, combining weighted features into weak classifiers, and generating strong classifiers through weighted integration, the specific steps are as follows:

[0046] S341, for each feature f ij Construct weak classifier h ij (x, y), the output is 1 or -1, indicating whether the position matches the target shape. The definition of the weak classifier is:

[0047] Among them, θ ij The feature f ij The classification threshold of

[0048] S342. All weak classifiers are combined into a strong classifier H(x,y) by weighted summation, which is defined as: Among them, α ij is the weight of the weak classifier, sign() represents the sign function, output 1 means the shape feature is detected, output -1 means it is not detected;

[0049] S35, use the strong classifier H(x,y) to detect each pixel and output the shape feature map F shape (x,y), the specific steps are:

[0050] S351, first use the strong classifier H(x,y) to determine whether each pixel (x,y) belongs to the target shape area:

[0051] S352, then perform morphological processing on the output shape feature map, including dilation and erosion operations, to eliminate isolated points and smooth boundaries to obtain the final target detection result.

[0052] As a further solution of the present invention, the specific method steps of step S4 are as follows:

[0053] S41, normalizing and expanding the boundary of the input visible light image to ensure image consistency and integrity;

[0054] S42, extract the color and texture features of the image layer by layer through convolution and pooling operations to generate multi-level feature representation. The specific encoding process is as follows:

[0055] S421, each convolution layer uses convolution kernel K to extract local features of the image, and the size of the convolution kernel in the lth layer is k l ×k l , the convolution kernel weight is W l , with a bias of b l , the output of the lth layer convolution is:

[0056] Among them, σ is a nonlinear activation function, I in (x,y) is the input feature map;

[0057] S422, apply pooling operation after convolution to reduce the spatial dimension, and set the pooling window size to p×

[0058] p, the feature map after pooling is: P l (x,y)=max u,v∈p×p F l (x+u,y+v);

[0059] S423, gradually building a multi-layer coding structure through multiple convolution and pooling operations to generate image feature maps of different scales and depths;

[0060] S43, enhancing the context information of color feature extraction by skip connection and context fusion, the specific steps are as follows:

[0061] S431, introduce a skip connection between the encoding layer and the decoding layer to retain the shallow feature information, and the skip connection output is: C l (x,y)=P l (x,y)+D l (x,y), where P l (x,y) and D l (x, y) represents the feature map of the lth layer in the encoding stage and the decoding stage respectively. The skip connection ensures that the high-resolution color information is not lost;

[0062] S432, concatenate the feature maps of different layers in the encoding stage to form a context feature representation:

[0063] C context (x,y)=concat{C1(x,y),C2(x,y),...,C l (x,y)};

[0064] S44, gradually upsampling and convolution to restore the image resolution and extract complete color features, the specific steps are as follows:

[0065] S441, each layer restores the low-resolution feature map to high resolution by upsampling, assuming upsampling

[0066] The factor is s, and the upsampled output of the lth layer is:

[0067] S442, after each upsampling, convolution is applied to extract details, and the output of the decoded convolution layer is:

[0068] Among them, W d and b d are the weights and biases of the convolution kernels in the decoding layer, k d is the size of the decoding convolution kernel, σ is the activation function;

[0069] S45, generate color feature map F by upsampling and decoding convolution color (x,y), and use the softmax function to enhance the color contrast information. After generating the feature map, the edges are smoothed through a refinement operation to preserve the color information of the target object in preparation for multimodal fusion.

[0070] As a further solution of the present invention, the specific method steps of step S5 are as follows:

[0071] S51. Linear normalize the edge, shape, and color feature maps respectively to ensure that the data range is consistent and avoid weight deviation due to numerical differences. The specific formula is as follows:

[0072]

[0073] Among them, F' edge (x,y), F' shape (x,y), F' color (x, y) are normalized edge, shape and color feature maps respectively;

[0074] S52, introducing feature weight coefficients α, β, and γ corresponding to edge, shape, and color feature maps respectively, the weight coefficients are determined by optimizing the training data, and satisfy α+β+γ=1, where α, β, γ∈[0,1];

[0075] S53, weighted fusion is performed on the normalized edge, shape and color feature maps according to the weight coefficient to generate a fused feature map F fuse (x,y),

[0076] F fuse (x,y)=α·F e ' dge (x,y)+β·F s ' hape (x,y)+γ·F c ' olor (x,y);

[0077] S54, use a high-pass filter to fusion feature map Ffuse (x, y) is convolved to obtain the enhanced feature map F enhance (x,y);

[0078] S55. Model the image as a graph G = (V, E), where V is a set of pixels, E is the connecting edge between pixels, and the weight w(i, j) of each edge is based on the feature similarity and spatial proximity between pixels, and is defined as:

[0079] Among them, σ and d are parameters that control feature similarity and spatial distance. The minimum cut of the graph G is calculated by the minimum cut maximum flow theory to obtain the segmentation result F graph (x,y),

[0080]

[0081] S56, according to the graph cut segmentation result F graph (x,y) is further processed to extract clear drowning person instances. The specific steps are as follows:

[0082] S561, apply expansion and corrosion operations to fill the holes in the segmentation results, remove noise points, and enhance the connectivity of the target area. The processing formula is: in,

[0083] represents the expansion operation, represents the corrosion operation, B and C are structural elements respectively;

[0084] S562. Perform connected domain analysis on the results of morphological processing to identify and extract independent targets

[0085] If N connected domains are detected, each connected domain R i The pixel set is:

[0086] R i ={(x,y)|F morph (x,y)=1 and (x,y)∈R i};

[0087] S563, for the connected domain R i Verify the size and shape, exclude false detection and non-target areas, and set the area threshold A min and Amax , only keep those that meet condition A min ≤|R i |≤ Amax The connected domain of is a valid drowning person instance;

[0088] S57. According to the matching degree between the detection result and the actual situation, the weight coefficients α, β and γ of the feature fusion are dynamically optimized to improve the accuracy of the feature weighted fusion. The specific adjustment formula is:

[0089] α (t+1) =α (t) +Δα,β (t+1) =β (t) +Δβ,γ (t+1) =γ (t) +Δγ;

[0090] Among them, Δα, Δβ and Δγ are adjustment increments.

[0091] Compared with the prior art, the present invention has the following beneficial effects:

[0092] 1. By combining infrared thermal imaging and visible light images, the present invention can obtain more comprehensive information about drowning incidents, especially in low-light or nighttime environments. The thermal radiation information of infrared images helps to identify drowning people, which can improve the detection rate of drowning incidents and reduce the risk of failing to discover drowning people in time due to environmental influences.

[0093] 2. The present invention integrates the three features of edge, shape and color, not only utilizing the change of thermal radiation, but also combining the shape and color information of the object, effectively avoiding the misjudgment that may be caused by a single feature, and further improving the accuracy and robustness of recognition.

[0094] 3. The present invention adopts advanced image processing methods such as neural pulse response network and convolutional neural network, which effectively improves the feature extraction accuracy of the image, especially in complex backgrounds, and can better extract the outline and body information of the drowning person and overcome the interference of complex background.

[0095] 4. The present invention can quickly transmit the location and status of the drowning person to the ground control center through real-time alarm and precise positioning, greatly shortening the rescue response time. This is of great significance for improving the efficiency of drowning rescue and reducing injuries and deaths.

[0096] 5. The present invention combines drone cruising with infrared thermal imaging technology and can be widely used in various high-incidence drowning accident scenarios such as sea, rivers, and lakes, providing an innovative solution for water safety monitoring and drowning accident prevention. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 is a flow chart of the overall method steps of the present invention;

[0098] Figure 2 is a method step flow chart of step S2 of the present invention;

[0099] Figure 3is a method step flow chart of step S3 of the present invention;

[0100] Figure 4 is a method step flow chart of step S4 of the present invention;

[0101] Figure 5 This is a method step flow chart of step S5 of the present invention. DETAILED DESCRIPTION

[0102] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0103] Embodiment 1:

[0104] like Figure 1 As shown, a drowning visual recognition method integrating infrared temperature difference features provided by an embodiment of the present invention includes the following steps:

[0105] S1. Image acquisition and feature extraction: When the drone is cruising, it is equipped with a visible light camera and infrared thermal imaging equipment to collect visible light images and infrared thermal images within the cruising range in real time;

[0106] S2. Edge feature map extraction: Preprocess the infrared thermal image and construct a neural impulse response network to extract edge information with significant changes in thermal radiation and highlight areas with significant changes in thermal intensity, thereby helping to detect the outline and important structures of the drowning person;

[0107] S3, shape feature map extraction: by performing multi-scale rectangular feature extraction and dynamic weighted screening on the input image, a strong classifier is constructed by combining feature combinations to further analyze the shape of objects in the image and effectively identify the body shape of the drowning person;

[0108] S4, color feature map extraction: multi-layer encoding of visible light images is performed through convolutional neural networks to extract color and texture features to enhance target detection in low-light environments;

[0109] S5, feature map fusion and segmentation: the edge feature map, shape feature map and color feature map extracted from steps S2-S4 are input into the deep learning model to generate a unified instance feature map, the outline of the drowning person is accurately extracted using the graph segmentation technology, and the target is locked by combining the heat map and visible light information;

[0110] S6. Drowning person instance identification and alarm triggering: Real-time alarm is issued for detected drowning person instances, and the alarm mechanism is triggered by the drowning person's location and status, and the drowning person's location and detection confidence are transmitted to the ground control station.

[0111] Specifically, Figure 2 As shown, the specific method steps of step S2 are as follows:

[0112] S21. First, the input infrared thermal image is standardized. Infrared thermal images are generally directly obtained by thermal sensors, where the value of each pixel represents the thermal radiation intensity at that location. Infrared thermal images are usually grayscale images. Therefore, before further processing, the image is normalized to ensure that its pixel values ​​are within a certain range. Each pixel value is scaled to the range of [0,1] to represent the relative value of the thermal intensity. Assume that the input infrared thermal image is, where represents the pixel coordinates of the image and the image size is. The image after standardization can be expressed as:

[0113] Among them, I min and I max and are the minimum and maximum pixel values ​​in the image, respectively;

[0114] S22. Construct a neuron impulse response network and extract edge features using changes in neuron membrane potential and neighborhood coupling relationships. The output of each neuron is determined by its input image value, local coupling strength, and global threshold. Let v(x,y,t) represent the membrane potential of the neuron at position (x,y) at time step t, v threshold is the threshold of the neuron, c(x,y) is the coupling strength of the neuron, and I'(x,y) is the intensity value of the input image. Then the dynamic equation of the neuron can be expressed as:

[0115]

[0116] Where N(x,y) is the neighborhood pixels around the position (x,y), and λ is the attenuation coefficient;

[0117] S23, when the neuron's membrane potential v(x,y,t) reaches the threshold v threshold When , the neuron will generate an impulse response, indicating that the feature of the position is activated, and the pulse generation rule is:

[0118]

[0119] Among them, δ(x,y,t) represents the pulse response. If the membrane potential exceeds the threshold, a pulse signal is output;

[0120] S24. The pulse response graph is used to simulate the coupling effect between neurons. By calculating the cumulative output of the pulse, the area where the thermal intensity changes dramatically in the image is highlighted, thereby generating an edge response graph E(x,y). Where T is the total time steps of pulse propagation, and E(x,y) represents the edge strength of the image at position (x,y). The higher the strength value, the more likely the position is an edge position.

[0121] S25, denoising and smoothing the preliminary edge information image obtained by accumulating the impulse response to generate a high-quality edge feature map F(x, y), the specific steps are as follows:

[0122] Denoising: By setting the threshold T of edge strength edge , remove the response intensity below the threshold T edge Pulse points to remove possible noise;

[0123] Edge smoothing: Smoothing is performed to remove isolated pixels and ensure edge continuity. The edge feature map F(x,y) is expressed as:

[0124] Where T edge is the threshold of edge strength. If the edge strength exceeds the threshold, the position is considered to be an edge. The final output edge feature map F edge (x, y) clearly shows the area with drastic changes in thermal intensity in the image, which is used for subsequent drowning person detection and target recognition.

[0125] Specifically, Figure 3 As shown, the specific method steps of step S3 are as follows:

[0126] S31. Standardize and denoise the infrared thermal image to ensure input stability. The specific steps are as follows:

[0127] S311, normalization processing: normalize the pixel values ​​to the interval [0,1], and the normalized image of the original image I(x,y) is I norm (x,y), then Among them, min(I) and max(I) are the minimum and maximum pixel values ​​in the image respectively;

[0128] S312, noise removal: Gaussian filtering is applied to reduce image noise, and the smoothed image I smooth (x,y) is: Where G(x,y,σ) is a filter with a standard deviation of σ;

[0129] S32, extracting rectangular features of different scales and directions on the preprocessed image, extracting local brightness features and constructing feature vectors, the specific steps are as follows:

[0130] S321, define rectangular areas: define several rectangular areas around each pixel point (x, y), namely R1, R2, ..., R m ,These rectangular regions vary in size, location, and orientation to capture a variety of shape features;

[0131] S322, brightness difference calculation: For each rectangular area pair (R i ,R j ), calculate the brightness difference:

[0132] where f ij (x,y) represents the position

[0133] The eigenvalues ​​of (x,y) are obtained by the rectangular area pair (R i , R j ) to describe the local brightness difference of the image. The model can effectively capture the different morphological and structural information of the image, providing a basis for subsequent classification and detection tasks.

[0134] S323, feature vector construction: At each pixel position, according to all rectangular features f ij Generate the feature vector F(x,y), that is: F(x,y)=[f 11 (x,y),f 12 (x,y),…,f mn (x,y)]; eigenvector

[0135] The dimension of F(x,y) is related to the number of selected rectangular feature combinations.

[0136] S33. In order to improve the detection accuracy, a dynamic weighting mechanism is introduced to assign different weights to each rectangular feature and gradually optimize the weights through multiple iterations. The specific steps are as follows:

[0137] S331, initial weight allocation: Set the initial weights of all features to be the same, that is, for the kth feature, the initial weight is:

[0138] S332, Error calculation and weight adjustment: In each round of iteration, select the feature combination with the smallest error and adjust its weight. Suppose feature f is selected in round t. ij As the main feature, its classification error is ε ij , then the corresponding weight update is: in,

[0139] is the adjustment coefficient, h ij Based on the feature f ijThe weak classifier output, l() represents the indicator function, which is used to determine whether the classification is correct;

[0140] S333. After updating the weights of all features, normalization is performed:

[0141] Through this dynamic weight update, the importance of features gradually focuses on features with better performance.

[0142] S34, combining multiple weighted features into a series of weak classifiers, and generating the final strong classifier through weighted integration. The specific steps are as follows:

[0143] S341, weak classifier construction: for each feature f ij Construct weak classifier h ij (x, y), the output is 1 or -1, indicating whether the position matches the target shape. The definition of the weak classifier is:

[0144] Among them, θ ij The feature f ij The classification threshold of

[0145] S342, strong classifier integration: All weak classifiers are combined into a strong classifier H(x,y) by weighted summation, which is defined as: Among them, α ij is the weight of the weak classifier, sign() represents the sign function, output 1 means the shape feature is detected, output -1 means it is not detected;

[0146] S35, use the strong classifier H(x,y) to detect each pixel and output the shape feature map F shape (x,y), the specific steps are:

[0147] S351, feature map generation: First use the strong classifier H(x,y) to determine whether each pixel (x,y) belongs to the target shape area:

[0148] S352, post-processing: In order to improve the continuity of the feature map, morphological operations are performed on the output shape feature map, including dilation and erosion operations, to eliminate isolated points and smooth the boundaries.

[0149] Specifically, Figure 4 As shown, the specific method steps of step S4 are as follows:

[0150] S41, normalizing and expanding the boundary of the input visible light image to ensure image consistency and integrity;

[0151] Normalization: Map each pixel value of the visible light image to the interval [0,1] to ensure that pixels of different intensities can be processed in the same numerical range. Let the input image be I RGB (x,y), its normalized image I norm (x,y) is represented as:

[0152]

[0153] Among them, min(I RGB ) and max(I RGB ) are the minimum and maximum pixel values ​​of the image, respectively.

[0154] Boundary expansion: In order to avoid edge information loss during convolution operations, symmetric padding is used to expand the image boundary.

[0155] S42. In the encoding stage, the input image is passed through a series of convolution and pooling operations layer by layer to generate image feature representations of different scales. The feature maps of each layer gradually focus on different color information and texture information from shallow to deep layers. Assuming that the input image size is H×W, the specific encoding process is as follows:

[0156] S421, convolution layer: Each convolution layer uses convolution kernel K to extract local features of the image, and the size of the convolution kernel in the lth layer is k l ×k l , the convolution kernel weight is W l , with a bias of b l , the output of the lth layer convolution is:

[0157] Among them, σ is a nonlinear activation function (such as ReLU), I in (x,y) is the input feature map;

[0158] S422, Pooling layer: After each convolution layer, the pooling operation is applied to reduce the spatial dimension of the feature map and improve the computational efficiency. Assuming the pooling window size is p×p, the feature map after pooling is:

[0159] P l (x,y)=max u,v∈p×p F l (x+u,y+v); The pooling operation reduces the spatial resolution of the feature map layer by layer and retains the main activation values ​​of each area.

[0160] S423, multi-layer coding: Repeat convolution and pooling operations to build a multi-layer coding structure. Assume that the number of coding layers is L, then the final generated coding feature map is represented as F L (x, y), this feature map contains the color and texture information of the original image at different scales and depths;

[0161] S43. In order to enhance the context information in color feature extraction, an aggregation layer is designed to concentrate the multi-scale color features into one representation by feature fusion. The specific steps are as follows:

[0162] S431, skip connection: introduce a skip connection between the encoding layer and the decoding layer to retain the shallow feature information. The output of the skip connection is: C l (x,y)=P l (x,y)+D l (x,y), where P l (x,y) and D l (x, y) represents the feature map of the lth layer in the encoding stage and the decoding stage respectively. The skip connection ensures that the high-resolution color information is not lost;

[0163] S432, context fusion: concatenate the feature maps of different layers in the encoding stage to form a context feature representation: C context (x,y)=concat{C1(x,y),C2(x,y),...,C l (x,y)};

[0164] S44, restore the feature map to the resolution of the original image by gradually upsampling, thereby extracting complete color feature information. The decoding layer corresponds to the encoding layer, and reconstructs a high-resolution color feature map in combination with context information. The specific steps are as follows:

[0165] S441, upsampling operation: Each layer restores the low-resolution feature map to high resolution by upsampling. Assuming the upsampling factor is s, the upsampling output of the lth layer is:

[0166] S442, decoding convolution layer: After each upsampling, convolution is applied to extract details. The output of the decoding convolution layer is:

[0167] Among them, W d and b d are the weights and biases of the convolution kernels in the decoding layer, k d is the size of the decoding convolution kernel, σ is the activation function;

[0168] S45, after layer-by-layer upsampling and decoding convolution operations, the final output is the color feature map F color (x,y), this feature map retains the color information of the input image and is suitable for subsequent feature fusion, including:

[0169] Feature map generation: The generation formula of the color feature map is:

[0170] Among them, the softmax function normalizes the output into a probability distribution to enhance the color contrast information of the image.

[0171] Post-processing: After the color feature map is generated, a refinement operation is applied to make the feature edges smoother. The final generated color feature map can retain the color information of the target object and effectively support subsequent multimodal fusion processing.

[0172] Specifically, Figure 5 As shown, the specific method steps of step S5 are as follows:

[0173] In the above steps, the edge feature maps F and F of the infrared thermal image are extracted respectively. edge (x, y), shape feature map F of infrared thermal image shape (x, y) and the color feature map F of the visible light image color (x,y). This step aims to effectively combine the above three feature maps through the feature weighted fusion method, and further segment the fused features to accurately extract the instance of the drowning person.

[0174] S51. Linear normalize the edge, shape, and color feature maps respectively to ensure that the data range is consistent and avoid weight deviation due to numerical differences. The specific formula is as follows:

[0175]

[0176] Among them, F' edge (x,y), F' shape (x,y), F' color (x, y) are normalized edge, shape and color feature maps respectively;

[0177] S52. In order to reasonably reflect the importance of each feature map in the fusion process, feature weight coefficients α, β, and γ are introduced to correspond to edge, shape, and color feature maps respectively. The weight coefficients are determined by optimizing the training data and satisfy α+β+γ=1, where α, β, γ∈[0,1];

[0178] S53, weighted fusion is performed on the normalized edge, shape and color feature maps according to the weight coefficient to generate a fused feature map F fuse (x,y),

[0179] F fuse (x,y)=α·F e ' dge (x,y)+β·F s ' hape (x,y)+γ·F c ' olor (x,y), where F fuse(x, y) combines the feature information of edge, shape and color, providing a rich and coordinated feature basis for subsequent segmentation.

[0180] S54, use the high-pass filter H(x,y) to fusion feature map F fuse (x, y) is convolved to obtain the enhanced feature map F enhance (x,y), F enhance (x,y)=F fuse (x,y)*H(x,y), where * represents the convolution operation;

[0181] S55. First, the image is modeled as a graph G = (V, E), where V is a set of pixels, E is the connecting edge between pixels, and the weight w(i, j) of each edge is based on the feature similarity and spatial proximity between pixels, and is defined as:

[0182] Among them, σ and d are parameters that control feature similarity and spatial distance. The minimum cut of the graph G is calculated by the minimum cut maximum flow theory to obtain the segmentation result F graph (x,y),

[0183]

[0184] S56, the graph cut segmentation result F obtained by the segmentation algorithm graph (x,y) needs to be further processed to extract clear instances of drowning people. The specific steps are as follows:

[0185] S561, morphological processing: Apply dilation and erosion operations to fill holes in the segmentation results, remove isolated noise points, and enhance the connectivity of the target area. The processing formula is:

[0186] in, represents the expansion operation, represents the corrosion operation, B and C are structural elements respectively;

[0187] S562, connected domain analysis: Perform connected domain analysis on the results after morphological processing to identify and extract independent target areas. If N connected domains are detected, each connected domain R i The pixel set is:

[0188] R i ={(x,y)|F morph (x,y)=1 and (x,y)∈R i};

[0189] S563, Target Verification: Connectivity domain R i Verify the size and shape, exclude false detection and non-target areas, and set the area threshold Amin and Amax , only keep those that meet condition A min ≤|R i |≤ Amax The connected domain of is taken as a valid drowning person instance, where |R i | is the connected domain R i The area (number of pixels), A min and Amax Set according to the actual size range of the target instance.

[0190] S57. To improve the adaptability of the system in different environments, dynamically adjust the feature weight coefficients α, β and γ to improve the accuracy of feature weighted fusion. Set up a feedback mechanism to adjust the weight coefficients by real-time monitoring of the matching degree between the detection results and the actual situation. The specific adjustment formula is as follows:

[0191] α (t+1) =α (t) +Δα,β (t+1) =β (t) +Δβ,γ (t+1) =γ (t) +Δγ;

[0192] Among them, Δα, Δβ and Δγ are adjustment increments, which are dynamically adjusted according to the detection accuracy and false detection rate to ensure the stability and accuracy of the fusion method under different waters and lighting conditions. The adjustment principle must meet α (t+1) +β (t+1) +γ (t+1) =1,α (t+1) ,β (t+1) ,γ (t+1) ∈[0,1],

[0193] Dynamic adjustments are made based on the detection accuracy and false detection rate to ensure the stability and accuracy of the fusion method under different water and lighting conditions.

[0194] Through the above steps, the present invention realizes the effective fusion and segmentation of multiple feature maps, significantly improves the accuracy and real-time performance of drowning detection, overcomes the defects of traditional monitoring methods such as slow response speed and limited coverage, and gives full play to the advantages of drone maneuverability and multi-sensor image processing.

[0195] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A drowning visual recognition method integrating infrared temperature difference features, characterized in that: The following steps are involved: S1. Image acquisition and feature extraction: When the drone is cruising, it is equipped with a visible light camera and infrared thermal imaging equipment to collect visible light images and infrared thermal images within the cruising range in real time; S2. Edge feature map extraction: Preprocess the infrared thermal image and construct a neural impulse response network to extract edge information with significant changes in thermal radiation and highlight areas with significant changes in thermal intensity, thereby helping to detect the outline and important structures of the drowning person; S3, shape feature map extraction: by performing multi-scale rectangular feature extraction and dynamic weighted screening on the input image, a strong classifier is constructed by combining feature combinations to further analyze the shape of objects in the image and effectively identify the body shape of the drowning person; S4, color feature map extraction: multi-layer encoding of visible light images is performed through convolutional neural networks to extract color and texture features to enhance target detection in low-light environments; S5, feature map fusion and segmentation: The edge, shape and color feature maps extracted in steps S2-S4 are input into the deep learning model for fusion to generate a unified instance feature map, and the outline of the drowning person is accurately extracted using the graph segmentation technology, and the target is locked by combining the heat map and visible light information; S6. Drowning person instance identification and alarm triggering: Real-time alarm is issued for detected drowning person instances, and the alarm mechanism is triggered by the drowning person's location and status, and the drowning person's location and detection confidence are transmitted to the ground control station.

2. The drowning visual recognition method integrating infrared temperature difference features according to claim 1 is characterized in that: The specific method steps of step S2 are as follows: S21, performing standardization processing on the input infrared thermal image, scaling the pixel values ​​in the image to the range of [0, 1], and eliminating the difference by adjusting the ratio of the minimum and maximum pixel values; S22. Construct a neuron impulse response network and extract edge features using changes in neuron membrane potential and neighborhood coupling. The dynamic equation of the neuron is: Where v(x,y,t) represents the membrane potential of the neuron at position (x,y) at time step t, v threshold is the threshold of the neuron, c(x,y) is the coupling strength of the neuron, I'(x,y) is the intensity value of the input image, N(x,y) is the neighborhood pixels around the position (x,y), and λ is the attenuation coefficient; S23, when the neuron's membrane potential v(x,y,t) reaches the threshold v threshold When , the neuron will generate an impulse response, indicating that the feature of the position is activated, and the pulse generation rule is: Among them, δ(x,y,t) represents the pulse response. If the membrane potential exceeds the threshold, a pulse signal is output; S24. The pulse response graph is used to simulate the coupling effect between neurons. By calculating the cumulative output of the pulse, the area where the thermal intensity changes dramatically in the image is highlighted, thereby generating an edge response graph E(x,y). Where T is the total time steps of pulse propagation, and E(x,y) represents the edge strength of the image at position (x,y); S25, denoising and smoothing the preliminary edge information image obtained by accumulating the impulse response to generate a high-quality edge feature map F(x, y), the specific steps are as follows: By setting the edge strength threshold T edge , remove the response intensity below the threshold T edge The pulse points are smoothed to remove isolated pixels to ensure edge continuity. The edge feature map F(x,y) is expressed as: The final output edge feature map F edge (x, y) clearly shows the area with drastic changes in thermal intensity in the image, which is used for subsequent drowning person detection and target recognition.

3. The drowning visual recognition method integrating infrared temperature difference features according to claim 1 is characterized in that: The specific method steps of step S3 are as follows: S31. Standardize and denoise the infrared thermal image to ensure input stability. The specific steps are as follows: S311, normalize the pixel values ​​to the interval [0,1], and the normalized image of the original image I(x,y) is I norm (x,y), then Among them, min(I) and max(I) are the minimum and maximum pixel values ​​in the image respectively; S312, apply Gaussian filtering to reduce image noise, then the smoothed image I smooth (x,y) is: Where G(x,y,σ) is a filter with a standard deviation of σ; S32, by calculating the brightness difference of rectangular areas of different scales and directions, extracting local brightness features and constructing feature vectors, the specific steps are as follows: S321, define multiple rectangular areas R1, R2, ..., R near the pixel point (x, y) m , covering different sizes, orientations and locations; S322, calculate the rectangular area pair (R i , R j )’s brightness difference: where f ij (x,y) represents the position The eigenvalues ​​of (x,y) are obtained by the rectangular area pair (R i , R j ) is calculated by the brightness difference As a result, the local brightness differences of the image are described; S323, according to all rectangular eigenvalues ​​f ij , generate a feature vector F(x,y) at each pixel position, It is expressed as: F(x,y)=[f 11 (x,y),f 12 (x,y),…,f mn (x,y)]; S33, introduce a dynamic weighting mechanism to assign different weights to each rectangular feature, and improve detection accuracy by iteratively optimizing feature weights. The specific steps are as follows: S331. Initialize the weights of all features to be the same. For the kth feature, the initial weight is: S332. In each round of iteration, select the feature combination with the smallest error and update its weight. Select feature f in round ij As the main feature, its classification error is ε ij , then the updated weight is: in, is the adjustment coefficient, h ij Based on the feature f ij The weak classifier output, l() represents the indicator function, which is used to determine whether the classification is correct; S333. After updating the weights of all features, normalization is performed: S34, combining weighted features into weak classifiers, and generating strong classifiers through weighted integration, the specific steps are as follows: S341, for each feature f ij Construct weak classifier h ij (x, y), the output is 1 or -1, indicating whether the position matches the target shape. The definition of the weak classifier is: Among them, θ ij The feature f ij The classification threshold of S342. All weak classifiers are combined into a strong classifier H(x,y) by weighted summation, which is defined as: Among them, α ij is the weight of the weak classifier, sign() represents the sign function, output 1 means the shape feature is detected, output -1 means it is not detected; S35, use the strong classifier H(x,y) to detect each pixel and output the shape feature map F shape (x,y), the specific steps are: S351, first use the strong classifier H(x,y) to determine whether each pixel (x,y) belongs to the target shape area: S352, then perform morphological processing on the output shape feature map, including dilation and erosion operations, to eliminate isolated points and smooth boundaries to obtain the final target detection result.

4. The drowning visual recognition method integrating infrared temperature difference features according to claim 1 is characterized in that: The specific method steps of step S4 are as follows: S41, normalizing and expanding the boundary of the input visible light image to ensure image consistency and integrity; S42, extract the color and texture features of the image layer by layer through convolution and pooling operations to generate multi-level feature representation. The specific encoding process is as follows: S421, each convolution layer uses convolution kernel K to extract local features of the image, and the size of the convolution kernel in the lth layer is k l ×k l , the convolution kernel weight is W l , with a bias of b l , the output of the lth layer convolution is: Among them, σ is a nonlinear activation function, I in (x,y) is the input feature map; S422, apply pooling operation after convolution to reduce the spatial dimension. Set the pooling window size to p×p. The feature map after pooling is: P l (x,y)=max u,v∈p×p F l (x+u,y+v); S423, gradually building a multi-layer coding structure through multiple convolution and pooling operations to generate image feature maps of different scales and depths; S43, enhancing the context information of color feature extraction by skip connection and context fusion, the specific steps are as follows: S431, introduce a skip connection between the encoding layer and the decoding layer to retain the shallow feature information, and the skip connection output is: C l (x,y)=P l (x,y)+D l (x,y), where P l (x,y) and D l (x, y) represents the feature map of the lth layer in the encoding stage and the decoding stage respectively. The skip connection ensures that the high-resolution color information is not lost; S432, concatenate the feature maps of different layers in the encoding stage to form a context feature representation: C context (x,y)=concat{C1(x,y),C2(x,y),...,C l (x,y)}; S44, gradually upsampling and convolution to restore the image resolution and extract complete color features, the specific steps are as follows: S441, each layer restores the low-resolution feature map to high resolution by upsampling. Assuming the upsampling factor is s, the upsampling output of the lth layer is: S442, after each upsampling, convolution is applied to extract details, and the output of the decoded convolution layer is: Among them, W d and b d are the weights and biases of the convolution kernels in the decoding layer, k d is the size of the decoding convolution kernel, σ is the activation function; S45, generate color feature map F by upsampling and decoding convolution color (x,y), and use the softmax function to enhance the color contrast information. After generating the feature map, the edges are smoothed through a refinement operation to preserve the color information of the target object in preparation for multimodal fusion.

5. The drowning visual recognition method integrating infrared temperature difference features according to claim 1 is characterized in that: The specific method steps of step S5 are as follows: S51. Linear normalize the edge, shape, and color feature maps respectively to ensure that the data range is consistent and avoid weight deviation due to numerical differences. The specific formula is as follows: Among them, F' edge (x,y), F' shape (x,y), F' color (x, y) are normalized edge, shape and color feature maps respectively; S52, introducing feature weight coefficients α, β, and γ corresponding to edge, shape, and color feature maps respectively, the weight coefficients are determined by optimizing the training data, and satisfy α+β+γ=1, where α, β, γ∈[0,1]; S53, weighted fusion is performed on the normalized edge, shape and color feature maps according to the weight coefficient to generate a fused feature map F fuse (x,y), F fuse (x,y)=α·F e ′ dge (x,y)+β·F s ′ hape (x,y)+γ·F c ′ olor (x,y); S54, use a high-pass filter to fusion feature map F fuse (x, y) is convolved to obtain the enhanced feature map F enhance (x,y); S55. Model the image as a graph G = (V, E), where V is a set of pixels, E is the connecting edge between pixels, and the weight w(i, j) of each edge is based on the feature similarity and spatial proximity between pixels, and is defined as: Among them, σ and d are parameters that control feature similarity and spatial distance. The minimum cut of the graph G is calculated by the minimum cut maximum flow theory to obtain the segmentation result F graph (x,y), 6. The drowning visual recognition method integrating infrared temperature difference features according to claim 5 is characterized in that: After step S55, the following step is further included: step S56, based on the graph cut segmentation result F graph (x,y) is further processed to extract clear drowning person instances. The specific steps are as follows: S561, apply expansion and corrosion operations to fill the holes in the segmentation results, remove noise points, and enhance the connectivity of the target area. The processing formula is: in, represents the expansion operation, represents the corrosion operation, B and C are structural elements respectively; S562, the result after morphological processing F morph (x, y) performs connected domain analysis to identify and extract independent target areas. If N connected domains are detected, each connected domain R i The pixel set is: R i = {(x, y)|F morph (x, y) = 1 and (x, y) ∈ R i}; S563, for the connected domain R i Verify the size and shape, exclude false detection and non-target areas, and set the area threshold A min and Amax , only keep those that meet condition A min ≤|R i |≤ Amax The connected domain of is taken as a valid drowning person instance.

7. The drowning visual recognition method integrating infrared temperature difference features according to claim 6 is characterized in that: After step S56, step S57 is also included, according to the matching degree between the detection result and the actual situation, the weight coefficients α, β and γ of the feature fusion are dynamically optimized to improve the accuracy of the feature weighted fusion. The specific adjustment formula is: a (t+1) =a (t) +Dα,β (t+1) =b (t) +Δβ,γ (t+1) =c (t) +Δγ; Among them, Δα, Δβ, and Δγ are adjustment increments, and α (t+1) + β (t+1) + γ (t+1) = 1, and α (t+1) , β (t+1) , γ (t+1) ∈ [0, 1].

Images