Multi-modal data fusion human body target identification method based on automatic control

Abstract

The invention relates to the technical field of target identification based on multi-modal data fusion, and aims to solve the problems that an existing life detection system is difficult to effectively distinguish spatial distribution of a human body target and a hot pseudo target and cannot realize real-time synchronous analysis of multi-source heterogeneous data, so that the target positioning reliability in an extreme environment is seriously restricted. The invention provides a multi-modal data fusion human body target recognition method based on automatic control, and the method comprises the steps: constructing a feature complementation and interference suppression model based on a deep neural network through fusing the cross-modal data of infrared thermal imaging and visible light imaging, and employing a multi-scale anti-interference network based on wavelet graph convolution and an attention mechanism to recognize a human body target. The environment noise broadband interference is suppressed through the time-frequency localization characteristic, high-robustness recognition of a human body target in severe environments such as dense smoke shielding and thermal radiation interference is achieved, and accurate distinguishing of the human body and a thermal pseudo target in a complex fire scene environment is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data fusion target recognition, and in particular to a multimodal data fusion human target recognition method based on automatic control. Background Art

[0002] Mine fires, due to their sudden nature and complex environments, have become a core risk threatening coal mine safety. In extreme fire environments, where visibility approaches zero and high-temperature radiation is intertwined with toxic smoke, modern intelligent firefighting equipment has introduced sensing technologies such as infrared thermal imaging and multispectral detection. However, practical applications still face multiple technical bottlenecks: First, dense smoke media differentially attenuates different spectral signals, leading to a sharp deterioration in the data quality of a single optical sensor. Second, sensor signal drift caused by dynamic thermal radiation fields results in nonlinear distortion of heat source target characteristics. Third, the spectral overlap between thermal pseudo-targets and human thermal radiation characteristics in high-temperature environments significantly interferes with the specificity of target recognition.

[0003] Existing life detection systems generally use independent analysis of visible light and infrared sensors, which expose defects under complex conditions such as thick smoke obscuration and heat flow disturbances: visible light imaging is easily affected by the scattering effect of smoke, and infrared thermal imaging is sensitive to dynamic thermal radiation interference and prone to feature confusion. It is difficult to effectively distinguish the spatial distribution of human targets and thermal pseudo-targets, and it is impossible to achieve real-time synchronous analysis of multi-source heterogeneous data, which seriously restricts the reliability of target positioning in extreme environments.

[0004] Traditional multimodal fusion methods, such as RGB-D camera collaboration, rely on the spatial consistency of visible light and depth information. However, in dense smoke, the Mie scattering effect of smoke particles on visible light causes the image signal-to-noise ratio to drop below -10dB, far exceeding the noise level of ordinary smog scenes. Furthermore, the dynamic diffusion of high-temperature air waves at a fire scene creates a non-uniform thermal radiation field in infrared thermal imaging, causing spectral aliasing of the thermal radiation signatures of human targets (32-42°C) and high-temperature metal structures in the 8-14μm band. This fundamentally differs from medical infrared imaging (where the temperature difference is stable within ±2°C) or industrial inspection (where the target temperature difference is >50°C).

[0005] The effectiveness of existing cross-modal alignment methods in common scenarios presupposes a stable geometric correspondence between the modalities. However, dense smoke in a fire can cause nonlinear optical distortion in visible light images, while thermal convection in infrared sensors can cause thermal shift. This can lead to spatial misalignment errors of up to 12-15 pixels, far exceeding the tolerance for multi-sensor fusion in autonomous driving (typically less than 3 pixels). This unstructured misalignment renders traditional modal alignment methods based on feature point matching completely ineffective.

[0006] Therefore, it is urgent to build a multimodal fusion recognition method with anti-interference ability. By establishing an analysis model driven by the complementary characteristics of visible light and infrared sensors and deep learning, we can break through the technical bottlenecks such as the strong heterogeneity of sensor data, high diversity of interference sources, and strict real-time response requirements in complex fire environments, and provide a core technical foundation for improving the environmental adaptability and actual combat effectiveness of intelligent rescue equipment. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the present invention provides a method for human target recognition based on multimodal data fusion using automated control. This method addresses technical issues such as low recognition accuracy caused by single sensors being susceptible to environmental interference and weak correlation between multimodal features in complex scenarios. By fusing cross-modal data from infrared thermal imaging and visible light imaging, the present invention constructs a feature complementation and interference suppression model based on a deep neural network, achieving highly robust recognition of human targets in harsh environments such as those obscured by dense smoke and interference from thermal radiation.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions: A multimodal data fusion human target recognition method based on automatic control includes the following steps: S1. Obtain smoke images of the fire scene collected by a CMOS camera and an infrared thermal imaging camera, respectively. Use a contour extraction algorithm to preprocess the visible light image and infrared thermal imaging image of the fire scene, and perform modal alignment and fusion on the visible light image and infrared thermal imaging image. S2. Build a multimodal anti-interference human target recognition network model based on convolutional neural network; S3, using the image after modal alignment and fusion in step S1 as input, training the human target recognition network model in step S2 to obtain an optimal human target recognition network model; S4. Use the optimal human target recognition network model to perform human target recognition on images collected in real fire and smoke scenes.

[0009] Furthermore, in step S1, a contour extraction algorithm is used to pre-process the visible light image and infrared thermal imaging image of the fire scene. The specific process includes the following steps: S11. Use a CMOS camera to dynamically capture visible light image streams at the fire scene, ensuring that the dynamic range of illumination intensity in the target area meets the requirements for analyzing human texture features and retaining edge details of the human body contour. S12. Use an infrared thermal imaging camera to collect thermal radiation distribution images of the target area to ensure that the radiation energy difference between the human target and the high-temperature background can be resolved; S13. The HOG algorithm and Canny edge detector are used to segment the contours of human targets in visible light images and infrared thermal images to improve the detection accuracy of human targets in dense smoke environments.

[0010] Furthermore, in step S13, the HOG algorithm and the Canny edge detector are used to segment the contour of the human target. The specific process includes the following steps: S131, after converting the visible light and thermal imaging images into grayscale images, using the oriented gradient histogram as a feature to describe the human body contour; S132, establishing a sliding window of fixed size in the grayscale image, traversing the entire grayscale image with a preset step size, calculating the directional gradient histogram within each sliding window, concatenating the gradient histograms of the sliding windows, and obtaining the HOG feature vector of the image interval of each sliding window; S133, using the Gaussian filter of the Canny algorithm to smooth the input visible light image and reduce the interference of high-frequency noise; S134, performing threshold processing on the filtered image, refining the edges in the visible light image, setting a high threshold and a low threshold, and segmenting the visible light image; S135, connect the edges in the visible light image based on the HOG feature vector, traverse the entire image, add the strong pixel points in the low threshold gradient image to the high threshold gradient image, and obtain the final edge gradient amplitude image .

[0011] Furthermore, in step S1, the visible light image and the infrared thermal imaging image are modally aligned and fused. Construct a modality fusion module for visible light images and infrared thermal imaging images based on a bilinear pooling network: ,in is the feature representation of the visible light image, is the mapping matrix, is the characteristic representation of infrared thermal imaging image, and the obtained Feature representation that fully reflects the essential characteristics of visible light image and infrared thermal imaging image data; Before the image features are input into the bilinear pooling network, an adaptive learning mechanism is introduced to better extract features from visible light images and infrared thermal imaging images.

[0012] Furthermore, in step S2, a multimodal anti-interference human body recognition network model based on a convolutional neural network is constructed. The specific process includes the following steps: S21. Use a wavelet graph convolutional neural network with an attention mechanism to perform multi-scale learning on the input visible light image and infrared thermal imaging image, and use the time-frequency localization characteristics of the wavelet graph convolutional neural network to suppress the broadband interference of environmental noise; S22. Construct a multi-view fusion module suitable for human target recognition to extract the consistency and complementary information between visible light images and infrared thermal images; S23. After fusion feature extraction of visible light images and infrared thermal imaging images, a cross-entropy loss function of a multimodal anti-interference human target recognition network is constructed to obtain heterogeneous features of visible light images and infrared thermal imaging images.

[0013] Furthermore, in step S21, a wavelet graph convolutional neural network with an attention mechanism is used to perform multi-scale learning on the input image. The specific process includes the following steps: S211. Extract key distinguishing features of images using wavelet graph convolutional networks; S212. Construct a propagation method between adjacent layers of a wavelet graph convolutional network to adjust neighbor nodes of different scales. S213. Use the attention mechanism configured in the wavelet graph convolutional network to enhance the distinctive features of the image.

[0014] Furthermore, in step S22, a multi-view fusion module suitable for human target recognition is used to extract consistency and complementarity information between the visible light image and the infrared thermal image. The specific process includes the following steps: S221. Use convolution kernels with different dilation rates to extract information from different receptive fields in visible light images and infrared thermal images. Add dense connection operations to the convolution operations between layers to enhance the flow of information between layers. S222, the feature extraction module consists of a 1×1 standard convolution and three 3×3 dilated convolutions in parallel. It captures heterogeneous features through multi-scale receptive fields and then fuses them through concat, and finally uses a 1×1 standard convolution to reduce the dimension of the output; S223, the dense connection structure adopts a cascade structure of three 3×3 convolutions + LeakyReLU, which alleviates the gradient attenuation of the deep network through the cross-layer feature reuse mechanism, and realizes information fusion through inter-layer feature stacking and 1×1 convolution.

[0015] Furthermore, in step S3, the human target recognition network model is trained to obtain the optimal human target recognition network model, and the specific process includes the following steps: S31. Obtain sample data for training a human target recognition network model, where the sample data is a visible light image and an infrared thermal imaging image of a simulated fire scene collected synchronously in real time; S32. Use the LabelImg annotation tool to manually annotate human targets in the image. Expand the dataset by random rotation, grayscale value perturbation, and Gaussian noise injection to improve the robustness of the multimodal anti-interference recognition network model to noise, brightness differences, and smoke. S33, dividing the expanded data set into a training set, a validation set, and a test set; S34. A training set is used to train the weight coefficients of the human target recognition network model, a validation set is used to verify the reliability of the weight coefficients and optimize the weight coefficients, and a test set is used to test the reliability of the weight coefficients finally obtained.

[0016] In summary, the invention has the following beneficial effects: 1. The present invention solves the problems of data quality degradation and weak correlation of multimodal features of a single sensor under thick smoke obscuration and thermal radiation interference through cross-modal complementary fusion of visible light texture features and infrared thermal radiation features. It significantly improves the accuracy and anti-interference ability of human target recognition, and realizes the accurate distinction between human bodies and thermal pseudo-targets in complex fire environments.

[0017] 2. The present invention adopts a multi-scale anti-interference network based on wavelet graph convolution and attention mechanism to suppress broadband interference of environmental noise through time-frequency localization characteristics, effectively overcome the interference of dynamic thermal radiation field signal drift and spectrum overlap, and enhance the robustness of the model to complex interference sources such as high-temperature metal reflection and electromechanical equipment radiation.

[0018] 3. The present invention strengthens the expression of local details while retaining the macroscopic morphological features of the human body contour, and combines the enhancement methods of random rotation, grayscale value perturbation, and Gaussian noise injection to ensure the real-time and environmental reliability of human target recognition in mine fire rescue. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 Flowchart of the human target recognition method of the present invention; Figure 2 A diagram showing the human target recognition network structure using the present invention; Figure 3 This is the pre-training recognition effect diagram of the present invention; Figure 4 This is the human body recognition result diagram of the present invention using visible light and infrared thermal imaging in a real fire scene. DETAILED DESCRIPTION

[0020] To make the above-mentioned objectives, features, and advantages of the present invention more clearly understood, the present invention is further described below in detail with reference to the accompanying drawings and specific embodiments. This will enable a full understanding of how this application uses technical means to solve technical problems and achieve technical effects, and to implement the invention accordingly.

[0021] Please refer to Figures 1-4 , shows a specific implementation method of this embodiment. This embodiment, through the cross-modal complementary fusion of visible light texture features and infrared thermal radiation features, combined with a deep learning-driven anti-interference model, can achieve highly robust recognition of human targets in harsh environments such as thick smoke obscuration and thermal radiation interference.

[0022] Please refer to Figure 1 The present invention discloses a multimodal data fusion human target recognition method based on automatic control. By constructing a multimodal anti-interference recognition network, and combining it with an adaptive bilinear pooling network and a wavelet graph convolution network as partial network modules, and combining it with mathematical transformation and loss function to form a convolution network, the method performs human target recognition on images collected in real fire and smoke scenes. The method includes the following steps: S1. Obtain smoke images of the fire scene collected by the CMOS camera and the infrared thermal imaging camera respectively, and use the contour extraction algorithm to preprocess the visible light image and the infrared thermal imaging image.

[0023] S11. Use a CMOS camera to dynamically capture visible light image streams at the fire scene. Utilize an existing adaptive white balance algorithm to compensate for color temperature shifts caused by dense smoke. Adaptively adjust the aperture and shutter speed to ensure that the dynamic range of illumination intensity in the target area meets the requirements for analyzing human texture features and preserves edge details of the human body contour. S12. Millisecond-level time synchronization of cross-modal data is achieved through the GPS timing module and the IEEE 1588 precision clock protocol. An infrared thermal imager is used to capture thermal radiation distribution images of the target area, ensuring that the radiation energy difference between the human target (32-42°C) and the high-temperature background (>200°C) can be resolved. The peak wavelength of thermal radiation corresponding to the surface temperature of the human target (32-42°C) is in the mid- and far-infrared bands (9–10 μm), while the peak wavelength of radiation from the high-temperature background (>200°C) is in the short-wave infrared band (4–5 μm). This order of magnitude difference in radiation energy density and the separation characteristics of the radiation bands provide a resolvable physical basis for the thermal imaging system.

[0024] S13. Use the HOG algorithm and Canny edge detector to segment the contours of human targets to improve the detection accuracy of human targets in dense smoke environments.

[0025] S131, HOG algorithm uses the histogram of oriented gradients as the feature to describe the human body contour. When calculating the HOG feature vectors of visible light images and thermal imaging images, the visible light and thermal imaging images are first converted into grayscale images, and then the gradient image is calculated. The formula is as follows: Calculate the gradient for each pixel (x, y) in the grayscale image, where Represents the coordinate points in the grayscale image of the visible light image and the thermal imaging image The grayscale value of the pixel at and They are points The gradients in the x and y directions are calculated using the following formula: The gradient magnitude and gradient direction of : in for point The gradient amplitude of for point The gradient direction.

[0026] S132. Divide the grayscale image into nine tic-tac-toe intervals in spatial order. Create an 8×8 pixel sliding window within the grayscale image and traverse the entire image with a step size of 5 pixels. Compute the directional gradient histogram within each sliding window to obtain the gradient histograms of all sliding windows in the interval. Concatenate the gradient histograms of the sliding windows to obtain the HOG feature vector for each image interval. This ensures that the human target is fully covered at different locations.

[0027] S133, Canny algorithm first uses Gaussian filter to smooth the input visible light image to reduce the interference of high-frequency noise. The Gaussian filter formula is as follows: in, is the original visible light image at coordinates The pixel value of the position, is the filtered image, is a Gaussian function, and * represents a convolution operation.

[0028] S134. Threshold processing is performed on the filtered image to refine the edges in the visible light image. The Canny algorithm uses a hysteresis threshold method, which sets two thresholds, a high threshold and a low threshold, to segment the visible light image. The following formula is used for grayscale image segmentation: in Represents the gradient magnitude image (Gradient Magnitude Image), that is, the image after Gaussian filtering in step S133 . is a high threshold image, is a low threshold image, is the high threshold, is the low threshold, and the high threshold is usually three times the low threshold. If the pixel point is greater than the high threshold, the pixel value of the current pixel point is retained, otherwise it is set to zero, thus obtaining a high threshold gradient image; if the pixel point is greater than the low threshold but less than the high threshold, the pixel value of the current pixel point is retained, otherwise it is set to zero, thus obtaining a low threshold gradient image.

[0029] S135, connect the edges in the visible light image based on the HOG feature vector, and traverse the entire image. Finally, add the strong pixel points in the low threshold gradient image to the high threshold gradient image to obtain the final edge gradient amplitude image , which is the processed edge image obtained by calculation.

[0030] By synergizing the HOG algorithm and the Canny edge detector, a multi-level contour representation system for visible light and infrared images was constructed. The HOG algorithm extracts structural features from the global gradient distribution, while the Canny algorithm accurately locates local details and effectively suppresses edge noise caused by smoke particles. This complementary fusion of the two not only preserves the macroscopic morphological features of the human silhouette but also enhances the microstructural representation of key limb nodes, significantly improving the edge detection integrity of human targets in smoke-obscured environments.

[0031] In order to effectively fuse the heterogeneous modal features of visible light images and infrared thermal imaging images, an adaptive bilinear pooling network based on transfer learning is used to map the heterogeneous modal features of visible light images and infrared thermal imaging images to a common space. The modal fusion module based on the bilinear pooling network is used to perform modal fusion of visible light images and infrared thermal imaging images. The calculation formula of the modal fusion module is: in It is the feature representation of the visible light image, that is, the contour gradient histogram is extracted through the HOG algorithm of S13. ( express The transpose of represents an m-dimensional real vector space), It is the feature representation of infrared thermal imaging images, that is, the contour gradient histogram is extracted through the HOG algorithm of S13. , ( represents an n-dimensional real vector space) is the mapping matrix, and the obtained A feature representation that can fully reflect the essential characteristics of visible light and thermal imaging image data.

[0032] In order to reduce the amount of complex calculations, the above formula is further decomposed into the product of two low-rank matrices: In the formula represents an m×k-dimensional real vector space, represents an n×k-dimensional real vector space, where yes and Dimension, symbol Indicates the product operation of two matrices. Is a vector whose elements are all 1, three-dimensional vector U, V (that is, all and collectively referred to as) need to be acquired through learning.

[0033] Before image features are input into the bilinear pooling network, an adaptive learning mechanism is introduced to better extract features from visible light and infrared thermal images. The adaptive learning mechanism module consists of a 1x1 convolutional layer, a nonlinear activation layer (ReLu), and a Softmax layer. A more specific feature representation is obtained by multiplying the intra-modal features with the weights output by the adaptive learning mechanism. Therefore: In the formula represents the sum pooling operation, and It is the embedded representation of visible light images and infrared thermal imaging images obtained through an adaptive learning mechanism.

[0034] The bilinear pooling network is explicitly modeled through tensor product operations, allowing cross-channel feature association between visible light images and infrared thermal imaging images, mapping heterogeneous modal features into a unified semantic space, and forming a nonlinear coupling expression of the texture features (visible light) and thermal radiation features (infrared) of human targets, significantly enhancing the feature robustness in complex scenarios.

[0035] S2. Build a multimodal anti-interference human recognition network model based on convolutional neural network; S21. Use a wavelet graph convolutional neural network with an attention mechanism to perform multi-scale learning on the input visible light images and infrared thermal imaging images, and use the time-frequency localization characteristics of the wavelet graph convolutional neural network to suppress the broadband interference of environmental noise.

[0036] S211. Wavelet graph convolutional networks extract more comprehensive and discriminative feature representations. Using wavelet graph convolutional networks to extract key distinguishing features of images, the neural network model extracts useful information from input data, which can help the model better understand and process data. The wavelet graph convolutional network formula is: in It is a matrix composed of feature vectors as columns. For the signal of graph structure data of graph convolutional neural network , its graph Fourier transform is , and the inverse Fourier transform of the graph structure data is , y is the convolution kernel on the graph, is the convolution operator on the graph, represents the Hadamard product; S212. Construct a propagation method between adjacent layers of a wavelet-based graph convolutional network, which can be expressed as follows: in, represents the Lth layer of the wavelet graph convolutional network, Represents the L+1th layer of the wavelet graph convolutional network, the learned parameter matrix ,in express dimensional real vector space, is the dimension of the input vector, and q is the dimension of the output vector. Represented as a standard matrix In the form of ; S213. Integrate the attention mechanism module into the wavelet graph convolutional network to enhance the distinguishing features of the image. For the multimodal anti-interference human target recognition network model, different frequency domain components have different contributions, so the attention mechanism is applied to the spectral domain. The formula is: in is the wavelet basis, s is the scale parameter, and multi-scale feature extraction is achieved by adjusting the scale factor s to adaptively capture the spatial neighborhood information of human targets. Is a diagonal matrix, using the softmax function Standardize.

[0037] Wavelet-based graph convolutional networks (GCNNs) have good sparsity properties, reduce computational complexity, and adjust neighbor nodes of different scales. Furthermore, through the embedding of attention mechanisms, GCNNs based on wavelet transforms can better extract discriminative feature representations from a global perspective.

[0038] S22. For the human body recognition network model, visible light images and infrared thermal images are equally important. Therefore, a multi-perspective fusion module suitable for human target recognition is constructed to extract the consistency and complementary information between visible light images and infrared thermal images to complete the modal fusion of images.

[0039] S221. Using a single-sized convolution kernel can only extract features from a single source image, failing to capture rich source image information. However, using convolution kernels with different dilation rates can extract information from different receptive fields in visible light and infrared thermal images. Therefore, adding dense connections to inter-layer convolution operations enhances information flow between layers and improves efficiency. S222. Feature extraction for different dilation rates consists of two parts. The first part uses a standard convolution with a 1×1 kernel, which does not resize the input image and maintains the original image's receptive field. The second part uses three dilated convolutions with 3×3 kernels and dilation rates of 1, 3, and 5, respectively. These different dilation rates enable the acquisition of receptive fields of varying scales, extracting multi-scale information from the input source image. The features extracted by the four different convolution kernels in the two parts are then concatenated, and the final result is output using a 1×1 convolution kernel.

[0040] S223, the densely connected structure includes three 3×3 convolutions and a Leaky Relu activation function. As the depth of a convolutional neural network increases, weight updates may become unavailable during network training. Therefore, dense connections are used to alleviate this problem. By adding the input of the previous layer to the input of the next layer, feature reuse is achieved between layers, preventing information loss in the convolutional neural network. The concat operation is used in the final layer to stack the information from the previous layers, completing the fusion between the limited layers. Finally, the output is processed through a 1×1 convolution.

[0041] S23. After extracting the fusion features of the visible light image and the infrared thermal image, for each input sample, a cross entropy loss function of the multimodal anti-interference human target recognition network is constructed to integrate the heterogeneous features of the visible light image and the infrared thermal image: In the above formula, when d=1, it represents the visible light image network ( ), d=2 represents the visible infrared thermal image network ( ), Is the logit vector, which is converted into a probability distribution by the Softmax function , cross entropy loss compares the predicted probability with the true label , calculate the distribution difference between the two, and calculate the loss for each branch network separately for visible light images and infrared thermal images to retain modality-specific features.

[0042] S3, training the human target recognition network model to obtain the optimal human target recognition network model; S31. Obtain sample data for training a multimodal anti-interference recognition network model, where the sample data is a visible light image and an infrared thermal imaging image of a simulated fire scene collected synchronously in real time; S32. Use the LabelImg annotation tool to manually annotate the human subjects in the image and generate a YOLO-formatted label file corresponding to the image. Expand the dataset by random rotation, grayscale perturbation, and Gaussian noise injection to improve the model's robustness to noise, brightness differences, and smoke. This dataset, derived from the expanded sample set, will be used for training, testing, and validation in subsequent steps.

[0043] S33, dividing the obtained image data into a training set, a validation set, and a test set. The specific division can be performed by conventional division means, which will not be described in detail as it is an existing technology; S34. A training set is used to train the weight coefficients of the deep neural network human body recognition model. A validation set is used to verify the reliability of the weight coefficients and optimize the weight coefficients. The performance of the network model is determined by the weight coefficients. Obtaining weight coefficients that meet the requirements means obtaining a network model that meets the requirements. The test set is used to test the reliability of the final weight coefficients.

[0044] The weight parameters are updated through the training set, validation set and test set to complete the error correction learning process, and then the optimal human target recognition network model is obtained.

[0045] S4. Use the optimal human target recognition network model to perform human target recognition on images collected in real fire and smoke scenes.

[0046] In the actual task of human target recognition in mine fires, it is only necessary to input the synchronously collected visible light and infrared thermal imaging images into the multimodal anti-interference recognition network. The adaptive bilinear pooling network completes cross-modal feature alignment and fusion, and combines the wavelet graph convolution network with an attention mechanism to perform multi-scale interference suppression. It can output high-confidence human target positioning information, thereby realizing human target recognition in scenarios with dense smoke obscuration and thermal radiation interference.

[0047] High-precision recognition of human targets in extreme fire environments is the core of the deployment of intelligent rescue equipment. In the present invention, facing multiple interferences such as thick smoke, high temperature, dynamic thermal pseudo-targets in mine fires, it is necessary to solve problems such as visible light imaging texture attenuation, infrared thermal radiation feature drift, and strong heterogeneity of multimodal data. Through the cross-modal feature complementary mechanism and the anti-interference model driven by deep learning, real-time and accurate positioning of human targets in complex environments is achieved. The present invention solves the technical problems of strong heterogeneity of sensor data and difficulty in identifying human targets in complex fire environments through the cross-modal complementary fusion of visible light image features and infrared thermal radiation features, combined with the anti-interference model driven by deep learning, and significantly improves the real-time and reliability of human target recognition in mine fire rescue.

[0048] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal data fusion human target recognition method based on automatic control, characterized in that: The following steps are involved: S1. Obtain smoke images of the fire scene collected by a CMOS camera and an infrared thermal imaging camera, respectively. Use a contour extraction algorithm to preprocess the visible light image and infrared thermal imaging image of the fire scene, and perform modal alignment and fusion on the visible light image and infrared thermal imaging image. S2. Build a multimodal anti-interference human target recognition network model based on convolutional neural network; S3, using the image after modal alignment and fusion in step S1 as input, training the human target recognition network model in step S2 to obtain an optimal human target recognition network model; S4. Use the optimal human target recognition network model to perform human target recognition on images collected in real fire and smoke scenes.

2. The method for human target recognition based on multimodal data fusion based on automatic control according to claim 1, characterized in that: In step S1, a contour extraction algorithm is used to pre-process the visible light image and infrared thermal imaging image of the fire scene. The specific process includes the following steps: S11. Use a CMOS camera to dynamically capture visible light image streams at the fire scene, ensuring that the dynamic range of illumination intensity in the target area meets the requirements for analyzing human texture features and retaining edge details of the human body contour. S12. Use an infrared thermal imaging camera to collect thermal radiation distribution images of the target area to ensure that the radiation energy difference between the human target and the high-temperature background can be resolved; S13. The HOG algorithm and Canny edge detector are used to segment the contours of human targets in visible light images and infrared thermal images to improve the detection accuracy of human targets in dense smoke environments.

3. The method for human target recognition based on multimodal data fusion based on automatic control according to claim 2, characterized in that: In step S13, the HOG algorithm and the Canny edge detector are used to perform contour segmentation on the human target. The specific process includes the following steps: S131, after converting the visible light and thermal imaging images into grayscale images, using the oriented gradient histogram as a feature to describe the human body contour; S132, establishing a sliding window of fixed size in the grayscale image, traversing the entire grayscale image with a preset step size, calculating the directional gradient histogram within each sliding window, concatenating the gradient histograms of the sliding windows, and obtaining the HOG feature vector of the image interval of each sliding window; S133, using the Gaussian filter of the Canny algorithm to smooth the input visible light image and reduce the interference of high-frequency noise; S134, performing threshold processing on the filtered image, refining the edges in the visible light image, setting a high threshold and a low threshold, and segmenting the visible light image; S135, connect the edges in the visible light image based on the HOG feature vector, traverse the entire image, add the strong pixel points in the low threshold gradient image to the high threshold gradient image, and obtain the final edge gradient amplitude image .

4. The method for human target recognition based on multimodal data fusion based on automatic control according to claim 1, characterized in that: In step S1, the visible light image and the infrared thermal imaging image are modally aligned and fused. Construct a modality fusion module for visible light images and infrared thermal imaging images based on a bilinear pooling network: ,in is the feature representation of the visible light image, is the mapping matrix, is the characteristic representation of infrared thermal imaging image, and the obtained Feature representation that fully reflects the essential characteristics of visible light image and infrared thermal imaging image data; Before the image features are input into the bilinear pooling network, an adaptive learning mechanism is introduced to better extract features from visible light images and infrared thermal imaging images.

5. The method for human target recognition based on multimodal data fusion based on automatic control according to claim 1, characterized in that: In step S2, a multimodal anti-interference human body recognition network model based on a convolutional neural network is constructed, and the specific process includes the following steps: S21. Use a wavelet graph convolutional neural network with an attention mechanism to perform multi-scale learning on the input visible light image and infrared thermal imaging image, and use the time-frequency localization characteristics of the wavelet graph convolutional neural network to suppress the broadband interference of environmental noise; S22. Construct a multi-view fusion module suitable for human target recognition to extract the consistency and complementary information between visible light images and infrared thermal images; S23. After fusion feature extraction of visible light images and infrared thermal imaging images, a cross-entropy loss function of a multimodal anti-interference human target recognition network is constructed to obtain heterogeneous features of visible light images and infrared thermal imaging images.

6. The method for human target recognition based on multimodal data fusion and automatic control according to claim 5, characterized in that: In step S21, a wavelet graph convolutional neural network with an attention mechanism is used to perform multi-scale learning on the input image. The specific process includes the following steps: S211. Extract key distinguishing features of images using wavelet graph convolutional networks; S212. Construct a propagation method between adjacent layers of a wavelet graph convolutional network to adjust neighbor nodes of different scales. S213. Use the attention mechanism configured in the wavelet graph convolutional network to enhance the distinctive features of the image.

7. The method for human target recognition based on multimodal data fusion and automatic control according to claim 5, characterized in that: In step S22, a multi-view fusion module suitable for human target recognition is used to extract consistency and complementarity information between the visible light image and the infrared thermal image. The specific process includes the following steps: S221. Use convolution kernels with different dilation rates to extract information from different receptive fields in visible light images and infrared thermal images. Add dense connection operations to the convolution operations between layers to enhance the flow of information between layers. S222, the feature extraction module consists of a 1×1 standard convolution and three 3×3 dilated convolutions in parallel. It captures heterogeneous features through multi-scale receptive fields and then fuses them through concat, and finally uses a 1×1 standard convolution to reduce the dimension of the output; S223, the dense connection structure adopts a cascade structure of three 3×3 convolutions + LeakyReLU, which alleviates the gradient attenuation of the deep network through the cross-layer feature reuse mechanism, and realizes information fusion through inter-layer feature stacking and 1×1 convolution.

8. The method for human target recognition based on multimodal data fusion and automatic control according to claim 1, characterized in that: In step S3, the human target recognition network model is trained to obtain the optimal human target recognition network model, and the specific process includes the following steps: S31. Obtain sample data for training a human target recognition network model, where the sample data is a visible light image and an infrared thermal imaging image of a simulated fire scene collected synchronously in real time; S32. Use the LabelImg annotation tool to manually annotate human targets in the image. Expand the dataset by random rotation, grayscale value perturbation, and Gaussian noise injection to improve the robustness of the multimodal anti-interference recognition network model to noise, brightness differences, and smoke. S33, dividing the expanded data set into a training set, a validation set, and a test set; S34. A training set is used to train the weight coefficients of the human target recognition network model, a validation set is used to verify the reliability of the weight coefficients and optimize the weight coefficients, and a test set is used to test the reliability of the weight coefficients finally obtained.

Images