Personnel intrusion detection method and system based on feature fusion

Abstract

The invention relates to a personnel intrusion detection method and system based on feature fusion, and relates to the technical field of visual detection and image detection, and the method comprises the steps: inputting a visible light image and an infrared image into a feature selection network, and carrying out the feature extraction and preliminary fusion; the extracted features are sent to a feature enhancement network for deep enhancement, and enhanced features are obtained; and inputting the enhanced features into the target detection network to complete personnel detection and outputting a result. The problems that a traditional intrusion detection system is unstable in performance and high in false alarm rate under complex illumination and severe weather are solved, and the robustness and accuracy of detection are improved.

Description

Technical Field

[0001] This invention relates to the fields of visual inspection and image inspection, and in particular to a method and system for detecting intrusion by persons based on feature fusion. Background Technology

[0002] With increasing awareness of security and technological advancements, intrusion detection systems are gradually evolving from traditional passive infrared detection and microwave radar technologies to more advanced multi-sensor fusion methods. In particular, detection methods combining infrared and visible light have garnered widespread attention due to their ability to provide complementary information under varying lighting conditions. Infrared light can capture the heat emitted by objects and remains effective in low-light or no-light environments; while visible light images provide rich color and texture details, aiding in more accurate target identification during the day or under good lighting conditions. Combining the advantages of both not only improves the accuracy of intrusion detection but also enhances the system's adaptability and reliability.

[0003] Traditional intrusion detection systems perform inconsistently under different lighting conditions (such as nighttime, low light, and direct sunlight), affecting detection accuracy. Environmental factors such as smoke, dense fog, and haze reduce the effectiveness of visible light cameras, while infrared light, although able to penetrate these obstacles, has lower resolution. Using a single sensor (such as those using only infrared or visible light) can lead to a high false alarm rate, for example, due to false triggers caused by animal activity or weather changes. How to effectively fuse data from different sources of visible and infrared light to more accurately understand and interpret the scene is also a pressing problem in the field of visual inspection. Summary of the Invention

[0004] The purpose of this invention is to provide a feature fusion-based method and system for personnel intrusion detection. By constructing a dual-modal image dynamic selection fusion framework, complementary enhancement of visible light and infrared images is achieved. This aims to solve the problems of unstable performance and high false alarm rate in traditional intrusion detection systems under complex lighting and adverse weather conditions, thereby improving the robustness and accuracy of detection.

[0005] To achieve the above objectives, the present invention provides a personnel intrusion detection method based on feature fusion, comprising the following steps: Step S1: Input the visible light image and infrared image into the feature selection network for feature extraction and preliminary fusion; Step S2: The extracted features are fed into a feature enhancement network for deep enhancement to obtain enhanced features; Step S3: Input the enhanced features into the target detection network to complete the personnel detection and output the results.

[0006] Preferably, in step S1, feature extraction and preliminary fusion are performed by a dynamic fusion layer with embedded parallel branches.

[0007] Preferably, the dynamic fusion layer uses a Transformer multi-head attention structure to calculate the interaction correlation between visible light and infrared features, and updates the feature representation through residual connections.

[0008] Preferably, the feature selection network uses the VGG16 architecture as the backbone network, and sets up two parallel branches with the same structure for visible light images and infrared images respectively. Three dynamic fusion layers are embedded in the middle of the two branches to generate fused feature maps P1, P2 and P3 in sequence.

[0009] Preferably, the dynamic fusion layer performs the following operations: The visible light feature map and the infrared feature map are flattened and then concatenated in parallel to obtain the concatenated feature vector. The concatenated feature vectors are input into a multilayer perceptron to generate Q, K, and V matrices; The interaction feature vector is calculated through a multi-head attention module, and after batch normalization and multilayer perceptron processing, it is connected with the concatenated feature vector to form a residual connection. The updated visible light and infrared features are obtained through the separation operation.

[0010] Preferably, in step S2, enhanced features are obtained through deep enhancement via multi-scale feature aggregation and triple attention mechanism.

[0011] Preferably, the feature enhancement network includes: The fused feature maps P1, P2 and P3 are input into a path aggregation network with a feature pyramid network structure to generate multi-scale feature maps. Multi-scale feature maps are reduced in dimensionality, upsampled, and concatenated in parallel to obtain multi-scale aggregated features. The aggregated features are adaptively updated sequentially through scale attention, spatial attention, and channel attention layers, and then the enhanced feature map is obtained through slicing and decoupling.

[0012] Preferably, scale attention and channel attention use an improved SENet structure to achieve global average pooling and channel weight learning, while spatial attention uses a deformable convolutional structure to adaptively adjust the feature sampling position.

[0013] The technical solution of the present invention also provides a personnel intrusion detection system based on feature fusion, comprising: The visible light image acquisition module and the infrared image acquisition module are used to simultaneously acquire visible light images and infrared images of the same scene; The computational processing unit carries the computational load of the feature selection network, feature enhancement network, and object detection network, and completes feature processing through a dynamic fusion layer and a triple attention mechanism. The detection output module is used to output the personnel detection results.

[0014] Preferably, the computing processing unit deploys a VGG16 backbone network to construct two parallel branches, embeds three dynamic fusion layers based on the Transformer structure between the branches, and uses a path aggregation network with an FPN+PAN structure combined with a triple attention mechanism for feature enhancement.

[0015] Compared with the prior art, the present invention has the following beneficial technical effects: This invention fully utilizes the rich texture details of visible light images and the temperature distribution characteristics of infrared images to achieve dual-modal complementary enhancement through dynamic feature fusion. This allows the fused image to retain both clear contours and thermal radiation information, significantly improving visual perception and scene analysis capabilities under harsh conditions such as drastic changes in lighting, low light, and smog. It also solves the technical problem of unstable performance of a single sensor in complex environments.

[0016] By employing multi-level dynamic feature selection and a triple attention mechanism, this invention effectively increases the useful information content in images, enhancing key features and suppressing redundant information. The multi-scale feature aggregation network integrates semantic information from different levels, combining adaptive correction of channel, spatial, and scale dimensions to make feature representations more discriminative and robust, thereby improving the ability to distinguish between human targets and interference sources (such as animal activity).

[0017] This invention supports the fusion processing of image sequence features from different time points, enabling the detection of dynamic changes in scenes or targets. It fully utilizes spectral information and temporal data to effectively improve spatial resolution. The end-to-end optimization framework organically combines dynamic feature selection, multi-scale enhancement, and a high-efficiency detection network, significantly improving the accuracy and real-time performance of personnel intrusion detection and greatly reducing false alarm rates caused by weather changes and environmental factors. This provides a highly adaptable and reliable technical solution for security monitoring. Attached Figure Description

[0018] Figure 1 This is a flowchart of a personnel intrusion detection method and system based on feature fusion according to the present invention. Figure 2 This invention relates to a feature selection network diagram in a personnel intrusion detection method and system based on feature fusion. Figure 3 This invention relates to a feature enhancement network diagram in a personnel intrusion detection method and system based on feature fusion. Figure 4This is a diagram of a dynamic fusion algorithm in a feature fusion-based personnel intrusion detection method and system according to the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1: This invention discloses a human intrusion detection method based on feature fusion. By constructing a target detection framework with dynamic selection of image features, it effectively addresses the technical problems of inconsistent performance of traditional intrusion detection systems under different lighting conditions and adverse weather conditions, as well as the high false alarm rate of single sensors.

[0021] This method uses visible light and infrared images as dual-source inputs, fully leveraging the rich texture details of visible light images and the advantages of infrared images in low-light environments and penetrating smoke and haze to achieve complementary enhancement. The specific steps are as follows: Step S1: Input the two image sources (infrared image and visible image) into the feature selection network for feature extraction and preliminary fusion; Step S2: The extracted image features are fed into a feature enhancement network for deep enhancement to obtain image augmented features; Step S3 involves inputting the enhanced features into the target detection network to complete personnel detection and output the results, forming a complete technical link from dual-modal data input to accurate detection output.

[0022] Specifically, step S1 includes: In step S11, in the specific construction of the feature selection network, the backbone network adopts the VGG16 architecture, and two parallel branches with the same structure are set for visible light images and infrared images respectively, so as to ensure that the two modal features can be effectively aligned and interacted in a unified feature space.

[0023] This invention embeds three dynamic fusion layers (DyFu) in the middle of two branches, updating the feature representation by dynamically fusing the feature maps of the two images. The specific fusion formula is as follows:

[0024] in and These represent the visible light and infrared image features before feature updates, respectively. and These are the updated visible light and infrared image features, respectively. This indicates a dynamic fusion layer.

[0025] Each dynamic fusion layer adds the fusion result to the original feature map of the corresponding branch, sequentially generating fused feature maps P1, P2, and P3, which are then fed into the subsequent feature enhancement network. The internal computation of the dynamic fusion layer first combines the feature maps of visible light and infrared images... and After flattening, the concatenations are connected in parallel to obtain the concatenated feature vector. Specifically

[0026] in and These represent parallel operation and flattening operation, respectively.

[0027] Subsequently, the concatenated feature vector is passed sequentially through three multilayer perceptrons to generate corresponding Q, K, and V matrices, which are then input into a multi-head attention module. The Transformer structure is then used to compute the interaction feature vector after feature fusion. Specifically

[0028]

[0029] Where MLP stands for Multilayer Perceptron, MHA stands for Multi-head Attention Layer, and α is the attention weight matrix.

[0030] This method of concatenating images before calculating attention weights allows for the simultaneous performance of four correlation operations related to image features, which are categorized by region as follows: autocorrelation of visible light image features, cross-correlation between visible and infrared image features, cross-correlation between infrared and visible light image features, and autocorrelation of infrared image features. These correlation operations enable better quantification of the feature matrices for both image modes. and The deep correlation between them.

[0031] Interactive feature vector After normalization and processing by a multi-layer perceptron, residual connections are formed with the previous variables, and finally, the updated features are obtained through a separation operation. and Specifically

[0032] in The delimiter represents the separation operation, and BN represents the batch normalization layer. It is worth noting that the multi-head attention module in this algorithm model is repeated eight times to fully explore the correlation information in different feature subspaces and enhance the robustness of the fusion process.

[0033] Furthermore, step S2 includes: Step S21: Input the three fused feature maps P1, P2, and P3 output by the feature selection network into the path aggregation network with a feature pyramid network structure to generate multi-scale feature maps. , and Specifically

[0034] in This represents a path aggregation network with a feature pyramid structure, which can effectively integrate semantic information at different levels and improve the model's ability to perceive multi-scale targets.

[0035] Step S22, will , and All features are fed into a multilayer perceptron layer for dimensionality reduction, so that the feature dimension is uniformly 128.

[0036] To standardize the size of feature maps, and After bilinear interpolation upsampling and Parallel concatenation yields a multi-scale aggregated feature FU, with a feature dimension of (3×80×80×128).

[0037] Where up represents the upsampling operation, 3 represents three feature scales used to detect targets of different sizes, 80×80 represents the feature map size, and 128 represents the number of feature channels.

[0038] Step S23, sequentially The features of 3, 80×80, and 128 dimensions are adaptively updated sequentially through scale attention, spatial attention, and channel attention layers, respectively.

[0039] Here, Cha, Spa, and Sca represent the channel attention, spatial attention, and scale attention layers, respectively. Through the cascading effect of this triple attention mechanism, the model can dynamically highlight key features and suppress redundant information. Finally, the model is decoupled by slicing along the scale dimension and downsampled to obtain the enhanced feature map. , and Specifically

[0040] In the specific implementation of the attention mechanism, this invention employs an improved SENet structure to implement scale attention and channel attention processes. This structure first compresses a feature matrix F of size (N, C) into a channel descriptor, generating channel statistics using global average pooling, where N can represent multiple feature dimensions. Subsequently, the descriptor is input into a multilayer perceptron layer with a bottleneck structure to learn the attention weights for each channel. Finally, a weighted feature matrix is ​​generated through the dot product operation between the original feature matrix F and the channel attention weights. Specifically

[0041] Where FC stands for fully connected layer. Representing the inner product of matrices, this design can effectively model the dependencies between channels and adaptively adjust the importance weights of each channel.

[0042] The spatial attention process employs a deformable convolutional network (DCN) structure. By learning offsets, the convolutional kernels adaptively sample on the feature map, thereby more accurately locating the target region and enhancing the feature representation capability in the spatial dimension. These three attention mechanisms refine the features from different dimensions, significantly improving the discriminativeness and robustness of feature representation.

[0043] Furthermore, in step S3, the enhanced features are input into the detection network, and the detection results are output, including the following steps: Step S31, enhance the feature map The input is fed into the detection network and fused using equal weights, with each weight allocated as 0.3. Specifically, this is expressed as follows:

[0044] This equal weighting strategy can effectively integrate multi-scale feature information while maintaining computational simplicity, avoiding the limitations of single-scale features.

[0045] In step S32, the obtained fused features are input into the YOLOv5 detection network to output the person detection results. With its efficient single-stage detection architecture and excellent real-time performance, YOLOv5 can quickly and accurately complete the determination of personnel intrusion, meeting the dual requirements of detection speed and accuracy in practical application scenarios.

[0046] like Figure 1As shown, the complete process of the personnel intrusion detection method based on feature fusion is demonstrated: using "visible light image" and "infrared image" as dual input sources, the two types of images are first input into the feature selection network to complete feature extraction and preliminary fusion; then the processed features are sent into the feature enhancement network to strengthen feature expression; then the enhanced features are input into the target detection network for detection operation; finally, the "personnel detection" result is output, which clearly reflects the whole link process from dual image input to detection result output.

[0047] like Figure 2 The diagram illustrates the specific architecture of the feature selection network: it comprises two parallel branches (corresponding to feature extraction branches for visible light and infrared images, respectively), each branch sequentially connecting a convolutional layer (labeled "C") to a feature processing module; three dynamic fusion layers are embedded between the two branches. ), each After the layer dynamically fuses the two feature maps, it adds them to the original features of the corresponding branches to generate fused feature maps P1, P2 and P3 in sequence. These feature maps will be fed into the subsequent feature enhancement network.

[0048] like Figure 3 The diagram illustrates the computational logic of the feature enhancement network: First, P1, P2, and P3 generated by the feature selection network are input into the path aggregation network with an FPN+PAN structure to aggregate multi-scale features; then, the aggregated features are fed into the DySe module (which performs scale, spatial, and channel attention processing) to enhance feature representation; finally, the enhanced feature map is output. , and This provides enhanced feature inputs for the object detection network.

[0049] like Figure 4 As shown, it demonstrates Internal computational flow: Input is visible light features With infrared features First, flatten and concatenate the two to obtain the spliced ​​feature vector. Subsequently, position encoding and batch normalization are performed sequentially, followed by input into a multi-head attention module (repeated 8 times). After batch normalization and multilayer perceptron processing, features are integrated through residual connections. Finally, the updated visible light features are obtained. With infrared features This enables the dynamic fusion and updating of the two types of features.

[0050] This invention organically combines dynamic feature selection, multi-scale feature enhancement, and a high-efficiency detection network to construct an end-to-end optimization framework. This framework not only enhances the model's understanding and analysis capabilities of complex scenes but also effectively detects changes in scenes or targets by fusing image sequence features from different times. It significantly improves the spatial resolution and spectral information utilization of images, thus maintaining stable and reliable personnel intrusion detection performance under harsh conditions such as drastic changes in lighting, low light, no light, and smog. It also effectively reduces the false alarm rate caused by factors such as animal activity and weather changes, providing a highly adaptable and robust technical solution for the security monitoring field.

[0051] Example 2: This embodiment discloses a personnel intrusion detection system based on feature fusion. The system is used to implement the method described in Embodiment 1 and includes a visible light image acquisition module, an infrared image acquisition module, a computing processing unit, and a detection output module.

[0052] Visible light image acquisition modules and infrared image acquisition modules are deployed in the monitoring area to simultaneously acquire visible light and infrared images of the same scene, ensuring that the dual-modal data matches in the spatiotemporal dimensions. The computational processing unit carries the computational load of the feature selection network, feature enhancement network, and target detection network. The VGG16 backbone network deployed within it provides feature extraction capabilities for two parallel branches. Three dynamic fusion layers are embedded between the branches, and the correlation calculation is repeatedly performed eight times through a multi-head attention mechanism to complete the dynamic selection and interactive updating of visible light and infrared features.

[0053] The feature enhancement network receives the fused feature maps P1, P2, and P3 output by the feature selection network, generates multi-scale feature maps via the path aggregation network, and achieves scale unification through bilinear interpolation upsampling to form multi-scale aggregated features with dimensions of (3×80×80×128). Then, it passes through scale attention, spatial attention, and channel attention layers in sequence. The scale and channel attention layers are based on the improved SENet structure to achieve global average pooling and channel weight learning, while the spatial attention layer adopts a deformable convolutional structure to adaptively adjust the feature sampling position. Finally, the enhanced feature map is obtained through slicing decoupling and downsampling.

[0054] The detection output module fuses the enhanced multi-scale features with equal weights and then sends them to the YOLOv5 detection network. This network is configured to output only the detection results of personnel categories. When an intrusion target is detected, an alarm signal is triggered. The alarm signal can be linked to external audible and visual warning devices or pushed to a remote monitoring platform.

[0055] The system supports GPU parallel acceleration at the hardware level to improve the computational efficiency of the Q, K, and V matrices in the dynamic fusion layer, and can achieve real-time processing through model optimization using a deep learning framework. In low-light, strong direct sunlight, or smoggy environments at night, the system adaptively adjusts the fusion weights of visible light and infrared features through a dual-modal feature complementarity mechanism. The infrared mode provides dominant thermal radiation information under low-light conditions, while the visible light mode supplements texture details under good lighting conditions. In severe weather, the dual-modal approach works together to penetrate obstacles, thereby reducing the risk of false alarms caused by changes in lighting, animal activity, or meteorological factors.

[0056] The system supports fine-tuning of the dynamic fusion layer and attention module using newly added bimodal data when the deployment environment changes. It can adapt to new scenarios without reconstructing the entire network, significantly shortening the system adaptation cycle and providing stable and reliable technical support for intelligent security monitoring.

[0057] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting intrusion by personnel based on feature fusion, characterized in that, Includes the following steps: Step S1: Input the visible light image and infrared image into the feature selection network for feature extraction and preliminary fusion; Step S2: The extracted features are fed into a feature enhancement network for deep enhancement to obtain enhanced features; Step S3: Input the enhanced features into the target detection network to complete the personnel detection and output the results.

2. The personnel intrusion detection method based on feature fusion according to claim 1, characterized in that, In step S1, feature extraction and preliminary fusion are performed through a dynamic fusion layer that embeds parallel branches.

3. The personnel intrusion detection method based on feature fusion according to claim 2, characterized in that, The dynamic fusion layer uses a Transformer multi-head attention structure to calculate the interaction correlation between visible light and infrared features, and updates the feature representation through residual connections.

4. The personnel intrusion detection method based on feature fusion according to claim 3, characterized in that, The feature selection network uses the VGG16 architecture as its backbone network. Two parallel branches with the same structure are set up for visible light images and infrared images respectively. Three dynamic fusion layers are embedded in the middle of the two branches to generate fused feature maps P1, P2 and P3 in sequence.

5. The personnel intrusion detection method based on feature fusion according to claim 4, characterized in that, The dynamic fusion layer performs the following operations: The visible light feature map and the infrared feature map are flattened and then concatenated in parallel to obtain the concatenated feature vector. The concatenated feature vectors are input into a multilayer perceptron to generate Q, K, and V matrices; The interaction feature vector is calculated through a multi-head attention module, and after batch normalization and multilayer perceptron processing, it is connected with the concatenated feature vector to form a residual connection. The updated visible light and infrared features are obtained through the separation operation.

6. A method for detecting intrusion based on feature fusion according to any one of claims 1-5, characterized in that, In step S2, enhanced features are obtained through deep enhancement via multi-scale feature aggregation and triple attention mechanism.

7. The personnel intrusion detection method based on feature fusion according to claim 6, characterized in that, Feature enhancement networks include: The fused feature maps P1, P2 and P3 are input into a path aggregation network with a feature pyramid network structure to generate multi-scale feature maps. Multi-scale feature maps are reduced in dimensionality, upsampled, and concatenated in parallel to obtain multi-scale aggregated features. The aggregated features are adaptively updated sequentially through scale attention, spatial attention, and channel attention layers, and then the enhanced feature map is obtained through slicing and decoupling.

8. The personnel intrusion detection method based on feature fusion according to claim 7, characterized in that, Scale attention and channel attention employ an improved SENet structure to achieve global average pooling and channel weight learning, while spatial attention uses a deformable convolutional structure to adaptively adjust the feature sampling position.

9. A system for implementing the personnel intrusion detection method based on feature fusion as described in any one of claims 1-8, characterized in that, include: The visible light image acquisition module and the infrared image acquisition module are used to simultaneously acquire visible light images and infrared images of the same scene; The computational processing unit carries the computational load of the feature selection network, feature enhancement network, and object detection network, and completes feature processing through a dynamic fusion layer and a triple attention mechanism. The detection output module is used to output the personnel detection results.

10. A personnel intrusion detection system based on feature fusion according to claim 9, characterized in that, The computing unit deploys a VGG16 backbone network to construct two parallel branches, embeds three dynamic fusion layers based on the Transformer structure between the branches, and uses a path aggregation network with an FPN+PAN structure combined with a triple attention mechanism for feature enhancement.

Images