A security inspection and anti-intrusion detection method based on semantic segmentation

Abstract

The present invention discloses a security inspection and anti-intrusion detection method based on semantic segmentation, including obtaining a data set for semantic segmentation; constructing a BiConA module; constructing a human body segmentation network; training the human body segmentation network with the data set to obtain a human body segmentation model; splitting the surveillance video containing the security inspection area into a surveillance image set and inputting the surveillance image set into the human body segmentation model to obtain a segmented image set; then calculating the confidence based on multiple influencing factors, and judging whether there is an intrusion behavior in the security inspection area according to the confidence threshold. The present invention designs the BiConA module and integrates it into the encoder module of the Segformer network, which can enhance the context understanding ability of the network, better process the irregular detail information of the human body area in the image, and improve the human body area segmentation accuracy of the entire network. Combining multiple influencing factors to calculate the confidence can effectively avoid the intrusion detection judgment error caused by the perspective effect and visual dislocation of the image, and can also improve the accuracy and robustness.

Description

Technical Field

[0001] The present invention relates to an image data processing method, and in particular to a security inspection and anti-intrusion detection method based on semantic segmentation. Background Art

[0002] In places with dense traffic and important security needs, such as airports and large distribution centers, in order to ensure public safety and prevent potential threats, some areas that are prohibited from entering and exiting are often set up, and human intrusion detection is performed in these areas. These areas are generally called security check areas. The current detection schemes mainly include: (1) Manual security: By arranging personnel to patrol the security check area at regular intervals and fixed points, suspected intruders are warned and stopped. This method requires extremely high manpower costs. If an accident occurs, the security personnel may not be able to report the security check intrusion incident in time. (2) Sensor equipment: Radar or ordinary infrared sensors are arranged at the boundary of the security check area. These methods can promptly respond to whether the boundary of the security check area has been invaded, but the disadvantages are also obvious. First, the equipment cost is high, and second, the misjudgment rate of non-personnel appearing in the detection area of ​​the sensor equipment is high. (3) Image method: Based on ordinary cameras, images of the security check area are obtained and human targets are detected. This method is low-cost and can detect the presence of people in the security check area, but it does not clearly determine the intrusion behavior, and still requires secondary confirmation by background personnel. (4) Target detection method based on deep learning: This method uses the human body in the image as the target for target detection. Although it has high accuracy, the effect is simple. It only anchors a rectangular coordinate frame in the area where the human body appears in the image, and it is difficult to judge the intrusion behavior in the security inspection area.

[0003] SegFormer (Simple and Efficient Design for Semantic Segmentation with Transformers) is a semantic segmentation network that successfully introduces a hierarchical structure into Transformer and extracts information of different scales. In the semantic segmentation task, its accuracy and speed are not inferior to OCRNet.

[0004] Regarding semantic segmentation networks, most of them are encoder-decoder architectures. The encoder part in the present invention comes from the SegFormer network. Among them, SegFormer has two main features: (1) It uses a hierarchical Transformer encoder, the input is an image sample, and the output is the corresponding multi-scale features of the image sample. This algorithm does not require position encoding, avoiding the problem of performance degradation caused by interpolating the position code when the test resolution is different from the training resolution. The hierarchical Transformer encoder is also called the encoder module of the SegFormer network. (2) A lightweight MLP decoder is used to fuse features at different levels. When encoding, the present invention uses the encoder module of the SegFormer network to provide two images of different scales. The decoder is a conventional decoder, which is used to input feature maps of different scales, perform upsampling, prediction and other operations, predict each pixel in the image, and obtain a semantic segmentation result. Summary of the invention

[0005] The purpose of the present invention is to provide a security inspection and anti-intrusion detection method based on semantic segmentation that solves the above-mentioned problems, improves segmentation accuracy, has a higher accuracy rate for human intrusion behavior, and efficiently avoids misjudgment.

[0006] In order to achieve the above-mentioned object, the technical solution adopted by the present invention is as follows: a security inspection and anti-intrusion detection method based on semantic segmentation comprises the following steps;

[0007] S1, obtaining a data set for semantic segmentation, wherein samples in the data set are images labeled with target categories, and the target category is human body;

[0008] S2, construct BiConA module;

[0009] The BiConA module is used to input the first feature map C1 and the second feature map C2, concatenate C1 and C2 through 3×3 convolution to obtain the third feature map C3, perform average pooling of convolution kernel sizes of 1, 3, and 5 on C3, and then perform 1×1 convolution to obtain the fourth feature map C4, the fifth feature map C5, and the sixth feature map C6, and then obtain the i-th difference feature map CF according to the following formula: i And the output feature map of the BiConA module ;

[0010] ,

[0011] ,

[0012] Where, 1≤i≤3, It is a 1×1 convolution operation;

[0013] S3, construct a human segmentation network, including an encoder module, a BiConA module and a decoder;

[0014] The encoder module is an encoder module in the SegFormer network, including four layers of Transformer encoders TB1-TB4 connected in sequence, which are used to input samples in the data set and obtain feature maps of different scales through TB1-TB4;

[0015] The input end of the BiConA module is connected to TB2 and TB4, and is used to use the output of TB2 as the first characteristic map C1 of the BiConA module, and the output of TB4 as the second characteristic map C2 of the BiConA module to obtain F BiConA ;

[0016] The decoder is used to input F BiConA and C2, after decoding, outputs the semantic segmentation result, and uses the semantic segmentation result as the human body segmentation image;

[0017] S4, use the data set to train the human segmentation network to obtain a human segmentation model;

[0018] S5, obtain the surveillance video containing the security inspection area, split it into a single-frame surveillance image set I={I t}, 1≤t≤T, T is the total number of images, I t is the tth image in I, from I t Mark the security check area X t , set I={I t} Input the human body segmentation model and get I t The corresponding human segmentation image A t , forming a segmented image set A={A t};

[0019] S6, confidence calculation based on multiple influencing factors, including S61~S63;

[0020] S61, from A t Get the irregular area of ​​the target and calculate the area S of the irregular area t , perimeter L t , center point O t , calculate O t-3 , O t-2 , O t-1 With X t The distance d from the center point t-3 d t-2 d t-1 , where O t-3 , O t-2 , O t-1 A t-3 , At-2 , A t-1 The center point of the irregular area;

[0021] S62, calculate S t With security area X t The overlap ratio P1,L t With X t The coverage rate P2,X t Position P3, human movement trend P4, and P1~P4 as influencing factors;

[0022] Among them, if O t Located in X t If yes, P3=1, otherwise P3=0;

[0023] If d t-3 >d t-2 >d t-1 , then P4=1, d t-3 <d t-2 <d t-1 , then P4=0, otherwise P4=0.5,

[0024] S63, calculate A according to the following formula t The confidence level R t ;

[0025] R t =ω1P1+ω2P2+ω3P3+ω4P 4，

[0026] Where ω1, ω2, ω3, and ω4 are the empirical coefficients of P1, P2, P3, and P4 respectively;

[0027] S7, preset confidence threshold R l , compared with R t and R l , if R t >R l , it is judged that there is an intrusion in the security inspection area.

[0028] Preferably, the dataset is a Cityscapes dataset.

[0029] Preferably, the decoder is a decoder of a DeepLabv3+ network.

[0030] Preferably, the BiConA module includes a 3×3 convolution layer, a first concatenation layer, a first average pooling layer, a second average pooling layer, a third average pooling layer, an i-th difference feature layer, and a feature fusion layer, wherein the convolution kernel sizes of the first average pooling layer, the second average pooling layer, and the third average pooling layer are 1, 3, and 5, respectively, and 1≤i≤3;

[0031] The first feature map C1 and the second feature map C2 are respectively subjected to a 3×3 convolution layer and a first concatenation layer to obtain a third feature map C3;

[0032] The third feature map C3 is divided into three paths. One path passes through the first average pooling layer and the 1×1 convolution layer to obtain the fourth feature map C4, one path passes through the second average pooling layer and the 1×1 convolution layer to obtain the fifth feature map C4, and one path passes through the third average pooling layer and the 1×1 convolution layer to obtain the sixth feature map C6;

[0033] The i-th difference feature layer includes the i-th difference layer and the 1×1 convolution layer, where the i-th difference layer is used to calculate C i With C i+3 The 1×1 convolution layer is used to perform a 1×1 convolution operation on the difference to obtain the i-th difference feature map CF i ;

[0034] The feature fusion layer is used according to the formula Perform feature fusion to obtain the output feature map F of the BiConA module.

[0035] Preferably, the feature fusion layer includes a first feature fusion layer, a second feature fusion layer and a third feature fusion layer, wherein the first feature fusion layer and the second feature fusion layer execute the formula , The third feature fusion layer is used to combine the outputs of the first feature fusion layer and the second feature fusion layer with Add.

[0036] As a preferred embodiment: in S62, , .

[0037] As a preference: 0.6≤R l ≤0.8.

[0038] Compared with the prior art, the advantages of the present invention are:

[0039] (1) In security monitoring images, there will be one or more human targets, and due to the dynamic behavior of the human body, its coverage area changes irregularly. In order to accurately obtain the area where the human body is distributed in the image, in addition to obtaining high-level semantic information, it is also necessary to obtain richer contextual information in the image. Therefore, the present invention proposes a BiConA module based on double-layer contextual attention, which convolves two feature maps C1 and C2 of different scales, performs average pooling with convolution kernel sizes of 1, 3, and 5, and then performs 1×1 convolution to obtain feature maps C4, C5, and C6. The output of the BiConA module is obtained by a series of multiplication and concatenation calculations of three contrast features. By performing difference calculation and convolution processing with the original C1, C2, and C3 feature maps, CF1, CF2, and CF3 are obtained, and then a series of multiplication, addition, and other operations are performed, so that the network can enhance the receptive field while strengthening the association of contextual information. Thus, the output features obtain richer contextual information.

[0040] (2) The BiConA module is integrated into the encoder module of the Segformer network. The encoder module of the Segformer network is first used to obtain feature maps of two different scales of the image as two inputs of the BiConA module. The two feature maps of different scales are then processed using the BiConA module to enhance the network's contextual understanding ability. This allows the network to better handle irregular detail information of the human body region in the image, which helps to improve the human body region segmentation accuracy of the entire network.

[0041] (3) An intrusion detection algorithm based on multiple confidence levels is designed. The output of the human segmentation network is used as a benchmark to determine whether it is an intrusion event based on the designated security inspection area and four influencing factors P1~P4. The human movement trend P4 is based on static data and incorporates the changes in the center point of the human body in multiple frames into the calculation. Therefore, the present invention can effectively avoid the intrusion detection judgment errors caused by the perspective effect and visual dislocation of the image, and can also improve the accuracy of intrusion detection and the robustness of intrusion detection. When the segmented image set A={A t}, t=1, 2, 3, then the first three frames of images are not enough to judge the movement trend of the human body, and P4=0.5 is defined. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the BiConA module structure;

[0043] Figure 2 Schematic diagram of the human body segmentation network structure;

[0044] Figure 3 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with the accompanying drawings.

[0046] Example 1: See Figures 1 to 3 ,A security inspection and anti-intrusion detection method based on semantic segmentation comprises the following steps;

[0047] S1, obtaining a data set for semantic segmentation, wherein samples in the data set are images labeled with target categories, and the target category is human body;

[0048] S2, construct BiConA module;

[0049] The BiConA module is used to input the first feature map C1 and the second feature map C2, concatenate C1 and C2 through 3×3 convolution to obtain the third feature map C3, perform average pooling of convolution kernel sizes of 1, 3, and 5 on C3, and then perform 1×1 convolution to obtain the fourth feature map C4, the fifth feature map C5, and the sixth feature map C6, and then obtain the i-th difference feature map CF according to the following formula: i And the output feature map of the BiConA module ;

[0050] ,

[0051] ,

[0052] Where, 1≤i≤3, It is a 1×1 convolution operation;

[0053] S3, construct a human segmentation network, including an encoder module, a BiConA module and a decoder;

[0054] The encoder module is an encoder module in the SegFormer network, including four layers of Transformer encoders TB1-TB4 connected in sequence, which are used to input samples in the data set and obtain feature maps of different scales through TB1-TB4;

[0055] The input end of the BiConA module is connected to TB2 and TB4, and is used to use the output of TB2 as the first characteristic map C1 of the BiConA module, and the output of TB4 as the second characteristic map C2 of the BiConA module to obtain F BiConA ;

[0056] The decoder is used to input F BiConA and C2, after decoding, outputs the semantic segmentation result, and uses the semantic segmentation result as the human body segmentation image;

[0057] S4, use the data set to train the human segmentation network to obtain a human segmentation model;

[0058] S5, obtain the surveillance video containing the security inspection area, split it into a single-frame surveillance image set I={I t}, 1≤t≤T, T is the total number of images, I t is the tth image in I, from I t Mark the security check area X t , set I={I t} Input the human body segmentation model and get I t The corresponding human segmentation image A t , forming a segmented image set A={A t};

[0059] S6, confidence calculation based on multiple influencing factors, including S61~S63;

[0060] S61, from A t Get the irregular area of ​​the target and calculate the area S of the irregular area t , perimeter L t , center point O t , calculate O t-3 , O t-2 , O t-1 With X t The distance d from the center point t-3 d t-2 d t-1 , where O t-3 , O t-2 , O t-1 A t-3 , A t-2 , A t-1 The center point of the irregular area;

[0061] S62, calculate S t With security area X t The overlap ratio P1,L t With X t The coverage rate P2,X t Position P3, human movement trend P4, and P1~P4 as influencing factors;

[0062] Among them, if O t Located in X t If yes, P3=1, otherwise P3=0;

[0063] If d t-3 >d t-2 >d t-1 , then P4=1, d t-3 <d t-2 <d t-1 , then P4=0, otherwise P4=0.5,

[0064] S63, calculate A according to the following formula t The confidence level R t ;

[0065] R t =ω1P1+ω2P2+ω3P3+ω4P 4，

[0066] Where ω1, ω2, ω3, and ω4 are the empirical coefficients of P1, P2, P3, and P4 respectively;

[0067] S7, preset confidence threshold R l , compared with R t and R l , if R t >R l , it is judged that there is an intrusion in the security inspection area.

[0068] Regarding irregular regions in this embodiment: When semantic segmentation is performed on an image, each pixel in the image is assigned to a predefined semantic category and a semantic segmentation mask, also called a semantic segmentation result or a semantic segmentation map, is output. The target in the image is not a regular rectangle, but an irregular region that contains all pixels of the target.

[0069] Example 2, see Figures 1 to 3 , based on Example 1:

[0070] In S1, the dataset is the Cityscapes dataset.

[0071] The decoder is a decoder of the DeepLabv3+ network. The BiConA module includes a 3×3 convolution layer, a first concatenation layer, a first average pooling layer, a second average pooling layer, a third average pooling layer, an i-th difference feature layer, and a feature fusion layer, wherein the convolution kernel sizes of the first average pooling layer, the second average pooling layer, and the third average pooling layer are 1, 3, and 5, respectively, and 1≤i≤3; the first feature map C1 and the second feature map C2 are respectively subjected to a 3×3 convolution layer and a first concatenation layer to obtain a third feature map C3; the third feature map C3 is divided into three paths, one path is subjected to a first average pooling layer and a 1×1 convolution layer to obtain a fourth feature map C4, one path is subjected to a second average pooling layer and a 1×1 convolution layer to obtain a fifth feature map C4, and one path is subjected to a third average pooling layer and a 1×1 convolution layer to obtain a sixth feature map C6; the i-th difference feature layer includes an i-th difference layer and a 1×1 convolution layer, wherein the i-th difference layer is used to calculate C i With C i+3 The 1×1 convolution layer is used to perform a 1×1 convolution operation on the difference to obtain the i-th difference feature map CF i ; The feature fusion layer is used according to the formula Perform feature fusion to obtain the output feature map F of the BiConA module.

[0072] In S62, , , S7, 0.6≤R l ≤0.8.

[0073] Example 3, see Figures 1 to 3 Based on Example 1, the feature fusion layer can be specifically subdivided into a first feature fusion layer, a second feature fusion layer and a third feature fusion layer, wherein the first feature fusion layer and the second feature fusion layer execute formula , The third feature fusion layer is used to combine the outputs of the first feature fusion layer and the second feature fusion layer with Add.

[0074] Example 4: Based on Example 1, we selected the Cityscapes dataset as the semantic dataset, and divided the dataset into a training set and a validation set at a ratio of 8:2. Four existing methods were compared with the method of the present invention, and objective comparison indicators such as accuracy and mean intersection-over-union (MioU) were used for experiments. The objective comparison indicators of the experimental results of each model are shown in Table 1.

[0075] Table 1. Comparison of multi-model experimental results

[0076] method Accuracy% mIoU % SegNet 77.65 66.83 DeeplabV3+ 84.30 76.24 PSPNet 85.44 78.02 HRNet 87.72 81.51 Human body segmentation model of the present invention 89.76 82.76

[0077] It can be seen from Table 1 that the human body segmentation model obtained by the present invention is superior to the four existing methods in terms of accuracy and average intersection-over-union ratio. By combining the human body segmentation model and the method for calculating confidence using multiple influencing factors described in step S6 of the present invention, the anti-intrusion detection method finally obtained is more accurate.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A security inspection and anti-intrusion detection method based on semantic segmentation, characterized by: The steps include: S1, obtaining a data set for semantic segmentation, wherein samples in the data set are images labeled with target categories, and the target category is human body; S2, construct BiConA module; The BiConA module is used to input the first feature map C1 and the second feature map C2, concatenate C1 and C2 through 3×3 convolution to obtain the third feature map C3, perform average pooling of convolution kernel sizes of 1, 3, and 5 on C3, and then perform 1×1 convolution to obtain the fourth feature map C4, the fifth feature map C5, and the sixth feature map C6, and then obtain the i-th difference feature map CF according to the following formula: i And the output feature map of the BiConA module ; ， ， Where, 1≤i≤3, It is a 1×1 convolution operation; S3, construct a human segmentation network, including an encoder module, a BiConA module and a decoder; The encoder module is an encoder module in the SegFormer network, including four layers of Transformer encoders TB1-TB4 connected in sequence, which are used to input samples in the data set and obtain feature maps of different scales through TB1-TB4; The input end of the BiConA module is connected to TB2 and TB4, and is used to use the output of TB2 as the first characteristic map C1 of the BiConA module, and the output of TB4 as the second characteristic map C2 of the BiConA module to obtain F BiConA ; The decoder is used to input F BiConA and C2, after decoding, outputs the semantic segmentation result, and uses the semantic segmentation result as the human body segmentation image; S4, use the data set to train the human segmentation network to obtain a human segmentation model; S5, obtain the surveillance video containing the security inspection area, split it into a single-frame surveillance image set I={I t }, 1≤t≤T, T is the total number of images, I t is the tth image in I, from I t Mark the security check area X t , set I={I t } Input the human body segmentation model and get I t The corresponding human segmentation image A t , forming a segmented image set A={A t }; S6, confidence calculation based on multiple influencing factors, including S61~S63; S61, from A t Get the irregular area of ​​the target and calculate the area S of the irregular area t , perimeter L t , center point O t , calculate O t-3 , O t-2 , O t-1 With X t The distance d from the center point t-3 d t-2 d t-1 , where O t-3 , O t-2 , O t-1 A t-3 , A t-2 , A t-1 The center point of the irregular area; S62, calculate S t With security area X t The overlap ratio P1,L t With X t The coverage rate P2,X t Position P3, human movement trend P4, and P1~P4 as influencing factors, , ; Among them, if O t Located in X t If yes, P3=1, otherwise P3=0; If d t-3 >d t-2 >d t-1 , then P4=1, d t-3 <d t-2 <d t-1 , then P4=0, otherwise P4=0.5, S63, calculate A according to the following formula t The confidence level R t ; R t =ω1P1+ω2P2+ω3P3+ω4P 4， Where ω1, ω2, ω3, and ω4 are the empirical coefficients of P1, P2, P3, and P4 respectively; S7, preset confidence threshold R l , compared with R t and R l , if R t >R l , it is judged that there is an intrusion in the security inspection area.

2. The security inspection and anti-intrusion detection method based on semantic segmentation according to claim 1 is characterized by: The dataset is the Cityscapes dataset.

3. The security inspection and anti-intrusion detection method based on semantic segmentation according to claim 1 is characterized by: The decoder is a decoder of the DeepLabv3+ network.

4. The security inspection and anti-intrusion detection method based on semantic segmentation according to claim 1 is characterized by: The BiConA module includes a 3×3 convolution layer, a first concatenation layer, a first average pooling layer, a second average pooling layer, a third average pooling layer, an i-th difference feature layer, and a feature fusion layer, wherein the convolution kernel sizes of the first average pooling layer, the second average pooling layer, and the third average pooling layer are 1, 3, and 5, respectively, and 1≤i≤3; The first feature map C1 and the second feature map C2 are respectively subjected to a 3×3 convolution layer and a first concatenation layer to obtain a third feature map C3; The third feature map C3 is divided into three paths. One path passes through the first average pooling layer and the 1×1 convolution layer to obtain the fourth feature map C4, one path passes through the second average pooling layer and the 1×1 convolution layer to obtain the fifth feature map C4, and one path passes through the third average pooling layer and the 1×1 convolution layer to obtain the sixth feature map C6; The i-th difference feature layer includes the i-th difference layer and the 1×1 convolution layer, where the i-th difference layer is used to calculate C i With C i+3 The 1×1 convolution layer is used to perform a 1×1 convolution operation on the difference to obtain the i-th difference feature map CF i ; The feature fusion layer is used according to the formula Perform feature fusion to obtain the output feature map of the BiConA module .

5. The security inspection and anti-intrusion detection method based on semantic segmentation according to claim 4 is characterized by: The feature fusion layer includes a first feature fusion layer, a second feature fusion layer and a third feature fusion layer, wherein the first feature fusion layer and the second feature fusion layer execute the formula , The third feature fusion layer is used to combine the outputs of the first feature fusion layer and the second feature fusion layer with Add.

6. The security inspection and anti-intrusion detection method based on semantic segmentation according to claim 1 is characterized by: 0.6≤R l ≤0.8。

Images