Early warning method and device based on anti-terrorist anti-riot anomaly detection

Abstract

The invention discloses an early warning method and device based on anti-terrorist and anti-riot anomaly detection. The method comprises the steps that multi-source situation data of a detection task area is acquired, video stream data in the multi-source situation data is adopted to carry out instrument holding target detection, and an instrument holding target screening result is obtained; performing suspicious behavior detection by adopting picture frame data in the multi-source situation data to obtain a behavior suspicious target screening result; target space-time trajectory data in the multi-source situation data are adopted for target yaw trajectory detection, and a yaw target screening result is obtained; and based on the instrument holding target screening result, the behavior suspicious target screening result and the yaw target screening result, the threat level of the detection task area is judged, and early warning is performed based on the threat level. According to the method, complementation is formed by using multi-source situation data, the trend of a suspicious target is mastered more comprehensively, real-time sensing of a field situation is realized, and the crisis response and disposal capability in anti-terrorism and anti-riot decision is improved.

Description

Technical Field

[0001] The present invention belongs to the field of anti-terrorism and anti-riot monitoring and early warning technology, and specifically relates to an early warning method, device, electronic device and computer-readable storage medium based on anti-terrorism and anti-riot anomaly detection. Background Art

[0002] Peacekeeping counter-terrorism and anti-riot mission areas usually cover complex terrain and climate environments, resulting in complex and diverse situation information, with a large number of types and quantities. Moreover, terrorist targets are usually camouflaged and hidden, with unusual and changeable movement types, and will be disturbed by the background crowd, which seriously affects the decision-making of counter-terrorism and anti-riot emergency response.

[0003] Existing anti-terrorism and riot prevention early warning technologies are aimed at normal areas, such as private residences, shopping malls, public areas, etc. The detection methods used are not targeted at anti-terrorism and riot prevention scenarios, and cannot accurately obtain threat information in terrorism and riot prevention scenarios; moreover, the existing anti-terrorism and riot prevention detection methods are single, lacking complementary linkage and integration of situation information, making it difficult to form a unified and integrated comprehensive assessment of multi-level threat information, and unable to meet the safety and rapid response requirements of anti-terrorism and riot prevention emergency response decisions. Summary of the invention

[0004] The purpose of the present invention is to comprehensively obtain information on the situation of violent terrorist attacks and to integrate and complement each other, so as to form a unified and integrated comprehensive judgment on multi-level threat information, improve the crisis response and handling capabilities in anti-terrorism and anti-riot decision-making, and solve the problems existing in the background technology.

[0005] To achieve the above objectives, in a first aspect, the present invention provides an early warning method based on anti-terrorism and anti-riot anomaly detection, comprising:

[0006] Acquire multi-source situation data of the detection task area, wherein the multi-source situation data is multi-modal situation data obtained by multiple detection and perception devices detecting the same detection task area, and the multi-source situation data includes video stream data, picture frame data, and target spatiotemporal trajectory data;

[0007] The video stream data in the multi-source situation data is used to detect armed targets, and the armed targets in the detection task area are screened to obtain the armed target screening results;

[0008] Use the picture frame data in the multi-source situation data to detect suspicious behaviors, screen suspicious behavior targets in the detection task area, and obtain the screening results of suspicious behavior targets;

[0009] The target spatiotemporal trajectory data in the multi-source situation data is used to detect the target deviation trajectory, and the deviation targets in the detection task area are screened to obtain the deviation target screening results;

[0010] Based on the armed target screening results, the suspicious behavior target screening results and the off-course target screening results, the threat level of the detection mission area is determined, and an early warning is issued based on the threat level.

[0011] In some embodiments of the present invention, video stream data in multi-source situation data is input into a pre-trained armed target detection neural network to perform armed target detection, and to screen and detect armed targets in the task area; wherein the armed target detection includes multi-layer feature extraction, pooling, upsampling, multi-scale feature fusion, key feature attention and multi-scale feature detection.

[0012] In some embodiments of the present invention, the armed target detection neural network includes a feature extraction module, a pooling module, a feature fusion module and a target detection module; wherein the video stream data in the multi-source situation data is input into the feature extraction module for multi-layer feature extraction, one output of the feature extraction module is input into the feature fusion module, and the other output is input into the feature fusion module after being pooled by the pooling module. The feature fusion module upsamples the two inputs, performs multi-scale feature fusion and focuses on key features, and the output of the feature fusion module is input into the target detection module. The target detection module performs multi-scale feature detection and then outputs the detection result of the armed target.

[0013] In some embodiments of the present invention, the pooling module includes a first convolution module, a second convolution module, a first strip pooling module, a second strip pooling module, a third strip pooling module, a first maximum pooling module, a second maximum pooling module, a third maximum pooling module, a first splicing module and a second splicing module; wherein the output of the feature extraction module is input into the first convolution module; the first output of the first convolution module is directly input into the first splicing module, the second output of the first convolution module is subjected to three strip poolings via the first strip pooling module, the second strip pooling module and the third strip pooling module, and the third output of the first convolution module is subjected to three strip poolings via the first maximum pooling module, the second maximum pooling module and the third strip pooling module. and the third maximum pooling module for three maximum pooling; the outputs of the first strip pooling module and the second strip pooling module are multiplied by the first weight and input into the first splicing module, the outputs of the first maximum pooling module and the second maximum pooling module are multiplied by the second weight and input into the first splicing module, the first splicing module splices the five-way input and inputs the second splicing module as output, the output of the third strip pooling module is multiplied by the first weight and input into the second splicing module, the output of the third maximum pooling module is multiplied by the second weight and input into the second splicing module, the second splicing module splices the three-way input and inputs into the second convolution module, and the second convolution module outputs the final pooling result after convolution.

[0014] In some embodiments of the present invention, picture frame data in multi-source situation data is input into a pre-trained suspicious behavior detection neural network for suspicious behavior detection, and suspicious behavior targets in the detection task area are screened; wherein the suspicious behavior detection neural network predicts a predicted frame of the corresponding N+1th picture frame for N consecutive picture frame inputs, thereby generating a predicted frame with a time sequence, and calculates the error between the predicted frame and the picture frame corresponding to the same time point, and determines the suspicious behavior target and its threatening intention corresponding to the time point according to whether the error exceeds a preset threshold.

[0015] In some embodiments of the present invention, the suspicious behavior detection neural network is a two-way constrained contrast detection network, which uses optical flow frame constraints and diffusion frame constraints to jointly optimize the network loss of the suspicious behavior detection neural network.

[0016] In some embodiments of the present invention, the target spatiotemporal trajectory data in the multi-source situation data is input into a pre-trained target yaw trajectory detection neural network to perform target yaw trajectory detection, and screen the yawed targets in the detection task area; wherein the target yaw trajectory detection neural network is a residual BiLSTM neural network.

[0017] In a second aspect, the present invention provides an early warning device based on anti-terrorism and anti-riot anomaly detection, comprising:

[0018] A data acquisition module is used to acquire multi-source situation data of the detection task area. The multi-source situation data is multi-modal data obtained by multiple detection and perception devices detecting the same detection task area. The multi-source situation data includes video stream data, picture frame data and target spatiotemporal trajectory data;

[0019] An armed target screening module is used to detect armed targets using video stream data in multi-source situation data, screen armed targets in the detection task area, and obtain armed target screening results;

[0020] A suspicious target screening module is used to detect suspicious behaviors using picture frame data in multi-source situation data, screen suspicious behavior targets in the detection task area, and obtain suspicious behavior target screening results;

[0021] The off-course target screening module is used to detect the off-course trajectory of the target using the target spatiotemporal trajectory data in the multi-source situation data, screen the off-course targets in the detection task area, and obtain the off-course target screening results;

[0022] The threat judgment and early warning module is used to judge the threat level of the detection mission area based on the armed target screening results, the suspicious behavior target screening results and the off-course target screening results, and to issue an early warning based on the threat level.

[0023] In a third aspect, the present invention provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the early warning method based on anti-terrorism and riot control anomaly detection.

[0024] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the early warning method based on anti-terrorism and riot prevention anomaly detection.

[0025] The present invention has the following beneficial effects:

[0026] The present invention utilizes multi-source situation data to complement each other, more comprehensively grasps the movements of suspicious targets, realizes real-time perception of on-site situations, and improves crisis response and handling capabilities in anti-terrorism and anti-riot decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The figure is a flow chart of an early warning method based on anti-terrorism and anti-riot anomaly detection in one embodiment of the present invention.

[0028] Figure 2 It is a flow chart of an early warning method based on anti-terrorism and anti-riot anomaly detection in another embodiment of the present invention.

[0029] Figure 3 FIG. 4 is a flow chart of armed target detection in one embodiment of the present invention.

[0030] Figure 4 FIG. 4 is a schematic diagram of the composition of a neural network for detecting armed targets in one embodiment of the present invention.

[0031] Figure 5 FIG. 4 is a schematic diagram of the composition of a pooling module in one embodiment of the present invention.

[0032] Figure 6 FIG. 1 is a schematic diagram of the module connection of an armed target detection neural network in one embodiment of the present invention.

[0033] Figure 7 FIG. 4 is a flow chart of suspicious behavior detection in one embodiment of the present invention.

[0034] Figure 8 Schematic diagram of a two-hop memory U-Net network structure in one embodiment of the present invention.

[0035] Fig. 9 Schematic diagram of the principle of training a dual-path constrained contrast detection network DCC-UNet in one embodiment of the present invention.

[0036] Fig.10 FIG. 4 is a schematic diagram of a flow chart of target yaw trajectory detection in one embodiment of the present invention.

[0037] Fig.11 Schematic diagram of the basic structure of LSTM in one embodiment of the present invention.

[0038] Fig.12 Schematic diagram of the residual double BiLSTM network structure in one embodiment of the present invention.

[0039] Fig.13 FIG. 1 is a schematic diagram of error accumulation using LSTM multi-step prediction in one embodiment of the present invention.

[0040] Fig.14 Schematic diagram of a dynamic time step prediction mechanism using LSTM in one embodiment of the present invention.

[0041] Fig.15 The figure is a schematic diagram of the threat assessment and early warning principle in one embodiment of the present invention.

[0042] Fig.16 The figure is a schematic diagram of the composition of an early warning device in one embodiment of the present invention.

[0043] Fig.17 The figure is a schematic diagram of the composition of an electronic device in one embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0045] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0046] The particularity of the application scenario of violent terrorist emergencies lies in the requirements for accuracy and speed of handling decisions. The mission area usually covers an environment with complex terrain and climate, and the situation information is often complex and diverse, with a large number of types and quantities. The target movement types are abnormal and changeable, and the background crowds are sometimes disturbing. All these factors lead to high safety requirements and rapid response requirements for emergency handling decisions.

[0047] The large amount of multi-dimensional data obtained by the front-end detector of situational awareness urgently needs effective analysis and processing methods, which require accurate extraction of potential threat factors of the target from multi-source data, threat assessment, effective warning, and reliable judgment for decision makers. Potential threat targets have distinct characteristics: potential threat individuals may be active at abnormal times, often wandering around, refusing to cooperate, etc., and their patterns are often inconsistent with the general behavior of the background population. Potential threat groups may carry the same signs and patterns on their skin or personal belongings, often gather illegally, provoke conflicts, etc. Targets usually hold or try to hide dangerous weapons such as "knives", "guns", and "sticks", and their movement trajectories and speeds are often sudden.

[0048] Therefore, the present invention provides an early warning method, device, platform and computer-readable storage medium based on anti-terrorism and anti-riot anomaly detection. The purpose is to comprehensively obtain information on the situation of violent terrorist incidents, integrate and complement them, realize unified and integrated comprehensive analysis of multi-level threat information, and improve the crisis response and handling capabilities in anti-terrorism and anti-riot decision-making.

[0049] First, as Figure 1 As shown, an early warning method based on anti-terrorism and anti-riot anomaly detection in one embodiment of the present invention includes:

[0050] Step S100, obtaining multi-source situation data of the detection task area, wherein the multi-source situation data is multi-modal situation data obtained by multiple detection and perception devices detecting the same detection task area, and the multi-source situation data includes video stream data, picture frame data, and target spatiotemporal trajectory data;

[0051] Step S200, using the video stream data in the multi-source situation data to perform armed target detection, screening armed targets in the detection task area, and obtaining armed target screening results;

[0052] Step S300, using the picture frame data in the multi-source situation data to perform suspicious behavior detection, screening suspicious behavior targets in the detection task area, and obtaining a suspicious behavior target screening result;

[0053] Step S400, using the target spatiotemporal trajectory data in the multi-source situation data to perform target yaw trajectory detection, screening yaw targets in the detection task area, and obtaining yaw target screening results;

[0054] Step S500, based on the armed target screening results, the suspicious behavior target screening results and the off-course target screening results, the threat level of the detection mission area is determined, and an early warning is issued based on the threat level.

[0055] In this embodiment, step S200 and step S400 are implemented in a serial processing manner. However, in other embodiments of the present invention, step S200 and step S400 may also be implemented in a parallel processing manner. Figure 2 shown.

[0056] Furthermore, in some embodiments of the present invention, in step S200, the video stream data in the multi-source situation data is input into a pre-trained armed target detection neural network to perform armed target detection, and the armed targets in the detection task area are screened; wherein, Figure 3 As shown, the armed target detection sequentially performs multi-layer feature extraction, pooling, upsampling, multi-scale feature fusion, key feature attention and multi-scale feature detection.

[0057] Specifically, Figure 4 As shown, the armed target detection neural network includes a feature extraction module, a pooling module, a feature fusion module and a target detection module; wherein, the video stream data in the multi-source situation data is input into the feature extraction module for multi-layer feature extraction, one output of the feature extraction module is input into the feature fusion module, and the other output is input into the feature fusion module after being pooled by the pooling module. The feature fusion module performs upsampling, multi-scale feature fusion and key feature attention on the two inputs, and the output of the feature fusion module is input into the target detection module. The target detection module performs multi-scale feature detection and then outputs the detection result of the armed target.

[0058] Pooling includes maximum pooling, average pooling, global average pooling, random pooling, etc. Preferably, in some embodiments of the present invention, in order to avoid false detection, missed detection and mixed detection in anti-terrorism and anti-riot scenarios, distinguish between people and dangerous equipment, and specifically detect abnormal targets of dangerous equipment, the pooling module adopts the following method: Figure 5The symmetrical weighted pooling structure shown includes a first convolution module, a second convolution module, a first strip pooling module, a second strip pooling module, a third strip pooling module, a first maximum pooling module, a second maximum pooling module, a third maximum pooling module, a first splicing module and a second splicing module; the output of the feature extraction module is input into the first convolution module; the first output of the first convolution module is directly input into the first splicing module, the second output of the first convolution module is subjected to three strip poolings via the first strip pooling module, the second strip pooling module and the third strip pooling module, and the third output of the first convolution module is subjected to three strip poolings via the first maximum pooling module, the second maximum pooling module and the third strip pooling module. And the third maximum pooling module performs three maximum pooling; the outputs of the first strip pooling module and the second strip pooling module are multiplied by the weight w1 and then input into the first splicing module, the outputs of the first maximum pooling module and the second maximum pooling module are multiplied by the weight w2 and then input into the first splicing module, the first splicing module splices the five-way input and inputs the second splicing module, the output of the third strip pooling module is multiplied by the weight w1 and then input into the second splicing module, the output of the third maximum pooling module is multiplied by the weight w2 and then multiplied into the second splicing module, the second splicing module splices the three-way input and inputs the second convolution module, and the second convolution module outputs the final pooling result.

[0059] In anti-terrorism and anti-riot scenarios, the above-mentioned pooling module can detect targets carrying weapons (such as guns, knives, sticks, etc.) without missing or false positives and with high accuracy.

[0060] Specifically, in some embodiments of the present invention, the structure of the armed target detection neural network is as follows: Figure 6 The meanings of the modules in the figure are as follows:

[0061] CBS: A basic convolutional combination module consisting of a convolutional layer Conv, a batch normalization layer BN, and an activation function Silu.

[0062] C3: A module composed of CBS, including three CBS modules, a bottleneck module and a Concat operation. Two CBS modules form the bottleneck layer.

[0063] Concat: A concatenation operation that concatenates the input in the channel dimension and outputs the result.

[0064] SPWP: A new pooling layer used in the embodiment of the present invention, Symmetric Pyramid Weighted Pooling, realizes maximum pooling and strip pooling at the same time, and also includes weighting coefficients and concatenation operations.

[0065] Upsample: An upsampling operation that expands the scale of the input.

[0066] GSConv: A convolution module that is lighter than the CBS module. It divides the input into two parts, one for ordinary convolution operation and the other for depth-wise separable convolution operation.

[0067] SA: A channel shuffle attention mechanism (Shuffle Attention), which groups the input, performs spatial and channel attention calculations separately, and finally uses Concat to concatenate.

[0068] Detect: A detection module that performs both classification and bounding box regression and can accept inputs of different scales.

[0069] Specifically, the input image is transmitted in the armed target detection neural network in the form of a feature map, and the transmission process is:

[0070] The first column includes the feature extraction module and the pooling module. The input image size is [640,640,3], which means the image width x height = 640x640 and the number of channels = 3. After the 0th layer CBS is transformed into the initial feature map, the output size is [320,320,64], and after the 1st-8th layer feature extraction module, the output size is [20,20,1024]; then it is input to the 9th layer SPWP for pooling, and the output size is [20,20,1024].

[0071] The second and third columns are feature fusion modules. The output feature map of the 9th layer passes through the 10th layer CBS and the 11th layer Upsample module, and the output size is [40,40,512]. It is concatenated with the feature map of the same size output by the 6th layer in the channel dimension at the 12th layer, and the size after concatenation is [40,40,1024]. It then passes through the 13th layer C3, the 14th layer CBS, and the 15th layer Upsample module, and the output size is [80,80,256]. It is concatenated with the feature map of the same size output by the 4th layer in the channel dimension at the 16th layer, and the size after concatenation is [80,80,512]. The feature map then passes through the 17th layer C3, and the output size is [80,80,256]. After passing through the 18th layer of GSConv, the output size is [40, 40, 256], and the feature map of the same size output by the 14th layer is concat-spliced ​​in the channel dimension at the 19th layer, and the size after splicing is [40, 40, 512]. After splicing, it passes through the 20th layer of C3, and the output size is [40, 40, 512]. Finally, the feature map passes through the 21st layer of GSConv, and the output size is [20, 20, 512]. It is concat-spliced ​​with the feature map of the same size output by the 10th layer at the 22nd layer in the channel dimension, and the size after splicing is [20, 20, 1024]. After splicing, it passes through the 23rd layer of C3 and the 24th layer of SA, and the output size is [20, 20, 1024].

[0072] The fourth column is the target detection module. It receives the output of the 17th, 20th, and 24th layers, covering the feature information of the shallow, middle, and high layers, and can predict small, medium, and large-scale targets. The 1*1 convolutional block of the detection layer converts the number of channels of the feature map to (number of categories + 5) x the number of anchors of each detection layer, where "5" represents the horizontal coordinate, vertical coordinate, width, height, and confidence of the center point of the prediction box. Here we set three categories of "stick", "knife", and "gun", and each detection layer has 3 anchors, so the number of reshaped channels is 24, that is, the Detect module receives the output of the 17th, 20th, and 24th layers as input, and the output sizes are [80,80,24], [40,40,24], [20,20,24], respectively. Then, the redundant prediction boxes are removed according to non-maximum suppression.

[0073] In particular, the symmetric pyramid weighted pooling module SPWP proposed in the present invention refers to Figure 5 ,The pooling structure uses both band pooling and maximum pooling to form a symmetrical pyramid structure. Before concatenating the two pooling layers, the band pooling weight w1 and the maximum pooling weight w2 are used to weight the concatenated input respectively.

[0074] Specifically, the symmetric pyramid weighted pooling module SPWP is located at the 9th layer in the armed target detection neural network, and receives the output from the 8th layer as input, with an input size of [20, 20, 1024]. After a CBS convolution, the output size is [20, 20, 512]. Then this output enters two pooling branches, where the left branch performs three strip poolings in sequence, the size remains unchanged after pooling, and the output after each pooling is multiplied by the weight coefficient ω1; the right branch performs three maximum poolings in sequence, the size remains unchanged after pooling, and the output after each pooling is multiplied by the weight coefficient ω2; the two side branches have a total of 6 outputs, all of which have a size of [20, 20, 512]. Concat operation is performed with the original output of the same size, and after a CBS convolution, the output size is [20, 20, 1024].

[0075] For the training of the armed target detection neural network, we use a training set containing three categories of "stick", "knife" and "gun" for training. After the training is completed, the monitoring video stream data is input into the model for prediction. After feature extraction, it is pooled at the SPWP structure, and the pooled feature map is upsampled in the network fusion module to obtain a multi-scale feature map. The features of multiple scales are fused, and the channel shuffle attention module is used to focus on key features. Finally, the results of multi-scale feature detection are output, including the detection box position, category and confidence information of the target. The detection box position and category of the target are used as real-time outputs of the submodule, and the confidence level information carrying the threat capability is passed to step S500 for threat assessment and early warning.

[0076] Further, in some embodiments of the present invention, in step S300, Figure 7 As shown, the process of suspicious behavior detection includes segmenting the target frame to be detected, generating a prediction frame, performing joint loss optimization on the generated prediction frame based on the diffusion frame constraint and the optical flow frame constraint, comparing the prediction error and the set abnormal threshold based on the abnormal score curve, and performing suspicious behavior judgment based on the comparison result.

[0077] Specifically, in some embodiments of the present invention, in step S300, the picture frame data in the multi-source situation data is input into a pre-trained suspicious behavior detection neural network to perform suspicious behavior detection, and screen suspicious behavior targets in the detection task area; wherein the suspicious behavior detection neural network predicts the corresponding predicted frame of the N+1th picture frame for the consecutive N picture frame inputs, thereby generating a predicted frame with a time sequence, and calculates the error between the predicted frame and the picture frame corresponding to the same time point, and determines the suspicious behavior target and its threatening intention corresponding to the time point according to whether the error exceeds a preset threshold.

[0078] In some embodiments of the present invention, the suspicious behavior detection neural network is a two-way constrained contrast detection network, which uses optical flow frame constraints and diffusion frame constraints to jointly optimize the network loss of the suspicious behavior detection neural network.

[0079] Specifically, the construction of the suspicious behavior detection neural network is to first improve the U-Net network structure into a two-hop memory U-Net, and then construct a dual constraint comparison detection network (DCC-UNet) based on the two-hop memory U-Net, which is used to identify suspicious behaviors of targets in the task area.

[0080] The two-hop memory U-Net network structure is as follows Figure 8 As shown. Assume that the size of the input data key picture frame is width x height = 256x256, and the initial number of channels of the RGB image is 3, that is, the input size is [256,256,3]. The first convolution downsampling module downblock1 first performs two convolution operations to expand the input in the channel dimension, and the convolution output size is [256,256,64], followed by one downsampling, and the downsampling output size is [128,128,64]; the second convolution downsampling module downblock2 first performs two convolution operations, and the convolution output size is [128,128,128], followed by one downsampling, and the downsampling output size is [64,64,128]; the third convolution downsampling module downblock3 first performs two convolution operations, and the convolution output size is [64,64,256], followed by one downsampling, and the downsampling output size is [32,32,256].

[0081] Mem is a memory module that stores feature maps and is used for addressing. The output of the third convolutional downsampling module downblock3 is first subjected to a convolution operation, and the size becomes [32,32,512]. Then the addressed feature map is output after a convolution operation, and the size remains [32,32,512] as the input of the third upsampling convolutional module upblock3.

[0082] The third upsampling convolution module upblock3 receives the feature map with an input of [32, 32, 512], and first performs an upsampling operation. The upsampling output size is [64, 64, 512]. At this time, the convolution output of the third convolution downsampling module downblock3 with a size of [64, 64, 256] is spliced ​​through a jump connection, and then two convolution operations are performed. The convolution output size is [64, 64, 256]; the second upsampling convolution module upblock2 receives the feature map with an input of [64, 64, 256], and first performs an upsampling operation. The sampling output size is [128, 128, 256]. At this time, the convolution output of downblock2 with a size of [128, 128, 128] is spliced ​​through a jump connection, and then two convolution operations are performed, and the convolution output size is [128, 128, 128]. The first upsampling convolution module upblock1 receives the feature map with an input of [128, 128, 128]. It first performs an upsampling operation, and the upsampling output size is [256, 256, 128]. Then, two convolution operations are performed, and the convolution output size is [256, 256, 64]. Finally, the output of the first upsampling convolution module upblock1 undergoes a convolution operation and changes its size to [256, 256, 3].

[0083] The dual-path constraint contrast detection network DCC-UNet built on the basis of the above two-hop memory U-Net is as follows Fig. 9 The pre-training process is as follows: The training data is a video segment divided into picture frames in time order, called real frames, denoted as I1, I2, …, I t ,I t+1 , I1,I2,…,I t Input the two-hop memory U-Net and generate the predicted t+1th frame through encoding and decoding It is called the prediction frame. In this process, the optical flow frame constraint and the diffusion frame constraint are used to jointly optimize the network loss.

[0084] Among them, the optical flow frame constraint means: the predicted frame and the real frame I at the previous moment t The optical flow is calculated by the Flownet module, and the real frame I at the corresponding moment is t+1 and the real frame I at the previous moment t The optical flow is calculated through the Flownet module, and the L1 regularization loss is used to constrain their consistency.

[0085] Diffusion frame constraint means: from the real frame I t+1 Generate latent variables and generate diffusion frames with structural feature constraints through the encoding and decoding process of the pre-trained diffusion U-Net model Diffusion Frame With predicted frame They are all generated images after the encoding and decoding process, so they should be aligned in structural features to make the model prediction more accurate, and L2 regularization loss is used to constrain their consistency.

[0086] Finally, under the action of the two-way constraint, the discriminator predicts the frame and the real frame I t+1 To make comparisons and determine authenticity, the dual-path constrained contrast detection network DCC-UNet is trained as a good model that can accurately predict the next frame.

[0087] After the training is completed, the process of using the pre-trained suspicious behavior detection neural network to perform suspicious behavior detection includes:

[0088] Get data input and pre-process the data. The input data is usually a situation monitoring video of the mission area (such as a terrorist battlefield). When segmenting the video into detection frames, you can choose a frame segmentation strategy based on application requirements. For example, you can directly extract each frame of the image, and you can extract frames from the video if low latency is required.

[0089] The frame data is passed through the trained dual-path constrained contrast detection network DCC-UNet, and a method of predicting the next frame by five consecutive frames is adopted to generate a predicted frame with a time sequence.

[0090] The error between the predicted frame and the real frame is calculated, and the PSNR value is used here to represent it. The PSNR value is normalized to the interval [0,1], converted into an anomaly score curve and used as the output of the submodule. At the same time, the threshold of the anomaly score is set according to the empirical value. Frames exceeding the threshold are regarded as suspicious behaviors, and the judgment of suspicious behaviors in the submodule is given. The normalized anomaly score is used as the confidence level, and the hierarchical information carrying the threat intent is passed to step S500 for threat assessment and early warning.

[0091] Furthermore, in some embodiments of the present invention, in step S400, the target spatiotemporal trajectory data in the multi-source situation data is input into a pre-trained target yaw trajectory detection neural network to perform target yaw trajectory detection, and screen the yawed targets in the detection task area; wherein, the target yaw trajectory detection neural network is a residual BiLSTM neural network.

[0092] Specifically, Fig.10As shown, the target deviation trajectory detection implementation process is as follows: obtain the trajectory data of the target, first perform preprocessing, that is, data segmentation, remove some duplicate values, drift values, etc.; construct the residual BiLSTM model and perform model pretraining; use the trained model to predict the trajectory data, and use the predicted result and the error between the predicted value and the actual value as output to make an abnormal trajectory judgment. Set the error threshold according to the average error value of the pretrained model. If the threshold is exceeded, it is considered that this trajectory point is deviated, and a judgment of trajectory abnormality is given. The error between the above predicted value and the actual value is normalized to [0,1], and is passed to step S500 as the hierarchical information of the confidence carrying threat opportunity for threat assessment and early warning.

[0093] The detection and perception equipment can obtain the continuous trajectory of some targets in the current detection area, including coordinate information such as longitude and latitude, speed and distance information, etc. Using these spatiotemporal data, the trajectory of the target can be predicted, and the BiLSTM model can be trained to learn the rules of the first k trajectory points and predict the k+1th trajectory point. When there is a large deviation between the actual trajectory of the target and the predicted trajectory, or when the distance or speed of the target changes suddenly, the target is considered to have a threat opportunity.

[0094] Traditional recurrent neural networks are prone to the "vanishing gradient" phenomenon when processing time-series tasks and cannot solve the long-term dependencies in trajectory sequences. The long short-term memory network (LSTM) network uses a unique gating logic to process long-term memory in time-series tasks, which can effectively alleviate the problem of vanishing gradients.

[0095] See also Fig.11 , the LSTM network uses a "gate" to control the amount of information passing through. t Indicates the current LSTM cell state, x t Indicates the current input, h t Indicates the current output, f t 、i t and t They are the three most important "gates" in LSTM, the forget gate, the input gate, and the output gate. represents the addition of two vectors, Represents the vector cross product.

[0096] The advantage of using Bi-LSTM is that it can capture both the forward and backward features of the input sequence. The impact of the trajectory's front-end on the back-end and the impact of the back-end on the front-end can be captured by the model at the same time, increasing the reliability of the extracted time features. The residual connection structure is used so that the output of the first-layer Bi-LSTM is connected to the output of the second layer in the form of a residual, ensuring that the multi-layer structure does not reduce the prediction effect.

[0097] Specifically, in some embodiments of the present invention, Fig.12As shown, the target yaw trajectory detection neural network adopts a residual double BiLSTM network, including an input layer composed of LSTM modules, an intermediate layer composed of LSTM modules, a fully connected layer and an activation layer. The input layer and the LSTM modules in the intermediate layer are connected to each other, the output end of the LSTM module of the input layer is connected one-to-one with the input end of the LSTM module of the intermediate layer, the output end of the LSTM module of the input layer is also connected to the output end of the corresponding LSTM module of the intermediate layer, the intermediate layer is connected to the fully connected layer, and the fully connected layer is connected to the activation layer.

[0098] Furthermore, single-step prediction using LSTM is more accurate, but only one future value is predicted at a time. In some embodiments of the present invention, multi-step prediction can be used to predict n future values, but since historical data and predicted data are used, the prediction error will increase over time, such as Fig.13 shown.

[0099] Taking into account the dynamic variability of application scenarios, preferably, in some embodiments of the present invention, a dynamic time step prediction mechanism is used to predict trajectory data using a trained model to alleviate the problem of LSTM error accumulation and the short time perceived by single-step prediction.

[0100] Specifically, see Fig.14 , let the current time stamp be t, the historical data update time interval be T, select a variable k value as the number of predicted data at the current time, which is dynamically changing. The variable k value is dynamically selected based on the timestamp t and whether the warning information is issued in the time period T starting from the previous time stamp tT. When the trajectory is predicted to be abnormal in the previous time period T, according to the fact correlation, it means that there is a high probability of abnormality in this period. Single-step prediction is used to update and predict until k is predicted. tY data to ensure accuracy; when no trajectory anomaly is predicted in the previous time period T, it means that the situation during this period is relatively stable, but we need to pay attention to the situation in a longer period of time, and use multi-step prediction to predict k at a time. tN Data, search for abnormal conditions as much as possible. In the next period T from timestamp t to timestamp t+T, update historical data and predict k again tN data.

[0101] Further, in some embodiments of the present invention, a simple implementation of step S500 is as follows: Fig.15As shown, it specifically includes: using the simple results of step S200-step S400 to perform threat assessment and early warning, that is, dividing the detection results according to threat factors. When it is detected that a person is holding a weapon such as a knife, gun, or stick, the target is considered to have threat capability, and 1 threat capability index is accumulated. When the inter-frame anomaly score is detected to exceed the set threshold, it is considered that suspicious behavior has occurred, and the target is considered to have threat intent, and 1 threat intent index is accumulated. When it is detected that there is a large deviation between the actual trajectory of the target and the predicted trajectory, or the target has a sudden change in distance or speed, the target is considered to have a threat opportunity, and 1 threat opportunity index is accumulated.

[0102] The warning level is determined based on the cumulative results of the threat index. When the cumulative index = 3, it is judged as a severe threat and a Level I warning is issued; when the cumulative index = 2, it is judged as a moderate threat and a Level II warning is issued; when the cumulative index = 1, it is judged as a mild threat and a Level III warning is issued.

[0103] Preferably, in some other embodiments of the present invention, in order to comprehensively and accurately perform threat assessment and early warning, steps S200 to S400 further include:

[0104] Step S200 outputs the results of multi-scale feature detection, including the detection box position, category and confidence information of the target. The detection box position and category of the target are used as real-time outputs of the submodule, and the confidence level information carrying the threat capability is passed to step S500 for threat assessment and early warning.

[0105] In step S300, the error between the predicted frame and the real frame is calculated, which is represented by the PSNR value. The PSNR value is normalized to the interval [0,1] and converted into an anomaly score curve as the output of the submodule. The normalized anomaly score is used as the confidence level, and the hierarchical information carrying the threat intent is passed to S500 for threat assessment and early warning.

[0106] In step S400, the predicted future results are used as the output of the submodule, and the error between the predicted value and the actual value is calculated. The error between the predicted value and the actual value is normalized to [0, 1] and passed to step S500 as the hierarchical information of the confidence carrying threat opportunities for threat assessment and early warning.

[0107]

[0108] In order to make full use of the hierarchical information carried by the confidence transmitted by steps S200-S400, it is stipulated that when the confidence of each step falls into the corresponding confidence interval in the first column of the table, an additive factor is reassigned to it. For example, when the confidence of step S200 falls into [0, α1), it is assigned the x1 factor; if the confidence of step S200 falls into [α1, α2), it is assigned the x2 factor, and so on.

[0109] According to the threat level regulations y1 < z1 < x1 < y2 < z2 < x2 < y3 < z3 < x3 of this application scenario, based on the confidence levels detected by different sub-modules, these factors are combined additively to classify the early warning levels.

[0110] Set the confidence levels α1 = 0.5 and α2 = 0.8. A feasible matrix is:

[0111]

[0112] The early warning thresholds are: [0,9] for low risk, mild threat, and level III early warning; (9,15] for medium risk, moderate threat, and level II early warning; (15,24] for high risk, severe threat, and level I early warning.

[0113] The present invention uses the data obtained from multi-source detectors for situation awareness and anomaly detection, can timely discover the potential threats of the target, analyze the threats of abnormal activities and give accurate early warnings, comprehensively master the movements of suspicious targets, realizes the real-time awareness of the decision-makers on the on-site situation, and improves the crisis response and emergency disposal capabilities in anti-terrorism and anti-riot decision-making.

[0114] The present invention can, in view of the complex terrain of the detection task area, the variety and quantity of targets, and the variable types of target movements, etc., adopt target perception data such as surveillance video streams, picture frames, and coordinate point trace matrices for anomaly detection, form a complementary situation, and more comprehensively master the movements of suspicious targets.

[0115] The present invention establishes and realizes a multi-level threat assessment according to real-time analysis and detection, can real-time master the key information of the threat level of the target, and thus effectively assist decision-makers in allocating disposal resources.

[0116] In the second aspect, as Fig.16 shown, the early warning device based on anti-terrorism and anti-riot anomaly detection in an embodiment of the present invention includes:

[0117] A data acquisition module, configured to acquire multi-source situation data of the detection task area, where the multi-source situation data is multi-modal data obtained by multiple detection and perception devices detecting the same detection task area, and the multi-source situation data includes video stream data, picture frame data, and target spatio-temporal trajectory data;

[0118] An armed target screening module, configured to use the video stream data in the multi-source situation data to detect armed targets, screen the armed targets in the detection task area, and obtain an armed target screening result;

[0119] A suspicious target screening module, configured to use the picture frame data in the multi-source situation data to detect suspicious behaviors, screen the behavior-suspicious targets in the detection task area, and obtain a behavior-suspicious target screening result;

[0120] The off-course target screening module is used to detect the off-course trajectory of the target using the target spatiotemporal trajectory data in the multi-source situation data, screen the off-course targets in the detection task area, and obtain the off-course target screening results;

[0121] The threat judgment and early warning module is used to judge the threat level of the detection mission area based on the armed target screening results, the suspicious behavior target screening results and the off-course target screening results, and to issue an early warning based on the threat level.

[0122] In a third aspect, the present invention provides an electronic device in one embodiment, such as Fig.17 As shown, the electronic device includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned early warning method based on anti-terrorism and riot prevention anomaly detection.

[0123] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits through interfaces, which are all well known in the art. The interface provides an interface between the bus and the transceiver, such as a communication interface and a user interface. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium through an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0124] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0125] In a fourth aspect, the present invention provides, in one embodiment, a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned early warning method embodiment based on anti-terrorism and anti-riot anomaly detection.

[0126] Existing anomaly detection technologies are often used in routine monitoring areas, such as private residences, shopping malls, public facilities, etc. The detection methods used are not targeted at emergency response incidents and cannot accurately obtain the diverse threat factors in terrorist response scenarios. The detection methods are single, lack complementarity and linkage, and cannot form unified and integrated high-level judgment criteria for multi-level threat factors, and cannot integrate situation information for system decision-making. They are even highly dependent on manual inspections and specialized monitoring personnel, with slow response speed and high resource consumption.

[0127] Therefore, the present invention utilizes multi-source situation data to form mutual evidence or supplement, solves the problem of efficient and accurate abnormal target detection in anti-terrorism and riot prevention scenarios, more comprehensively grasps the movements of suspicious targets, realizes real-time perception of on-site situations, and improves crisis response and handling capabilities in anti-terrorism and riot prevention decision-making.

[0128] The present invention coordinates multiple neural network functions, establishes a multi-level threat assessment system architecture, performs threat assessment and early warning through confidence, grasps key information on the target threat level, and can effectively assist decision makers in allocating disposal resources.

[0129] Those skilled in the art can understand from the above description that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including several instructions for making a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes but is not limited to various media that can store program codes, such as USB flash drives, mobile hard disks, magnetic storage devices, and optical storage devices.

[0130] In the several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules or units, which can be electrical, mechanical or other forms.

[0131] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0132] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0133] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0134] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. An early warning method based on anti-terrorism and anti-riot anomaly detection, characterized in that: include: Acquire multi-source situation data of the detection task area, wherein the multi-source situation data is multi-modal situation data obtained by multiple detection and perception devices detecting the same detection task area, and the multi-source situation data includes video stream data, picture frame data, and target spatiotemporal trajectory data; The video stream data in the multi-source situation data is used to detect armed targets, and the armed targets in the detection task area are screened to obtain the armed target screening results; Use the picture frame data in the multi-source situation data to detect suspicious behaviors, screen suspicious behavior targets in the detection task area, and obtain the screening results of suspicious behavior targets; The target spatiotemporal trajectory data in the multi-source situation data is used to detect the target deviation trajectory, and the deviation targets in the detection task area are screened to obtain the deviation target screening results; Based on the armed target screening results, the suspicious behavior target screening results and the off-course target screening results, the threat level of the detection mission area is determined, and an early warning is issued based on the threat level.

2. The method according to claim 1, characterized in that The video stream data in the multi-source situation data is input into the pre-trained armed target detection neural network for armed target detection, and the armed targets in the detection task area are screened; wherein the armed target detection includes multi-layer feature extraction, pooling, upsampling, multi-scale feature fusion, key feature attention and multi-scale feature detection.

3. The method according to claim 2, characterized in that The armed target detection neural network includes a feature extraction module, a pooling module, a feature fusion module and a target detection module; wherein the video stream data in the multi-source situation data is input into the feature extraction module for multi-layer feature extraction, one output of the feature extraction module is input into the feature fusion module, and the other output is input into the feature fusion module after being pooled by the pooling module. The feature fusion module performs upsampling, multi-scale feature fusion and key feature attention on the two inputs, and the output of the feature fusion module is input into the target detection module. The target detection module performs multi-scale feature detection and then outputs the detection result of the armed target.

4. The method according to claim 3, characterized in that The pooling module includes a first convolution module, a second convolution module, a first strip pooling module, a second strip pooling module, a third strip pooling module, a first maximum pooling module, a second maximum pooling module, a third maximum pooling module, a first splicing module and a second splicing module; wherein, The output of the feature extraction module is input into the first convolution module; the first output of the first convolution module is directly input into the first splicing module, the second output of the first convolution module is subjected to three strip poolings through the first strip pooling module, the second strip pooling module and the third strip pooling module, and the third output of the first convolution module is subjected to three maximum poolings through the first maximum pooling module, the second maximum pooling module and the third maximum pooling module; The outputs of the first strip pooling module and the second strip pooling module are multiplied by the first weight and input into the first splicing module respectively. The outputs of the first maximum pooling module and the second maximum pooling module are multiplied by the second weight and input into the first splicing module respectively. The first splicing module splices the five inputs and inputs them as output into the second splicing module. The output of the third strip pooling module is multiplied by the first weight and input into the second splicing module. The output of the third maximum pooling module is multiplied by the second weight and input into the second splicing module. The second splicing module splices the three inputs and inputs them into the second convolution module. The second convolution module outputs the final pooling result after convolution.

5. The method according to claim 1, characterized in that The picture frame data in the multi-source situation data is input into a pre-trained suspicious behavior detection neural network for suspicious behavior detection, and suspicious behavior targets in the detection task area are screened; wherein the suspicious behavior detection neural network predicts the corresponding predicted frame of the N+1th picture frame for the consecutive N picture frame inputs, thereby generating a predicted frame with a time sequence, and calculating the error between the predicted frame and the picture frame corresponding to the same time point, and judging the suspicious behavior target and its threatening intention corresponding to the time point according to whether the error exceeds a preset threshold.

6. The method according to claim 5, characterized in that The suspicious behavior detection neural network is a two-way constraint contrast detection network, which adopts optical flow frame constraint and diffusion frame constraint to jointly optimize the network loss of the suspicious behavior detection neural network.

7. The method according to claim 1, characterized in that The target spatiotemporal trajectory data in the multi-source situation data is input into a pre-trained target yaw trajectory detection neural network to perform target yaw trajectory detection, and screen the yawed targets in the detection task area; wherein the target yaw trajectory detection neural network is a residual BiLSTM neural network.

8. An early warning device based on anti-terrorism and anti-riot anomaly detection, comprising: A data acquisition module is used to acquire multi-source situation data of the detection task area. The multi-source situation data is multi-modal data obtained by multiple detection and perception devices detecting the same detection task area. The multi-source situation data includes video stream data, picture frame data and target spatiotemporal trajectory data; An armed target screening module is used to detect armed targets using video stream data in multi-source situation data, screen armed targets in the detection task area, and obtain armed target screening results; A suspicious target screening module is used to detect suspicious behaviors using picture frame data in multi-source situation data, screen suspicious behavior targets in the detection task area, and obtain suspicious behavior target screening results; The off-course target screening module is used to detect the off-course trajectory of the target using the target spatiotemporal trajectory data in the multi-source situation data, screen the off-course targets in the detection task area, and obtain the off-course target screening results; The threat judgment and early warning module is used to judge the threat level of the detection mission area based on the armed target screening results, the suspicious behavior target screening results and the off-course target screening results, and to issue an early warning based on the threat level.

9. An electronic device, comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the early warning method based on anti-terrorism and anti-riot anomaly detection as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, the early warning method based on anti-terrorism and anti-riot anomaly detection according to any one of claims 1 to 7 is implemented.

Images