Fence crossing behavior detection method based on multi-modal fusion

Abstract

The invention relates to a multi-modal fusion-based fence crossing behavior detection method, which comprises the following steps of: firstly, judging whether a person in a video is overlapped with a fence or not through a semantic segmentation algorithm module so as to judge whether the person is in contact with the fence or not, and then acquiring distance information between the person and the fence by utilizing a monocular depth estimation algorithm module; the abnormal behavior detection algorithm module is used for analyzing whether the behavior of the current person deviates from a normal mode, and finally, the detection results of the three algorithm modules are fused to comprehensively judge whether the person is crossing the fence; according to the method, multiple information sources including semantics, depth, behaviors and the like are utilized, the spanning behaviors can be judged more comprehensively and accurately, meanwhile, the method can be achieved only through a monocular camera, the deployment and maintenance cost is lower, the method is more suitable for complex scenes such as shielding and angles, some tiny abnormal behaviors can be captured, and the method is suitable for popularization and application. All-weather automatic monitoring for efficiently and stably detecting the fence crossing behavior in a complex environment is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and security, and in particular to a fence crossing behavior detection method based on multimodal fusion. Background Art

[0002] Fence crossing behavior detection has always been an important technology for security management, border protection and asset protection. Traditional monitoring methods rely on manual surveillance, which involves deploying cameras to monitor fenced areas. However, this method not only relies on manual judgment, but also the monitoring range is limited by the camera angle and image quality, making it difficult to achieve comprehensive and effective monitoring.

[0003] In order to improve efficiency, the industry has introduced many solutions. For example, the prior art 202310633024.X discloses a method and system for detecting workers climbing over fences based on computer vision. Although it uses a segmentation algorithm and a 3D skeleton detection algorithm, it can extract the position and posture information of the human body from the video data to assist in monitoring fence crossing behavior. However, such algorithms are easily affected by factors such as occlusion and lighting in complex environments, and have poor detection accuracy and stability, making it difficult to meet the needs of practical applications. At the same time, these algorithms usually have high computational complexity, which is not conducive to deployment and application in real-time monitoring systems. Therefore, there is an urgent need for a new method that can efficiently and stably detect fence crossing behavior in complex environments. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a fence crossing behavior detection method based on multimodal fusion to solve the problem of poor detection accuracy and stability of the prior art in complex environments.

[0005] To achieve the above object, the technical solution adopted by the present invention is: a fence crossing behavior detection method based on multimodal fusion, comprising the following steps:

[0006] Through simulation experiments in advance, obtain a data set of people crossing fences, and manually annotate the data set;

[0007] Through the pre-trained semantic segmentation algorithm module, the area of ​​people and fences is identified from the data set of people crossing fences, and the intersection area of ​​people and fences is obtained to determine whether people and fences overlap;

[0008] Using the pre-trained monocular depth estimation algorithm module, the relative distance information between pedestrians and fences is obtained from the data set of people crossing fences to determine the distance between people and fences and avoid misjudgment.

[0009] The pre-trained abnormal behavior detection algorithm module has learned the characteristic distribution of normal behavior. When a piece of data is input, it compares the data with the characteristic distribution of "normal" behavior learned in advance, calculates the difference between the two, and if it exceeds a certain threshold, it determines that the behavior is "abnormal", that is, there may be a situation of crossing the fence;

[0010] The information fusion module is used to fuse and analyze the overlapping information between the person and the fence, the distance information between the person and the fence, and the judgment information on whether it is an "abnormal" behavior. If it is judged that the person and the fence overlap, the distance between the person and the fence is close and it is judged to be an "abnormal" behavior, it is determined that the person is crossing the fence and the detection result is output.

[0011] Preferably, during the simulation experiment, a distinction is made between "normal" pedestrian fence behavior and "abnormal" crossing behavior; wherein "normal" pedestrian fence behavior is divided into the following situations:

[0012] Passing the fence: People pass the fence along a predetermined path, keeping a certain distance;

[0013] Wandering near the fence: People wander near the fence, keeping a certain distance;

[0014] Activities within the fence: People are inside the fence and doing certain activities, such as working or resting;

[0015] Standing by the fence: A person stands by the fence or a person leans against the fence, such as talking to someone else;

[0016] Far from the fence: The person is far away from the fence, but it can be observed that the person and the fence overlap;

[0017] Except for "normal" fence-crossing behaviors, all others are considered "abnormal" fence-crossing behaviors.

[0018] Preferably, the semantic segmentation algorithm module is divided into two implementation modes, one is Unet semantic segmentation based on manual annotation, and the other is direct semantic segmentation based on SAM.

[0019] Preferably, the specific process of determining whether a person and a fence overlap is as follows:

[0020] The semantic segmentation algorithm module is used to obtain the areas of the person and the fence in the data set, thereby calculating the intersection area of ​​the two and then the overlap ratio. If the overlap ratio exceeds the preset threshold, it is determined that there is a significant overlap between the person and the fence.

[0021] Preferably, the calculation expression for determining the overlap ratio between a person and a fence is:

[0022] C=A∪B / Min(A,B)

[0023] In the formula, C represents the overlap ratio, A represents the area of ​​the person, B represents the area of ​​the fence, and Min represents the minimum value.

[0024] Preferably, the monocular depth estimation algorithm module performs depth analysis on the person and the fence based on the DPT algorithm, thereby obtaining distance information between the person and the fence.

[0025] Preferably, the abnormal behavior detection algorithm module detects abnormal behavior based on a Gaussian mixture model; the abnormal behavior is judged as follows: for a newly input data, its log-likelihood value under the algorithm is calculated, and if it is lower than a threshold, it is judged as abnormal.

[0026] Preferably, the information fusion module performs final output based on fusion rules.

[0027] Due to the application of the above technical solution, the present invention has the following advantages compared with the prior art:

[0028] 1. The present invention utilizes multiple information sources, including semantics, depth, behavior, etc., which can more comprehensively and accurately judge the crossing behavior, thereby realizing all-weather automatic monitoring of human fence crossing behavior;

[0029] 2. This method can be implemented with only a monocular camera, with lower deployment and maintenance costs, providing technical support for secure management;

[0030] 3. This method is more applicable to complex scenes such as occlusion and angle, and can capture some subtle abnormal behaviors;

[0031] In general, the method of the present invention fully utilizes the advantages of multi-source information, and has significant advantages in detection accuracy, robustness, application scenarios, etc., providing strong support for improving the safety of workers, students and other personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The technical solution of the present invention is further described below in conjunction with the accompanying drawings:

[0033] Attached Figure 1 This is a system block diagram of the fence crossing behavior detection method based on multimodal fusion described in the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0035] Attached Figure 1 The fence crossing behavior detection method based on multimodal fusion described in the present invention comprises the following steps:

[0036] S1. Obtain a dataset of people crossing fences through simulation experiments in advance, and manually annotate the dataset;

[0037] S2. Using the pre-trained semantic segmentation algorithm module, identify the area of ​​people and fences from the data set of people crossing fences, and obtain the intersection area of ​​people and fences to determine whether people and fences overlap;

[0038] S3. Using the pre-trained monocular depth estimation algorithm module, the relative distance information between pedestrians and fences is obtained from the data set of people crossing fences to determine the distance between people and fences and avoid misjudgment.

[0039] S4, a pre-trained abnormal behavior detection algorithm module, which has learned the characteristic distribution of normal behavior; when a piece of data is input, it compares the data with the characteristic distribution of "normal" behavior learned in advance, calculates the difference between the two, and if it exceeds a certain threshold, it determines that the behavior is "abnormal", that is, there may be a situation of crossing the fence;

[0040] S5. Use the information fusion module to fuse and analyze the above-mentioned overlapping information between the person and the fence, the distance information between the person and the fence, and the judgment information of whether it is an "abnormal" behavior. If it is judged that the person and the fence overlap, the distance between the person and the fence is close and it is judged to be an "abnormal" behavior, it is determined that the person is crossing the fence and the detection result is output.

[0041] Furthermore, in the above step S1, when performing the simulation experiment, a distinction is made between "normal" pedestrian fence behaviors and "abnormal" crossing behaviors; wherein "normal" pedestrian fence behaviors are divided into the following situations:

[0042] Passing the fence: People pass the fence along a predetermined path, keeping a certain distance;

[0043] Wandering near the fence: People wander near the fence, keeping a certain distance;

[0044] Activities within the fence: People are inside the fence and doing certain activities, such as working or resting;

[0045] Standing by the fence: A person stands by the fence or a person leans against the fence, such as talking to someone else;

[0046] Far from the fence: The person is far away from the fence, but it can be observed that the person and the fence overlap;

[0047] Except for "normal" fence crossing behaviors, all others are considered "abnormal" fence crossing behaviors;

[0048] Through simulation experiments, we obtained a dataset of a person crossing a fence. Then, based on the “normal” and “abnormal” classification rules, we manually added “normal” or “abnormal” labels to the images. In addition, we also manually annotated the people and fences in the images at the pixel level.

[0049] Furthermore, in the above step S2, the semantic segmentation algorithm module is divided into two implementation modes, one is Unet semantic segmentation based on manual annotation, and the other is direct semantic segmentation based on SAM;

[0050] The details of the two methods are as follows:

[0051] S21. The first implementation method - semantic segmentation based on Unet

[0052] Use supervised learning to train the Unet semantic segmentation model, try different Unet network structures, such as Unet, Unet++, Linknet, etc., and select the best model;

[0053] First, the original image and the annotated data are preprocessed by scale normalization, color enhancement, etc.

[0054] Then, using the Unet network structure, the preprocessed image is input, and image enhancement operations including rotation, scaling, flipping, etc. are performed on the image, and finally the corresponding semantic segmentation mask is output;

[0055] Use the cross entropy loss function to train the model and optimize the network parameters through back propagation;

[0056] The segmentation performance of the trained Unet model is evaluated on the validation set using IoU as the evaluation metric;

[0057] Based on the evaluation results, the model is further optimized and adjusted by adjusting hyperparameters, adding regularization, etc.

[0058] S22. Second implementation method - direct semantic segmentation based on SAM

[0059] Load the pre-trained Segment Anything Model (SAM) model without manual annotation. SAM is a general segmentation model that can handle various types of objects;

[0060] Input the image to be segmented into the SAM model to obtain the semantic segmentation mask of people and fences;

[0061] Post-process the segmentation mask output by the SAM model: filtering, morphological operations, etc. to improve the quality of the segmentation results;

[0062] According to application requirements, use IoU as an evaluation metric to evaluate the segmentation accuracy and performance indicators of the SAM model. Analyze the segmentation results and further optimize the post-processing steps according to requirements;

[0063] According to the output results of the two implementation methods S21 and S22, it is determined whether the person and the fence overlap. The specific process is as follows:

[0064] The semantic segmentation algorithm module is used to obtain the area of ​​the person and the fence in the data set, thereby calculating the intersection area of ​​the two and then the overlap ratio. If the overlap ratio exceeds the preset threshold, it is determined that the person and the fence have obvious overlap.

[0065] The calculation expression for determining the overlap ratio between a person and a fence is:

[0066] C=A∪B / Min(A,B)

[0067] In the formula, C represents the overlap ratio, A represents the area of ​​the person, B represents the area of ​​the fence, and Min represents the minimum value.

[0068] Furthermore, in the above step S3, the monocular depth estimation algorithm module performs a depth analysis on the person and the fence based on the DPT algorithm, thereby obtaining the distance information between the person and the fence; the specific steps are as follows:

[0069] Load the pre-trained DPT model, which already has powerful depth perception capabilities;

[0070] The preprocessed image is input into the DPT model, and the model outputs the depth information of each pixel in the image.

[0071] Furthermore, in the above step S4, the abnormal behavior detection algorithm module detects abnormal behavior based on a Gaussian mixture model; the specific steps are as follows:

[0072] Use the collected normal image feature data to train a Gaussian mixture model (GMM). GMM can learn the probability distribution of normal image features, which is represented as a combination of several Gaussian distributions.

[0073] The parameters of GMM, including mean, variance and mixing coefficient, are iteratively optimized through the EM algorithm;

[0074] Input the features of the new image into the trained GMM model and calculate the log-likelihood value of the feature under GMM. If the log-likelihood value of the new image is lower than the set threshold, it is judged as an abnormal image, otherwise, it is judged as a normal image.

[0075] For the detected abnormal images, we further analyze the differences between their features and those of normal images. By visualizing the GMM model parameters, we can understand the differences between abnormal images and normal images in the feature space. Based on the analysis results, we can further optimize the anomaly detection algorithm and improve the detection accuracy.

[0076] Furthermore, in the above step S5, the specific implementation steps of the information fusion module are as follows:

[0077] Define a series of fusion rules based on the input semantic segmentation results, depth information and abnormal behavior detection results;

[0078] If there is an overlapping area between the person and the fence AND the minimum distance between the person and the fence is less than the threshold AND the abnormal behavior detection is "abnormal", it is judged as "crossing the fence";

[0079] If there is no overlapping area between the person and the fence AND the minimum distance between the person and the fence is less than the threshold AND the abnormal behavior detection is "abnormal", it is judged as "crossing the fence";

[0080] If there is an overlapping area between the person and the fence AND the minimum distance between the person and the fence is greater than the threshold AND the abnormal behavior detection is "normal", it is determined as "normal behavior".

[0081] The above are only specific application examples of the present invention and do not constitute any limitation on the protection scope of the present invention. Any technical solution formed by equivalent transformation or equivalent replacement shall fall within the protection scope of the present invention.

Claims

1. A fence crossing behavior detection method based on multimodal fusion, characterized in that: The following steps are involved: Through simulation experiments in advance, obtain a data set of people crossing fences, and manually annotate the data set; Through the pre-trained semantic segmentation algorithm module, the area of ​​people and fences is identified from the data set of people crossing fences, and the intersection area of ​​people and fences is obtained to determine whether people and fences overlap; Using the pre-trained monocular depth estimation algorithm module, the relative distance information between pedestrians and fences is obtained from the data set of people crossing fences to determine the distance between people and fences and avoid misjudgment. The pre-trained abnormal behavior detection algorithm module has learned the characteristic distribution of normal behavior. When a piece of data is input, it compares the data with the characteristic distribution of "normal" behavior learned in advance, calculates the difference between the two, and if it exceeds a certain threshold, it determines that the behavior is "abnormal", that is, there may be a situation of crossing the fence; The information fusion module is used to fuse and analyze the above-mentioned overlapping information between the person and the fence, the distance information between the person and the fence, and the judgment information on whether it is an "abnormal" behavior. If it is judged that the person and the fence overlap, the distance between the person and the fence is close and it is judged to be an "abnormal" behavior, it is determined that the person is crossing the fence and the detection result is output.

2. The fence crossing behavior detection method based on multimodal fusion according to claim 1 is characterized by: During the simulation experiment, a distinction is made between "normal" pedestrian fence behavior and "abnormal" crossing behavior; "Normal" pedestrian fence behavior is divided into the following situations: Passing the fence: People pass the fence along a predetermined path, keeping a certain distance; Wandering near the fence: People wander near the fence, keeping a certain distance; Activities within the fence: People are inside the fence and doing certain activities, such as working or resting; Standing by the fence: A person stands by the fence or a person leans against the fence, such as talking to someone else; Far from the fence: The person is far away from the fence, but it can be observed that the person and the fence overlap; Except for "normal" fence-crossing behaviors, all others are considered "abnormal" fence-crossing behaviors.

3. The fence crossing behavior detection method based on multimodal fusion according to claim 1 is characterized in that: The semantic segmentation algorithm module is divided into two implementation modes, one is Unet semantic segmentation based on manual annotation, and the other is direct semantic segmentation based on SAM.

4. The fence crossing behavior detection method based on multimodal fusion according to claim 1 is characterized in that: The specific process of determining whether a person and a fence overlap is as follows: The semantic segmentation algorithm module is used to obtain the area of ​​people and fences in the data set, thereby calculating the intersection area of ​​the two and then calculating the overlap ratio; If the overlap ratio exceeds a preset threshold, it is determined that the person and the fence overlap significantly.

5. The fence crossing behavior detection method based on multimodal fusion according to claim 4 is characterized in that: The calculation expression for determining the overlap ratio between a person and a fence is: C=A∪B / Min(A,B) In the formula, C represents the overlap ratio, A represents the area of ​​the person, B represents the area of ​​the fence, and Min represents the minimum value.

6. The fence crossing behavior detection method based on multimodal fusion according to claim 1 is characterized in that: The monocular depth estimation algorithm module performs depth analysis on a person and a fence based on a DPT algorithm, thereby obtaining distance information between the person and the fence.

7. The fence crossing behavior detection method based on multimodal fusion according to claim 1 is characterized in that: The abnormal behavior detection algorithm module detects abnormal behavior based on a Gaussian mixture model.

8. The fence crossing behavior detection method based on multimodal fusion according to claim 1 is characterized in that: The information fusion module performs final output based on fusion rules.

Images