Fence crossing behavior detection method based on multi-modal fusion

Abstract

The invention discloses a fence crossing behavior detection method based on multi-modal fusion, and the method comprises the steps: obtaining a human fence crossing behavior monocular RGB image data set, and distinguishing a normal pedestrian fence behavior from an abnormal crossing behavior; a fence crossing behavior detection network is constructed and trained, and after a monocular RGB image is received, a semantic segmentation module is used for judging that a person and a fence are obviously overlapped; the monocular depth estimation module judges the distance between a person and a fence to avoid misjudgment; an abnormal behavior detection module is used for comparing a monocular RGB image with characteristic distribution of a normal behavior learned in advance, whether the behavior is abnormal or not is judged, and an information fusion module is used for carrying out fusion analysis on overlapping information of a person and a fence, distance information of the person and the fence and judgment information of whether the behavior is abnormal or not. According to the invention, a computer vision technology and a machine learning technology are adopted, and fusion analysis is carried out by adopting the overlapping information of the person and the fence, the distance information of the person and the fence and the judgment information of whether an abnormal behavior exists, so that all-weather automatic monitoring of the person fence crossing behavior is realized.

Description

Technical Field

[0001] The present invention belongs to the technologies of pedestrian detection and recognition and deep learning, and particularly relates to a method for detecting fence-crossing behavior based on multi-modal fusion. Background Art

[0002] Fence-crossing behavior detection has always been an important technology for security management, boundary protection, and asset protection. Traditional monitoring methods mostly rely on manual surveillance, and cameras are deployed to monitor the fence area. This method not only depends on manual judgment, but also the monitoring range is limited by the camera angle and image quality, making it difficult to achieve comprehensive and effective monitoring.

[0003] To improve efficiency, many technical solutions have been introduced in this field, such as infrared pair-beam solutions, microwave pair-beam solutions, electronic fences, power grids, etc. However, they all have many deficiencies: the protection level is relatively low, and it is easy to cross or avoid; it is not suitable for special weather, and is affected by temperature, light, weather, etc., with a high false alarm rate; the power consumption is very large; there is danger; it is impossible to determine whether the alarm is a false alarm, and personnel need to be dispatched to check on the spot. To improve the monitoring efficiency, there are also existing technologies that combine algorithms based on semantic segmentation and 3D skeleton recognition. These methods can extract the position and posture information of the human body from video data to assist in monitoring fence-crossing behavior. Compared with traditional monitoring solutions, such existing methods achieve all-weather automated monitoring, with a relatively low deployment cost and certain flexibility.

[0004] However, the above-mentioned existing technologies are easily affected by factors such as occlusion and light in complex environments, with poor detection accuracy and stability, making it difficult to meet the requirements of practical applications, and usually having a high computational complexity, which is not conducive to deployment and application in real-time monitoring systems.

[0005] Therefore, there is an urgent need for a new method that can efficiently and stably detect fence-crossing behavior in complex environments. Summary of the Invention

[0006] Object of the Invention: The object of the present invention is to solve the deficiencies existing in the prior art and provide a method for detecting fence-crossing behavior based on multi-modal fusion.

[0007] Technical Solution: A method for detecting fence-crossing behavior based on multi-modal fusion according to the present invention includes the following steps.

[0008] Step (1): Obtain monocular RGB images of human fence-crossing behavior, form a data set, and perform manual annotation on the data set to distinguish normal pedestrian fence behavior from abnormal crossing behavior.

[0009] Step (2): Construct and train a fence-crossing behavior detection network, which includes a semantic segmentation module, a monocular depth estimation module, an abnormal behavior detection module, and an information fusion module;

[0010] Step (2.1): After receiving a monocular RGB image, use the semantic segmentation module to identify the regions of people and fences, as well as the corresponding areas, from the dataset of people crossing fences, calculate the intersection area between people and fences, and then calculate the overlap ratio C. If the overlap ratio C exceeds the preset threshold, it is determined that there is an obvious overlap between people and fences;

[0011] Step (2.2): After receiving a monocular RGB image, the monocular depth estimation module, based on the ResNet50 model, obtains the relative distance information between pedestrians and fences from the dataset of people crossing fences, and judges the distance between people and fences through the relative distance information to avoid misjudgment;

[0012] Step (2.3): Use the abnormal behavior detection module based on the Gaussian mixture model to learn the feature distribution of normal behaviors; when inputting a monocular RGB image data, compare the monocular RGB image with the previously learned feature distribution of "normal" behaviors, calculate the difference degree between the two, and if it exceeds a certain threshold, it is determined that the behavior is an "abnormal" behavior, that is, there may be a situation of crossing the fence;

[0013] Step (2.4): Use the information fusion module to perform fusion analysis on the overlap information between people and fences, the distance information between people and fences, and the judgment information on whether it is an abnormal behavior. If it is judged that there is an overlap between people and fences, the distance between people and fences is relatively close and it is judged to be an abnormal behavior, then it is determined that the person is crossing the fence and the detection result is output.

[0014] Furthermore, when distinguishing normal pedestrian fence behaviors and abnormal crossing behaviors in step (1), the normal pedestrian fence behaviors are divided into the following five situations:

[0015] Situation 1: Passing by the fence: People pass by the fence along a predefined path, keeping a certain distance (usually about 1 - 2 meters here, neither too close nor too far);

[0016] Situation 2: Wandering near the fence: People wander near the fence, keeping a certain distance (usually between 0.5 - 2 meters here, a relatively safe distance);

[0017] Situation 3: Activity inside the fence: People are inside the fence doing some activities, such as working or resting (at this time, people are inside the fence and have basically no distance from the fence)

[0018] Situation 4: Standing beside the fence: A person stands beside or leans against the fence, such as talking to others (at this time, the distance between the person and the fence is greater than 2 - 3 meters, but still within the monitoring range);

[0019] Situation 5: Far from the fence: A person is far from the fence, but it can be observed that the person and the fence overlap;

[0020] Except for the above five normal fence behaviors, all others are regarded as abnormal crossing behaviors.

[0021] Furthermore, the calculation expression for determining the overlap ratio between the person and the fence in step (2.1):

[0022] C = A ∪ B / Min(A, B);

[0023] In the formula, C represents the overlap ratio, A represents the area of the pedestrian, B represents the area of the fence, and Min represents taking the minimum value;

[0024] The calculation methods for the above-mentioned pedestrian area A and fence area B are as follows:

[0025] After the images in the dataset are input into the semantic segmentation module, the corresponding segmentation map is first obtained. The segmentation map is a two-dimensional array, and each pixel in the segmentation map represents the category label of the pixel point. By traversing all the segmentation maps, the number of pixels of the pedestrian category 1 and the number of pixels of the fence category 2 are respectively counted, and finally the pedestrian area A and the fence area B are obtained.

[0026] Furthermore, the semantic segmentation module uses a Unet semantic segmentation module based on manual annotation or a direct semantic segmentation module based on SAM;

[0027] The Unet semantic segmentation module based on manual annotation uses a U-Net encoder-decoder structure. The encoder includes a series of convolutional and pooling layers for extracting multi-scale features, specifically including: 2 3x3 convolutional layers and a 2x2 max pooling layer; the decoder uses transposed convolution for upsampling and fusing the feature maps, specifically including: a 3x3 transposed convolutional layer and skip connections to fuse features of different scales;

[0028] The output layer of the Unet semantic segmentation module based on manual annotation uses a 1x1 convolution, and the number of output channels is 2, corresponding to the segmentation results of pedestrians and fences respectively; its parameter settings are: the initial learning rate is 0.001, dynamically adjusted; the batch size is 8, the number of iterations is 200, and the early stopping method is adopted. The optimizer is Adam (β1 = 0.9, β2 = 0.999);

[0029] The image segmentation model SAM of the SAM-based direct semantic segmentation module adds a semantic segmentation head with 3 3x3 convolutional layers and 1 1x1 convolutional output layer to its backbone network; its parameter settings are as follows: the optimizer is SGD, the initial learning rate is 0.01, and the momentum is 0.9; the loss function is weighted cross-entropy loss, and the weights are dynamically adjusted according to the proportion of pedestrian and fence pixels.

[0030] Furthermore, the feature extraction module of the monocular depth estimation module uses a pre-trained ResNet50 model to extract rich visual features from monocular RGB images. The depth estimation head of the monocular depth estimation module converts the output of the feature extraction network into a dense depth prediction map, which is implemented using convolutional layers and upsampling layers; the monocular depth estimation module performs depth analysis on the people and fences in the monocular RGB image based on the DPT algorithm to obtain the distance information between people and fences.

[0031] Furthermore, the anomaly detection algorithm module detects abnormal behaviors based on the Gaussian mixture model GMM. The specific method is as follows: use the data features of the normal behaviors labeled in step (1) to train the Gaussian mixture model and fit its probability distribution (log-likelihood probability), compare the log-likelihood probability with a preset anomaly threshold, and the data below the threshold is determined to be an abnormal behavior; finally, output the detected abnormal behaviors and their relevant information.

[0032] Furthermore, the information fusion module performs the final output based on the fusion rules, including the following situations:

[0033] Situation 1): If there is an overlapping area between the person and the fence, and the minimum distance between the person and the fence is less than the threshold and the abnormal behavior detection is "abnormal", it is determined that the fence has been crossed;

[0034] Situation 2): If there is no overlapping area between the person and the fence, and the minimum distance between the person and the fence is less than the threshold and the abnormal behavior detection is "abnormal", it is determined that the fence has been crossed;

[0035] Situation 3): If there is an overlapping area between the person and the fence, and the minimum distance between the person and the fence is greater than the threshold and the abnormal behavior detection is "normal", it is determined to be a normal behavior.

[0036] Beneficial effects: The present invention adopts computer vision technology and machine learning technology, and performs fusion analysis using the overlapping information between the person and the fence, the distance information between the person and the fence, and the judgment information on whether it is an "abnormal" behavior, so as to realize all-weather automatic monitoring of the behavior of people crossing the fence. It has better interpretability for the detection of people crossing the fence behavior and improves the detection effect. The invention only uses public cameras as the data source, reducing labor and financial costs, and providing technical support for safety management. Description of the Drawings

[0037] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0038] Figure 2 It is a schematic diagram of the semantic segmentation module in the embodiment;

[0039] Figure 3 It is a schematic diagram of the monocular depth estimation module in the embodiment;

[0040] Figure 4 It is a schematic diagram of the information fusion module in the embodiment. Specific implementation manner

[0041] The technical solution of the present invention will be described in detail below, but the protection scope of the present invention is not limited to the described embodiments.

[0042] As Figure 1 shown, a method for detecting fence-crossing behavior based on multi-modal fusion of the present invention includes the following steps

[0043] Step (1), obtain monocular RGB images of human fence-crossing behavior, form a data set, and perform manual annotation on the data set to distinguish normal pedestrian fence behavior and abnormal crossing behavior;

[0044] Step (2), construct and train a fence-crossing behavior detection network, and the fence-crossing behavior detection network includes a semantic segmentation module, a monocular depth estimation module, an abnormal behavior detection module, and an information fusion module;

[0045] Step (2.1), use the semantic segmentation module to identify the regions of people and fences, as well as the corresponding areas, from the human fence-crossing data set, calculate the intersection area between people and fences, and then calculate the overlap ratio C. If the overlap ratio C exceeds a preset threshold, it is determined that there is an obvious overlap between people and fences;

[0046] Step (2.2), the monocular depth estimation module is based on the ResNet50 model, obtains the relative distance information between pedestrians and fences from the human fence-crossing data set, and judges the distance between people and fences through the relative distance information to avoid misjudgment;

[0047] Step (2.3), use the abnormal behavior detection module based on the Gaussian mixture model to learn the feature distribution of normal behavior; when an input data is given, the data is compared with the previously learned feature distribution of "normal" behavior, and the difference degree between the two is calculated. If it exceeds a certain threshold, it is determined that the behavior is an "abnormal" behavior, that is, there may be a situation of crossing the fence;

[0048] Step (2.4): Use the information fusion module to perform fusion analysis on the overlapping information between the person and the fence, the distance information between the person and the fence, and the judgment information on whether it is an abnormal behavior. If it is determined that the person overlaps with the fence, the distance between the person and the fence is relatively close, and it is judged as an abnormal behavior, then it is determined that the person is crossing the fence, and the detection result is output.

[0049] In step (1) of this embodiment, when distinguishing between normal pedestrian fence behaviors and abnormal crossing behaviors, the normal pedestrian fence behaviors are divided into the following five situations:

[0050] Situation 1: Passing by the fence: The person passes by the fence along the established path, maintaining a certain distance (the distance here is usually about 1-2 meters, neither too close nor too far).

[0051] Situation 2: Wandering near the fence: The person wanders near the fence, maintaining a certain distance (the distance here is generally between 0.5-2 meters, a relatively safe distance).

[0052] Situation 3: Activity inside the fence: The person is located inside the fence to carry out certain activities, such as working or resting (at this time, the person is inside the fence and has basically no distance from the fence).

[0053] Situation 4: Standing beside the fence: The person stands beside the fence or leans against the fence, such as talking to others (at this time, the distance between the person and the fence is greater than 2-3 meters, but still within the monitoring range).

[0054] Situation 5: Far from the fence: The person is far from the fence, but it can be observed that the person and the fence overlap.

[0055] Except for the above five normal fence behaviors, other behaviors are regarded as abnormal crossing behaviors.

[0056] In step (2.1) of this embodiment, the calculation expression for judging the overlap ratio between the person and the fence is:

[0057] C = A ∪ B / Min(A, B);

[0058] In the formula, C represents the overlap ratio, A represents the pedestrian area, B represents the fence area, and Min represents taking the minimum value.

[0059] The calculation methods for the above-mentioned pedestrian area A and fence area B are as follows:

[0060] After the image in the dataset is input into the semantic segmentation module, the corresponding segmentation map is first obtained. The segmentation map is a two-dimensional array, and each pixel in the segmentation map represents the category label of the pixel point. Traverse all segmentation maps, respectively count the number of pixels of pedestrian category 1 and the number of pixels of fence category 2, and finally obtain the pedestrian area A and the fence area B.

[0061] Such as Figure 2As shown in the figure, the semantic segmentation module adopts a Unet semantic segmentation module based on manual annotation or a direct semantic segmentation module based on SAM. The Unet semantic segmentation module based on manual annotation adopts a U-Net encoder-decoder structure. The encoder includes a series of convolutional and pooling layers for extracting multi-scale features, specifically including: 2 3x3 convolutional layers and a 2x2 max pooling layer; the decoder uses transposed convolution for upsampling and fusing feature maps, specifically including: a 3x3 transposed convolutional layer and skip connections to fuse features of different scales. The output layer of the Unet semantic segmentation module based on manual annotation uses a 1x1 convolution, and the number of output channels is 2, corresponding to the segmentation results of pedestrians and fences respectively. The direct semantic segmentation module based on SAM is the image segmentation model SAM, and its backbone network adds a semantic segmentation head with 3 3x3 convolutional layers and 1 1x1 convolutional output layer.

[0062] As Figure 3 shown, the feature extraction module of the monocular depth estimation module uses a pre-trained ResNet50 model to extract rich visual features from the monocular RGB image. The depth estimation head of the monocular depth estimation module converts the output of the feature extraction network into a dense depth prediction map, which is implemented using convolutional layers and upsampling layers; the monocular depth estimation module performs depth analysis on the people and fences in the monocular RGB image based on the DPT algorithm, so as to obtain the distance information between people and fences.

[0063] The specific working process of the monocular depth estimation module in this embodiment is as follows: input a monocular RGB image, use ResNet50 to extract image features, input the features into the depth estimation head to generate a depth prediction map, calculate the loss function according to the ground truth depth map to optimize the model parameters, repeat the above steps until the model converges or reaches the expected accuracy, and apply the trained model to a new image to output the depth prediction result.

[0064] Furthermore, the anomaly detection algorithm module detects abnormal behaviors based on the Gaussian Mixture Model (GMM). The specific method is as follows: use the data features of the normal behaviors labeled in step (1) to train the Gaussian mixture model and fit its probability distribution (log-likelihood probability), compare the log-likelihood probability with a preset anomaly threshold, and the data below the threshold is determined to be an abnormal behavior; finally, output the detected abnormal behaviors and their relevant information.

[0065] The Gaussian mixture model (Gaussian Mixture Model, GMM) in this embodiment uses GMM to fit the probability distribution of "normal" behavior data. The GMM consists of multiple Gaussian distribution components and their weight coefficients. The specific method for anomaly evaluation is as follows: for new input data, calculate its log-likelihood probability under the trained GMM model, and the data with low probability is considered an abnormal behavior.

[0066] As Figure 4 shown, the information fusion module makes the final output based on the fusion rules, including the following situations:

[0067] Situation 1): If there is an overlapping area between the person and the fence, and the minimum distance between the person and the fence is less than the threshold and the abnormal behavior detection is abnormal, it is determined that the fence has been crossed;

[0068] Situation 2): If there is no overlapping area between the person and the fence, and the minimum distance between the person and the fence is less than the threshold and the abnormal behavior detection is abnormal, it is determined that the fence has been crossed;

[0069] Situation 3): If there is an overlapping area between the person and the fence, and the minimum distance between the person and the fence is greater than the threshold and the abnormal behavior detection is normal, it is determined that the behavior is normal.

[0070] The present invention determines whether a person in the video overlaps with the fence through the semantic segmentation module, thereby determining whether the person touches the fence. Then, the monocular depth estimation module is used to obtain the distance information between the person and the fence, and the abnormal behavior detection module is combined to analyze whether the current person's behavior deviates from the normal mode. Finally, based on the above detection results, it is comprehensively determined whether the person is crossing the fence.

[0071] Compared with the prior art, this method utilizes multiple information sources, including semantics, depth, behavior, etc., and can judge the crossing behavior more comprehensively and accurately. At the same time, this method can be realized only by relying on a monocular camera, and the deployment and maintenance costs are lower. In addition, this method is more applicable to complex scenarios such as occlusion and angle, and can capture some subtle abnormal behaviors. Generally speaking, the method of the present invention makes full use of the advantages of multi-source information and has significant advantages in terms of detection accuracy, robustness, application scenarios, etc., and is expected to provide more effective support for improving the safety of workers, students and other personnel.

Claims

1. A fence crossing behavior detection method based on multimodal fusion, characterized in that: The following steps are included: Step (1), obtaining monocular RGB images of people crossing fences to form a data set, manually annotating the data set to distinguish normal pedestrian fence behaviors from abnormal crossing behaviors; Step (2), constructing and training a fence crossing behavior detection network, the fence crossing behavior detection network including a semantic segmentation module, a monocular depth estimation module, an abnormal behavior detection module and an information fusion module; Step (2.1), after receiving the monocular RGB image, use the semantic segmentation module to identify the area of ​​the person and the fence from the data set of people crossing the fence, as well as the corresponding area, calculate the intersection area of ​​the person and the fence, and then calculate the overlap ratio C. If the overlap ratio C exceeds the preset threshold, it is determined that the person and the fence overlap significantly; Step (2.2), after receiving the monocular RGB image, the monocular depth estimation module obtains the relative distance information between pedestrians and fences from the data set of people crossing fences based on the ResNet50 model, and judges the distance between people and fences based on the relative distance information to avoid misjudgment; Step (2.3), using the abnormal behavior detection module based on the Gaussian mixture model, learn the characteristic distribution of normal behavior; when a monocular RGB image is input, the monocular RGB image is compared with the characteristic distribution of "normal" behavior learned in advance, and the difference between the two is calculated. If it exceeds a certain threshold, the behavior is determined to be "abnormal", that is, there may be a situation of crossing the fence; Step (2.4): Use the information fusion module to fuse and analyze the overlapping information between the person and the fence, the distance information between the person and the fence, and the judgment information of whether it is abnormal behavior. If it is judged that the person and the fence overlap, the distance between the person and the fence is close and it is judged to be abnormal behavior, it is determined that the person is crossing the fence and the detection result is output.

2. The fence crossing behavior detection method based on multimodal fusion according to claim 1 is characterized in that: When the step (1) distinguishes between normal pedestrian fence behavior and abnormal crossing behavior, normal pedestrian fence behavior is divided into the following five situations: Situation 1: Passing by the fence: People pass by the fence along a predetermined path, keeping a certain distance; Situation 2: Wandering near the fence: The person wanders near the fence, keeping a certain distance; Situation 3: Activities within the fence: People are inside the fence and doing certain activities, such as working or resting; Situation 4: Standing by the fence: A person stands by the fence or leans against the fence, such as talking to someone else; Case 5: Far from the fence: The person is far away from the fence, but it can be observed that the person and the fence overlap; Except for the five normal fence behaviors mentioned above, all others are considered abnormal crossing behaviors.

3. The fence crossing behavior detection method based on multimodal fusion according to claim 1 is characterized in that: The calculation expression for determining the overlap ratio C between a person and a fence in step (2.1) is: C = A∪B / Min(A,B); In the formula, C represents the overlap ratio, A represents the pedestrian area, B represents the fence area, and Min represents the minimum value; The calculation method of the above pedestrian area A and fence area B is: After the images in the dataset are input into the semantic segmentation module, the corresponding segmentation map is first obtained. The segmentation map is a two-dimensional array. Each pixel in the segmentation map represents the category label of the pixel point. All segmentation maps are traversed, and the number of pixels of pedestrian category 1 and the number of pixels of fence category 2 are counted respectively, and finally the pedestrian area A and fence area B are obtained.

4. The fence crossing behavior detection method based on multimodal fusion according to claim 1 or 3 is characterized in that: The semantic segmentation module adopts a Unet semantic segmentation module based on manual annotation or a direct semantic segmentation module based on SAM; The Unet semantic segmentation module based on manual annotation adopts a U-Net encoder-decoder structure, and the encoder includes multiple convolutional layers and pooling layers; The decoder uses transposed convolution to upsample and fuse the feature maps; the output layer uses 1x1 convolution with 2 channels, corresponding to the segmentation results of pedestrians and fences respectively; The SAM-based direct semantic segmentation module image segmentation model SAM has a backbone network that adds three 3x3 convolutional layers and one 1x1 convolutional output layer to the semantic segmentation head.

5. The fence crossing behavior detection method based on multimodal fusion according to claim 1 is characterized in that: The feature extraction module of the monocular depth estimation module uses a pre-trained ResNet50 model to extract rich visual features from the monocular RGB image. The depth estimation head of the monocular depth estimation module converts the output of the feature extraction network into a dense depth prediction map, which is implemented using convolutional layers and upsampling layers. The monocular depth estimation module performs depth analysis on people and fences in the monocular RGB image based on the DPT algorithm to obtain the distance information between people and fences.

6. The fence crossing behavior detection method based on multimodal fusion according to claim 1 is characterized in that: The anomaly detection algorithm module detects abnormal behavior based on the Gaussian mixture model GMM. The specific method is: use the data features of the normal behavior marked in step (1) to train the Gaussian mixture model, and fit the log-likelihood probability, compare the log-likelihood probability with the preset anomaly threshold, and judge the data below the preset anomaly threshold as abnormal behavior; finally output the detected abnormal behavior and related information.

7. The fence crossing behavior detection method based on multimodal fusion according to claim 1 is characterized in that: The information fusion module performs the final output based on the fusion rules, including the following situations: Case 1): If there is an overlapping area between the person and the fence, and the minimum distance between the person and the fence is less than the threshold and the abnormal behavior detection is abnormal, it is determined to be crossing the fence; Case 2): If there is no overlapping area between the person and the fence, and the minimum distance between the person and the fence is less than the threshold and the abnormal behavior detection is abnormal, it is determined to be crossing the fence; Case 3) If there is an overlapping area between the person and the fence, and the minimum distance between the person and the fence is greater than the threshold and the abnormal behavior detection is normal, it is determined to be normal behavior.

Images