Fighting detection method and system based on linkage of depth camera and PTZ dome camera

Abstract

The application discloses a fighting detection method and system based on linkage of a depth camera and a PTZ dome camera, and belongs to the technical field of city, community and public security supervision. The suspected fighting detection of the depth camera in a large scene guides real-time zoom tracking of the PTZ dome camera, and the rotation and zoom of the dome camera are adjusted. Human body segmentation based on depth data is performed, the segmentation result is synchronously mapped to a PTZ dome camera video stream through spatial relationship, and fighting data of the video stream is extracted after image segmentation of the PTZ dome camera video stream. A 3D convolution network is fused with a depth stream video stream, local information of the depth data and local information of the PTZ dome camera video stream are respectively sent into the 3D convolution network to extract features, and the fused features are used as a discrimination criterion for fighting recognition. The fighting detection method provided by the application supplements extraction of information features in a depth direction, and improves the overall fighting detection effect through fusion of video stream features.

Description

Technical Field

[0001] This invention relates to the fields of urban, community, and public security supervision technology, and more specifically to a method and system for detecting fights based on the linkage of a depth camera and a PTZ PTZ camera. Background Technology

[0002] Fighting is an illegal and disorderly act that negatively impacts social stability and people's lives. Currently, the primary method for handling fights is to notify the police after the fact, making it difficult to obtain immediate reports and hindering subsequent investigations. However, with the continuous development of science and technology, techniques for detecting fighting have gradually emerged.

[0003] Existing methods for fighting detection mainly include: 1) Depth camera analysis: This method can more accurately describe human actions and behaviors, and can also locate the coordinates of the human body in three-dimensional space. In addition, depth cameras can also filter out interference from changes in light and shadow. The disadvantage is that depth cameras with only point cloud information do not contain color and texture information. Also, because the three-dimensional imaging of depth cameras will have a certain attenuation for distant targets, it also reduces the accuracy of distant target detection. 2) Depth camera and surveillance camera linkage analysis: This method combines depth information and video image information, which to some extent solves the shortcomings of using a single depth camera for fighting detection or human behavior analysis. However, it does not offer a better solution for distant targets. When applied to fighting detection, it does not drive the suspicious areas detected by the depth camera to be magnified and focused in real time in the surveillance camera, making it impossible to perform more detailed detection.

[0004] Therefore, proposing a fighting detection method and system based on the linkage of a depth camera and a PTZ PTZ camera to further improve the fighting detection effect is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a fighting detection method and system based on the linkage of a depth camera and a PTZ PTZ camera, which improves the accuracy of fighting detection results.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] On the one hand, this invention proposes a method for detecting fighting based on the linkage of a depth camera and a PTZ PTZ camera, including the following steps:

[0008] Pre-detection of fighting behavior is performed based on the depth data stream acquired by the depth camera, and the depth stream fighting region is segmented.

[0009] The PTZ PTZ camera's rotation and scaling are guided in real time by the segmentation results of the deep flow fighting region.

[0010] The deep flow fighting region is mapped onto the rotated and scaled PTZ PTZ camera video stream, and the video stream is segmented into fighting region images to obtain the video stream fighting region.

[0011] The sequence of small images of the conflicting regions of the depth stream and the conflicting regions of the video stream are input into the depth stream and video stream fusion model to determine whether a conflicting event has occurred. If a conflicting event is detected, an alarm is triggered.

[0012] Preferably, the fighting behavior is pre-detected based on the depth data stream acquired by the depth camera, and the depth stream fighting region is segmented, including:

[0013] Obtain a two-dimensional projection image of the data stream;

[0014] The two-dimensional projection image is input into a pre-trained fighting behavior detection model, and the coordinates of the human bounding box of the person with fighting action are output after convolution calculation.

[0015] The average depth value within the body frame of each person engaging in fighting is calculated based on the body frame coordinates.

[0016] When the center points of the human frames of two or more persons engaged in fighting are less than the threshold T1 in horizontal distance and less than the threshold T2 in depth distance, a depth camera fighting pre-detection alarm is triggered.

[0017] The alarm area is defined as the smallest bounding rectangle of the human frames of multiple people engaged in fighting. The depth region of the alarm area is the depth flow fighting region.

[0018] Preferably, the PTZ PTZ camera's rotation and scaling are guided in real time by the segmentation results of the depth flow fighting region, including:

[0019] For camera rotation, the depth camera and the PTZ PTZ camera are calibrated in terms of device structure; the x and y coordinates of the center point of the alarm area on the projection plane of the depth camera are taken, and the center of the PTZ PTZ camera screen is rotated to point (x, y);

[0020] For camera scaling, the scaling ratio S of the PTZ PTZ camera is determined by pre-calibrating the distance value with the depth camera.

[0021] Preferably, the depth stream fighting region is mapped onto the rotated and scaled PTZ PTZ camera video stream, and the video stream is segmented into fighting region images to obtain the video stream fighting region, including:

[0022] For the pre-detection bounding box obtained from the segmentation of the deep flow fighting region (i.e., the border of the aforementioned deep flow fighting region), the width of the box is w. d The height is h d The width w of the corresponding video stream collision areav for:

[0023] w v =w d ×S;

[0024] The corresponding height h of the video stream collision area v for

[0025] h v =h d ×S;

[0026] The conflict area of ​​the video stream is a width of w v Height is h v The rectangle is a rectangle whose center is the center of the PTZ camera screen.

[0027] Preferably, the deep stream and video stream fusion model uses a video stream C3D branch network and a deep stream C3D branch network for fusion;

[0028] The video stream C3D branch network includes, in sequence, a convolutional layer Conv1a, a pooling layer Pool1, a convolutional layer Conv2a, a pooling layer Pool2, convolutional layers Conv3a and Conv3b, a pooling layer Pool3, convolutional layers Conv4a and Conv4b, a pooling layer Pool4, convolutional layers Conv5a and Conv5b, a pooling layer Pool5, two fully connected layers fc6 and fc7, and a softmax layer;

[0029] The deep flow C3D branch network includes, in sequence, a convolutional layer Conv1a-1, a pooling layer Pool1-1, a convolutional layer Conv2a-1, a pooling layer Pool2-1, a convolutional layer Conv3a-1, Conv3b-1 and Conv3c, a pooling layer Pool3-1, a convolutional layer Conv4a-1, Conv4b-1 and Conv4c, a pooling layer Pool4-1, a convolutional layer Conv5a-1, a pooling layer Pool5-1, and two fully connected layers fc8 and fc9.

[0030] Preferably, the pooling layer Pool2 is connected to the pooling layer Pool2-1; the pooling layer Pool3 is connected to the pooling layer Pool3-1; the pooling layer Pool4 is connected to the pooling layer Pool4-1; and the fully connected layer fc9 is connected to the fully connected layer fc7. The features extracted from the depth stream and the features extracted from the video stream are fused, and the fused features are input into the softmax layer for classification.

[0031] Preferably, the softmax layer outputs the probability value of a fighting action. When the probability value exceeds a preset threshold, it is determined that a fighting event has occurred, and an alarm is triggered.

[0032] On the other hand, the present invention also proposes a fighting detection system based on the linkage of a depth camera and a PTZ PTZ camera, for implementing the aforementioned fighting detection method based on the linkage of a depth camera and a PTZ PTZ camera, comprising:

[0033] The depth stream fighting pre-detection module is used to pre-detect fighting behavior based on the depth data stream acquired by the depth camera and segment the depth stream fighting region.

[0034] The depth flow real-time PTZ camera guidance module is used to guide the PTZ camera to rotate and zoom in real time based on the segmentation results of the depth flow fighting area.

[0035] The video stream conflict detection module is used to map the deep stream conflict region onto the rotated and scaled PTZ PTZ camera video stream, and to perform conflict region image segmentation on the video stream to obtain the video stream conflict region;

[0036] The 3D convolution module for fusing depth stream and video stream is used to input the sequence of small images of the conflicting regions of the depth stream and the conflicting regions of the video stream into the fusion model of depth stream and video stream to determine whether a conflicting event has occurred. If a conflict is detected, an alarm is triggered.

[0037] As can be seen from the above technical solutions, compared with the prior art, the fighting detection method and system based on the linkage of a depth camera and a PTZ PTZ camera disclosed in this invention has the following beneficial effects:

[0038] 1. Detecting suspected fights in large scenes using depth cameras and guiding real-time magnification and tracking by PTZ cameras facilitates further analysis and judgment of video stream details.

[0039] 2. By mapping the human body segmentation of depth data to the PTZ camera video stream, the segmentation accuracy and precision are superior to video human body segmentation, reducing interference from data in areas where there is no fighting. At the same time, the real-time guidance of the depth segmentation results for the PTZ camera zoom ensures that the PTZ camera's viewpoint and zoom always follow the fighters, reducing the impact of excessive human body displacement.

[0040] 3. The deep stream and video stream fusion model of the present invention, namely the trained deep stream and video stream fusion 3D convolutional network, supplements the extraction of depth direction information features. By fusing with video stream features, it improves the overall fight detection effect. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 This is a flowchart of a fighting detection method based on the linkage of a depth camera and a PTZ PTZ camera, as proposed in an embodiment of the present invention.

[0043] Figure 2 This is a diagram of the YOLOv4 network structure for pre-detection of fighting behavior provided in an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the 3D convolutional network structure for deep stream video stream fusion according to the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] On one hand, embodiments of the present invention disclose a method for detecting fighting based on the linkage of a depth camera and a PTZ PTZ camera, referring to... Figure 1 It includes the following steps:

[0047] 1. Pre-detect fighting behavior based on the depth data stream acquired by the depth camera, and segment the depth stream fighting region.

[0048] For depth flow detection of fighting, the conventional method is to perform pose determination or motion detection based on human skeleton detection using 3D point clouds. However, for distant targets, the accuracy of depth cameras decreases at long distances, which is not conducive to the detection of 3D human skeletons at a distance. Therefore, this paper adopts a pre-detection scheme for fighting behavior by combining 2D projection pose detection of depth cameras with the depth relationship of the corresponding targets. The process is as follows:

[0049] (1) For depth flow, obtain its two-dimensional projection image. The gray value of the image is the normalization of the distance value between the point (x,y) on the image and 0-255.

[0050] (2) Several fighting actions are defined, including arm swinging, kicking, and hugging. Data is collected to train the corresponding models. There are many methods for 2D image detection, such as SSD, YOLOv3, and YOLOv4. Here, we take the YOLOv4 target detection scheme as an example. The pre-trained model detects individuals with fighting behaviors on a 2D projected image. The YOLOv4 network structure is as follows: Figure 2 As shown, the two-dimensional projection image of the depth flow is fed into the YOLOv4 network, and the coordinates of the human bounding box of the person with fighting action are output after convolution calculation.

[0051] (3) For each person with a fighting action obtained in step (2), calculate the average depth D within the human frame based on the human frame coordinates calculated in step (2). ave Let I(i,j) be a two-dimensional projection image of the depth flow, where i is the abscissa of the image, j is the ordinate of the image, x1 is the abscissa of the top-left corner of the human body bounding box, y1 is the ordinate of the top-left corner of the human body bounding box, w is the width of the human body bounding box, h is the height of the human body bounding box, and D is the average depth value within the human body bounding box. ave The calculation method is as follows:

[0052]

[0053] (4) Find two or more human frames with fighting actions whose center points are less than the threshold T1 in horizontal distance and less than the threshold T2 in depth distance. Use this as the judgment standard for the depth camera fighting pre-detection alarm. The alarm position (i.e. the alarm area) is the smallest bounding rectangle of multiple human frames. The segmented depth area is the depth flow fighting area segmentation result.

[0054] 2. The PTZ PTZ camera is rotated and scaled in real time based on the segmentation results of the deep flow fighting area.

[0055] The aforementioned depth flow fight pre-detection generates suspected alarms and the spatial three-dimensional position of the suspected alarm area. When a fight occurs, the suspected alarm area detected by the depth flow changes in real time according to the change of the fight area. This step sends the spatial position of the suspected fight area located in real time by the depth camera to the PTZ camera for rotation and zoom control, which can ensure that the imaging position of the suspected area in the PTZ camera is always at the most appropriate size.

[0056] For suspected areas detected by the depth flow fight pre-detection module, the center point of the alarm is taken, that is, the center of the alarm area rectangle P(x,y,z), where x and y are the horizontal and vertical coordinates of the alarm center point on the projection plane of the depth camera, and z is the distance from the alarm center point to the depth camera.

[0057] Regarding camera rotation, since the depth camera and PTZ PTZ camera have been calibrated in the system structure, the imaging area of ​​the PTZ PTZ camera in the preset position is consistent with that of the depth camera. Therefore, for PTZ PTZ camera rotation control, it is only necessary to rotate the PTZ camera to rotate the center of the camera image to point (x,y).

[0058] For camera scaling, the scaling ratio of the PTZ camera needs to be pre-calibrated with the distance value of the depth camera. When a person stands at the closest point to the depth camera image (distance D1), the scaling factor for the PTZ preset position to maximize the human image is S1. When the person stands at the farthest point to the depth camera image (distance D2), the scaling factor for the PTZ preset position to maximize the human image is S2. The scaling ratio of the PTZ camera changes linearly with the camera distance, with the coefficient R being...

[0059]

[0060] If the point Q closest to the camera in the depth flow fighting region is m away from the camera, then the scaling factor S of the PTZ camera is:

[0061] S = (m - D1) × R.

[0062] 3. Map the depth stream collision region onto the rotated and scaled PTZ PTZ camera video stream, and perform collision region image segmentation on the video stream to obtain the video stream collision region, including:

[0063] For the pre-detection bounding box obtained from the segmentation of the deep flow fighting region (i.e., the border of the aforementioned deep flow fighting region), the width of the box is w. d The height is h d The width w of the corresponding video stream collision area v for:

[0064] w v =w d ×S;

[0065] The corresponding height h of the video stream collision area v for

[0066] h v =h d ×S;

[0067] The conflict area of ​​the video stream is a width of w v Height is h v The rectangle is centered on the PTZ camera screen.

[0068] 4. Input the sequence of small images of the conflicting regions of the depth stream and the video stream into the depth stream and video stream fusion model to determine whether a conflicting event has occurred. If a conflicting event is detected, an alarm is triggered.

[0069] This step involves designing a depth-stream and video-stream fusion 3D convolutional network. Local information from both the depth data and the PTZ camera video stream is fed into the 3D convolutional network to extract features. These fused features are then used as the criterion for fight detection. This depth-stream and video-stream fusion 3D convolutional network supplements the extraction of depth-direction information features, and by fusing them with video stream features, it improves the overall fight detection performance. The depth-stream and video-stream fusion C3D network structure is as follows: Figure 3 As shown:

[0070] The C3D network structure for fusing depth stream and video stream employs a fusion of a video stream C3D branch network and a depth stream C3D branch network.

[0071] The video stream C3D branch network consists of the following layers connected in sequence: Conv1a, Pool1, Conv2a, Pool2, Conv3a and Conv3b, Pool3, Conv4a and Conv4b, Pool4, Conv5a and Conv5b, Pool5, two fully connected layers fc6 and fc7, and a softmax layer.

[0072] The deepflow C3D branch network consists of the following convolutional layers in sequence: Conv1a-1, Pool1-1, Conv2a-1, Pool2-1, Conv3a-1, Conv3b-1 and Conv3c, Pool3-1, Conv4a-1, Conv4b-1 and Conv4c, Pool4-1, Conv5a-1, Pool5-1, and two fully connected layers fc8 and fc9.

[0073] Compared to the video stream C3D branch network, the depth stream C3D branch network structure adds a Conv3c convolutional layer for convolution operation in the third convolutional layer, adds a Conv4c convolutional layer for convolution operation in the fourth convolutional layer, and reduces one convolution operation in the fifth convolutional layer. This results in better extraction of depth stream detail features.

[0074] In the fusion of the two branch networks, the second pooling layer Pool2 of the video stream is connected to the second pooling layer Pool2-1 of the deep stream, the third pooling layer Pool3 of the video stream is connected to the third pooling layer Pool3-1 of the deep stream, the fourth pooling layer Pool4 of the video stream is connected to the fourth pooling layer Pool4-1 of the deep stream, and the fully connected layer fc7 of the video stream is connected to the fully connected layer fc9 of the deep stream. This is to fuse the features extracted by the deep stream and the features extracted by the video stream. Finally, the fused 4096-dimensional features are used for classification judgment through a softmax layer to calculate the probability value of fighting action. When the probability value exceeds a preset threshold, it is judged as fighting.

[0075] The workflow for fight detection using a C3D network that fuses depth stream and video stream is as follows:

[0076] (1) Collect samples of the conflict between depth stream and video stream in different scenarios to segment the depth scene and video scene, obtain a synchronized dataset of depth stream and video stream, and send the dataset into the C3D network for fusion of depth stream and video stream to train the fusion model of depth stream and video stream.

[0077] (2) The deep flow fighting region segmentation data obtained based on the deep flow fighting pre-detection and the video flow fighting region segmentation data obtained by the deep flow real-time guided PTZ camera are combined to establish a 5-second buffer for a total of 125 frames during real-time analysis, and the buffer is updated in real time.

[0078] (3) Extract 125 small images of fighting regions from the depth stream and 125 small images of fighting regions from the video stream in the buffer, respectively. In real time, send the buffered small image sequence into the fusion model of depth stream and video stream. The depth stream sequence and video stream sequence are convolutionally processed through two branch networks. The feature vectors are added in the 2nd, 3rd and 4th pooling layers and in the fully connected layer. Finally, the 4096-dimensional features obtained by fusing the depth stream and video stream through the C3D network are sent into the softmax layer for classification and judgment. The probability value of the fighting action is calculated. When the probability value exceeds the preset threshold, the system generates an alarm.

[0079] On the other hand, this invention also proposes a fighting detection system based on the linkage of a depth camera and a PTZ PTZ camera, used to implement the aforementioned fighting detection method based on the linkage of a depth camera and a PTZ PTZ camera, including:

[0080] The depth stream fighting pre-detection module is used to pre-detect fighting behavior based on the depth data stream acquired by the depth camera and segment the depth stream fighting region.

[0081] The deep flow real-time PTZ camera guidance module is used to guide the rotation and scaling of the PTZ camera in real time based on the segmentation results of the deep flow overlapping area;

[0082] The video stream conflict detection module is used to map the conflict region of the depth stream onto the rotated and scaled PTZ PTZ camera video stream, and to perform conflict region image segmentation on the video stream to obtain the video stream conflict region;

[0083] The 3D convolution module for fusing depth stream and video stream is used to input the sequence of small images of conflicting regions in the depth stream and video stream into the fusion model of depth stream and video stream to determine whether a conflict event has occurred. If a conflict is detected, an alarm is triggered.

[0084] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0085] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting fighting based on the linkage of a depth camera and a PTZ PTZ camera, characterized in that, Includes the following steps: Pre-detection of fighting behavior is performed based on the depth data stream acquired by the depth camera, and the depth stream fighting region is segmented. The PTZ PTZ camera's rotation and scaling are guided in real time by the segmentation results of the deep flow fighting region. The deep flow fighting region is mapped onto the rotated and scaled PTZ PTZ camera video stream, and the video stream is segmented into fighting region images to obtain the video stream fighting region. The sequence of small images of the conflicting regions of the deep stream and the conflicting regions of the video stream are input into the deep stream and video stream fusion model to determine whether a conflicting event has occurred. If a conflicting event is detected, an alarm is triggered. The deep stream and video stream fusion model uses a video stream C3D branch network and a deep stream C3D branch network for fusion. The video stream C3D branch network includes, in sequence, a convolutional layer Conv1a, a pooling layer Pool1, a convolutional layer Conv2a, a pooling layer Pool2, convolutional layers Conv3a and Conv3b, a pooling layer Pool3, convolutional layers Conv4a and Conv4b, a pooling layer Pool4, convolutional layers Conv5a and Conv5b, a pooling layer Pool5, two fully connected layers fc6 and fc7, and a softmax layer; The deepflow C3D branch network includes, in sequence, a convolutional layer Conv1a-1, a pooling layer Pool1-1, a convolutional layer Conv2a-1, a pooling layer Pool2-1, a convolutional layer Conv3a-1, Conv3b-1 and Conv3c, a pooling layer Pool3-1, a convolutional layer Conv4a-1, Conv4b-1 and Conv4c, a pooling layer Pool4-1, a convolutional layer Conv5a-1, a pooling layer Pool5-1, and two fully connected layers fc8 and fc9; Pooling layer Pool2 is connected to pooling layer Pool2-1; pooling layer Pool3 is connected to pooling layer Pool3-1; pooling layer Pool4 is connected to pooling layer Pool4-1; fully connected layer fc9 is connected to fully connected layer fc7. Features extracted from the depth stream and features extracted from the video stream are fused, and the fused features are input into the softmax layer for classification.

2. The method for detecting fighting based on the linkage of a depth camera and a PTZ PTZ camera according to claim 1, characterized in that, Based on the depth data stream acquired by the depth camera, pre-detection of fighting behavior is performed, and the depth stream fighting region is segmented, including: Obtain a two-dimensional projection image of the data stream; The two-dimensional projection image is input into a pre-trained fighting behavior detection model, and the coordinates of the human bounding box of the person with fighting action are output after convolution calculation. The average depth value within the body frame of each person engaging in fighting is calculated based on the body frame coordinates. When the center points of the human frames of two or more persons engaged in fighting are less than the threshold T1 in horizontal distance and less than the threshold T2 in depth distance, a depth camera fighting pre-detection alarm is triggered. The alarm area is defined as the smallest bounding rectangle of the human frames of multiple people engaged in fighting. The depth region of the alarm area is the depth flow fighting region.

3. The method for detecting fighting based on the linkage of a depth camera and a PTZ PTZ camera according to claim 2, characterized in that, The PTZ PTZ camera's rotation and scaling are guided in real time based on the segmentation results of the deep flow fighting region, including: For camera rotation, the depth camera and the PTZ PTZ camera are calibrated in terms of device structure; the x-coordinate and y-coordinate of the center point of the alarm area on the projection plane of the depth camera are taken, and the center of the PTZ PTZ camera screen is rotated to point (x, y). For camera scaling, the scaling ratio S of the PTZ PTZ camera is determined by pre-calibrating the distance value with the depth camera.

4. The method for detecting fighting based on the linkage of a depth camera and a PTZ PTZ camera according to claim 3, characterized in that, The depth flow fighting region is mapped onto the rotated and scaled PTZ PTZ camera video stream, and the video stream is segmented into fighting region images to obtain the video stream fighting region, including: The bounding box of the deep flow fighting region obtained by segmenting the deep flow fighting region has a width of... The height is The width of the corresponding video stream collision area. for: ； The height of the corresponding video stream collision area for ； The area of ​​contention in the video stream is [width value missing]. Height is The rectangle is a rectangle whose center is the center of the PTZ camera screen.

5. The method for detecting fighting based on the linkage of a depth camera and a PTZ PTZ camera according to claim 1, characterized in that, The softmax layer outputs the probability value of a fighting action. When the probability value exceeds a preset threshold, it is determined that a fighting event has occurred and an alarm is triggered.

6. A fighting detection system based on the linkage of a depth camera and a PTZ camera, used to implement the fighting detection method based on the linkage of a depth camera and a PTZ camera as described in any one of claims 1-5, characterized in that, include: The depth stream fighting pre-detection module is used to pre-detect fighting behavior based on the depth data stream acquired by the depth camera and segment the depth stream fighting region. The depth flow real-time PTZ camera guidance module is used to guide the PTZ camera to rotate and zoom in real time based on the segmentation results of the depth flow fighting area. The video stream conflict detection module is used to map the deep stream conflict region onto the rotated and scaled PTZ PTZ camera video stream, and to perform conflict region image segmentation on the video stream to obtain the video stream conflict region; The 3D convolution module for fusing depth stream and video stream is used to input the sequence of small images of the conflicting regions of the depth stream and the conflicting regions of the video stream into the fusion model of depth stream and video stream to determine whether a conflicting event has occurred. If a conflict is detected, an alarm is triggered.

Images