A method and device for detecting a fight behavior based on TDN and related components

Abstract

The application discloses a kind of based on TDN's fighting behavior detection method, device and related components, it is related to video monitoring technical field.The method comprises the following steps: collecting the video data of fighting and non-fighting in different time periods in multiple scenes, segmenting and frame extracting video data to obtain picture set, the obtained picture set is classified and labeled, and is divided into training set and verification set according to proportion;Data in training set is input into the TDN model after spatial information enhancement to train, obtain weight model and use the data in verification set to verify it;Real-time frame extraction is carried out on monitoring data by video stream to obtain detection picture, the obtained detection picture is inferred using the trained weight model, to judge whether there is fighting behavior, if so, then trigger alarm.The method deploys the trained model in cloud, realizes the real-time monitoring of multiple scenes, so as to automatically detect the occurrence of fighting event and trigger alarm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning and computer vision, and particularly relates to a TDN-based fighting behavior detection method and device and related components. BACKGROUND

[0002] In recent years, with the popularity of monitoring systems, monitoring cameras are almost everywhere in property scenes, which provides convenience for staff to monitor whether dangerous behaviors such as fighting occur, so that the staff can arrive at the scene in the first time to dissuade, and to a certain extent, guarantee the community residents' peace and social stability.

[0003] However, in the usual case, one or two staff need to observe dozens of camera pictures in real time to determine whether dangerous behaviors such as fighting occur. Due to limited human resources, it is difficult to realize 24-hour uninterrupted observation of multiple monitoring pictures by manpower, which will miss some fighting events in the community to a certain extent. SUMMARY

[0004] The present application provides a TDN-based fighting behavior detection method, device and related components, aiming to solve the problem that only manpower cannot observe all fighting events.

[0005] In a first aspect, the present application provides a TDN-based fighting behavior detection method, comprising:

[0006] Collecting video data of fighting and non-fighting in different time periods in multiple scenes, segmenting and frame extracting the video data to obtain a picture set, classifying and labeling the obtained picture set, and dividing it into a training set and a validation set according to a proportion;

[0007] Inputting the data in the training set into a TDN model with enhanced spatial information for training, obtaining a weight model and verifying it using the data in the validation set; wherein the spatial information enhancement process of the TDN model comprises: using a pre-trained VGG16 network to extract high-level spatial features and low-level spatial features from two continuous adjacent pictures, learning local spatial features by combining the low-level spatial features of the two adjacent pictures, learning global spatial features by combining the high-level spatial features of the two adjacent pictures, and combining the local spatial features and the global spatial features to obtain fusion features;

[0008] Real-time frame extracting of monitoring data by a video stream to obtain a detection picture, using the trained weight model to infer the obtained detection picture to determine whether there is a fighting behavior, and if so, triggering an alarm.

[0009] In a second aspect, an embodiment of the present application provides a TDN-based fighting behavior detection device, which comprises:

[0010] a collection unit configured to collect video data of fighting and non-fighting in different time periods in multiple scenes, segment and frame the video data to obtain a picture set, classify and label the obtained picture set, and divide the picture set into a training set and a verification set according to a proportion;

[0011] a training unit configured to input data in the training set into a TDN model with enhanced spatial information for training, obtain a weight model, and verify the weight model by using data in the verification set; wherein the process of enhancing spatial information of the TDN model comprises: extracting high-level spatial features and low-level spatial features from two continuous adjacent pictures by using a pre-trained VGG16 network, learning local spatial features by combining the low-level spatial features of the two adjacent pictures, learning global spatial features by combining the high-level spatial features of the two adjacent pictures, and obtaining fusion features by combining the local spatial features and the global spatial features;

[0012] an inference unit configured to obtain detection pictures by real-time frame extraction on monitoring data in a video stream, perform inference on the obtained detection pictures by using the trained weight model, judge whether there is a fighting behavior, and if so, trigger an alarm.

[0013] In a third aspect, an embodiment of the present application provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the TDN-based fighting behavior detection method according to the first aspect when executing the computer program.

[0014] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program causes a processor to execute the TDN-based fighting behavior detection method according to the first aspect when the computer program is executed by the processor.

[0015] The embodiment of the present application provides a fight behavior detection method and device based on TDN and related components. The method comprises collecting fight and non-fight video data in different time periods in multiple scenes, segmenting and frame extracting the video data to obtain a picture set, classifying and labeling the obtained picture set, and dividing the picture set into a training set and a verification set according to a proportion; inputting data in the training set into a TDN model with enhanced spatial information for training, obtaining a weight model, and verifying the weight model by using data in the verification set; obtaining detection pictures by real-time frame extracting of a video stream on monitoring data, using the trained weight model to infer the obtained detection pictures, judging whether a fight behavior exists, and if so, triggering an alarm. The method provides a fight detection model based on TDN and enhanced spatial information, to realize automatic detection of fight and triggering of an alarm, and timely reminding of staff. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 The flowchart of the fight behavior detection method based on TDN provided by the embodiment of the present application is shown in the figure.

[0018] Figure 2 The sub-flowchart of the fight behavior detection method based on TDN provided by the embodiment of the present application is shown in the figure.

[0019] Figure 3 Another sub-flowchart of the fight behavior detection method based on TDN provided by the embodiment of the present application is shown in the figure.

[0020] Figure 4 Another sub-flowchart of the fight behavior detection method based on TDN provided by the embodiment of the present application is shown in the figure.

[0021] Figure 5 Another sub-flowchart of the fight behavior detection method based on TDN provided by the embodiment of the present application is shown in the figure.

[0022] Figure 6 Another sub-flowchart of the fight behavior detection method based on TDN provided by the embodiment of the present application is shown in the figure.

[0023] Figure 7 Another sub-flowchart of the fight behavior detection method based on TDN provided by the embodiment of the present application is shown in the figure.

[0024] Figure 8 Another sub-process diagram of the TDN-based fighting behavior detection method provided by the embodiment of the present application is shown in FIG. 6;

[0025] Figure 9 A schematic block diagram of the TDN-based fighting behavior detection device provided by the embodiment of the present application is shown in FIG. 7;

[0026] Figure 10 A sub-unit schematic block diagram of the TDN-based fighting behavior detection device provided by the embodiment of the present application is shown in FIG. 8;

[0027] Figure 11 A sub-unit schematic block diagram of the TDN-based fighting behavior detection device provided by the embodiment of the present application is shown in FIG. 9;

[0028] Figure 12 A sub-unit schematic block diagram of the TDN-based fighting behavior detection device provided by the embodiment of the present application is shown in FIG. 10;

[0029] Figure 13 A sub-unit schematic block diagram of the TDN-based fighting behavior detection device provided by the embodiment of the present application is shown in FIG. 11. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0031] It should be understood that, when used in the specification and the appended claims, the terms “comprise” and “include” indicate the presence of described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0032] It should also be understood that the terms used in the present application specification are only for the purpose of describing particular embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, the singular forms “a”, “an” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0033] It should be further understood that the term “and / or” used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0034] Please refer to Figure 1 , Figure 1 A flowchart of a TDN-based fighting behavior detection method provided by an embodiment of the present application is shown in FIG. 1. The method can include steps S101-S103.

[0035] S101, collect video data of fighting and non-fighting in multiple scenes and different time periods, segment and frame the video data to obtain a picture set, classify and label the obtained picture set, and divide the picture set into a training set and a validation set according to a proportion;

[0036] In step S101, because the probability of fighting event occurrence is low, in order to collect fighting data in multiple scenes and different time periods, personnel need to be arranged to perform fighting drills under the camera. In order to maintain the diversity of data, the fighting actions need to be as different as possible, the number of fighters also needs to be different, and data of day and night also needs to be collected. In addition, a small amount of non-fighting video data needs to be added as negative samples for reference.

[0037] After the video data is collected, the video data needs to be segmented and framed to obtain a picture set. Specifically, as shown in FIG. 2, the step S101 includes steps S201-S203. Figure 2

[0038] S201, segment the video data by equal time interval to obtain multiple sub-videos, and frame each sub-video at a specified frame rate to obtain a picture set;

[0039] S202, label the data in the picture set to obtain a picture set containing fighting label and non-fighting label;

[0040] S203, divide the picture set according to a predetermined proportion to obtain a training set and a validation set.

[0041] In this embodiment, the video data is first segmented by equal time interval using a video segmentation tool to obtain sub-videos, and then each sub-video is framed at a specified frame rate to obtain a picture set corresponding to the sub-video.

[0042] It should be noted that the segmentation time can be a specified time set by the user for subsequent calculation. For example, it can be set to segment once every 5 seconds, or it can be set to segment once every 10 seconds. The specific setting time can be determined according to the specific situation, for example, according to the length of the video data, to improve the detection efficiency and reduce the calculation amount.

[0043] ​Then, the image sets of all sub-videos are labeled as follows: if the image sets of a sub-video all depict fighting scenes, then the image sets of that sub-video are labeled as fighting scenes; if the image sets of a sub-video all depict non-fighting everyday scenes, then the image sets of that sub-video are labeled as non-fighting scenes. All labeled image sets are then divided into training and validation sets according to a predetermined ratio for subsequent training and validation of the TDN model.

[0044] It should be noted that the predetermined ratio can be set according to specific circumstances. In this embodiment, the training set and the validation set are divided into a ratio of 80% and 20%, respectively. The training set has a larger proportion of data, so the final trained weight model is closer to the true value, and the results obtained when entering the validation process are more accurate.

[0045] S102. Input the data in the training set into the spatially enhanced TDN model for training to obtain the weight model and validate it using the data in the validation set.

[0046] In this embodiment, the TDN model is a Temporal Difference Network (TDN), which is a video-level architecture that explicitly computes short-term and long-term motion information by introducing a temporal difference module (TDM), thereby extracting multi-scale temporal information.

[0047] However, since the original TDN model does not extract enough spatial information, this embodiment introduces a spatial information enhancement module to enhance the spatial information extraction capability of the TDN model.

[0048] Among them, such as Figure 3 As shown, the spatial information enhancement process of the TDN model includes steps S301 to S303:

[0049] S301. Use a pre-trained VGG16 network to extract both low-level and high-level spatial features from two consecutive adjacent images.

[0050] S302. Learn local spatial features by combining the low-level spatial features of two adjacent images and learn global spatial features by combining the high-level spatial features of two adjacent images.

[0051] S303. Combining local spatial features and global spatial features, we obtain fused features.

[0052] This embodiment provides a further explanation of the process of extracting low-level and high-level spatial features.

[0053] In step S301, spatial features can be learned in two consecutive ith and (i+1)th pictures (i.e., input frame i and input frame i+1): low-level spatial features and high-level spatial features are extracted in both input frame i and input frame i+1 by using a pre-trained VGG16 network.

[0054] In step S302, after the spatial feature extraction is completed, the low-level feature map (i.e., low-level spatial feature F l (i)) of input frame i is combined with the low-level feature map (i.e., low-level spatial feature F l (i+1)) of input frame i+1 to learn local spatial features. Similarly, the high-level feature map (i.e., high-level spatial feature F h (i)) of input frame i is combined with the high-level feature map (i.e., high-level spatial feature F h (i+1)) of input frame i+1 to learn global spatial features.

[0055] In an embodiment, step S302 includes:

[0056] The low-level spatial features and the high-level spatial features of the two adjacent pictures are fused according to the following formula:

[0057] C l = F l (i) + F l (i+1)

[0058] C h = F h (i) + F h (i+1)

[0059] wherein C l represents the fused low-level spatial features, F l (i) represents the low-level spatial features of the ith picture, F l (i+1) represents the low-level spatial features of the (i+1)th picture, C h represents the fused high-level spatial features, F h (i) represents the high-level spatial features of the ith picture, and F h (i+1) represents the high-level spatial features of the (i+1)th picture.

[0060] wherein F l (i), F h (i), h and w represent height and width, and d represents channel depth. The subscripts l and h are used to distinguish the low-level spatial features and the high-level spatial features.

[0061] The "+" in the formula represents the fusion of feature maps, which can be fused in the manner of channel splicing.

[0062] In an embodiment, as shown in Figure 4 Step S302 further includes the following steps:

[0063] S401, learning the joint motion information of the low-level spatial features by using a first WDRB module, and performing convolution on the output of the first WDRB module by using a first convolution module to output local spatial features;

[0064] S402, learning the joint motion information of the high-level spatial features by using a second WDRB module, and performing convolution on the output of the second WDRB module by using a second convolution module, and then performing upsampling to output global spatial features.

[0065] Steps S401 and S402 are to learn the joint motion information of the fused low-level spatial features and the joint motion information of the high-level spatial features by using two WDRB modules respectively, and then perform convolution by using two convolution modules respectively, wherein step S401 directly outputs local spatial features, and step S402 still needs to perform upsampling, and finally outputs global spatial features.

[0066] The processing processes of the first WDRB module and the second WDRB module are the same, as shown in Figure 5 , which include the following steps:

[0067] S501, activating the input by using a ReLu activation function, performing batch normalization operation, performing first convolution operation, and performing dropout operation according to a probability to obtain a first output;

[0068] S502, activating the first output by using a ReLu activation function, performing batch normalization operation, performing second convolution operation, and performing dropout operation according to a probability to obtain a second output;

[0069] S503, activating the second output by using a ReLu activation function, performing batch normalization operation, and performing third convolution operation to obtain a final output.

[0070] Among them, Figure 5 BN is batch normalization operation, Conv is convolution operation, and Dropout is dropout operation.

[0071] In this embodiment, that is, before each convolution operation, the ReLu activation function is used for activation and batch normalization processing, and after the output obtained by the convolution operation, the dropout operation is performed according to a predetermined probability, for example, a dropout operation with a probability of 0.3 is added, and finally the output result is obtained.

[0072] In this embodiment, the WDRB module uses a residual layer module and increases the depth to improve the accuracy of the network. The WDRB module is based on a stack of three convolutional layers, of which the first two convolutional layers have a total of 512 filters with a size of 3x3, while the last convolutional layer in the WDRB module has a filter size of 1x1 with a total of 256 filters. In addition to widening the depth of the layer, the first two layers of the WDRB module are densely connected, because densely connected layers can better utilize all hierarchical features between layers, making full use of hierarchical features from two convolutional layers.

[0073] In step S303, the local spatial features and the global spatial features are finally combined to obtain the fused features, wherein the local spatial features and the global spatial features are fused in a channel splicing manner to obtain the fused features.

[0074] Finally, as shown in Figure 6 , the fused features output by the spatial information enhancement module (SFLM) are added to the output of the short-term network in the TDN model to obtain a result input into the long-term network to enhance the motion change information.

[0075] Figure 6 In this embodiment, the i-th picture and the i+1-th picture introduce the spatial information enhancement module, the i+1-th picture and the i+2-th picture also introduce the spatial information enhancement module, and so on, that is, every two adjacent pictures introduce the spatial information enhancement module, and the fused features finally output by the spatial information enhancement module are added to the output of the corresponding two adjacent pictures in the short-term network, and then the result is input into the long-term network.

[0076] In an embodiment, as shown in Figure 7 , the step S102 includes steps S701-S705:

[0077] S701, using a pre-trained VGG16 network to extract low-level spatial features and high-level spatial features from both of the two consecutive adjacent pictures;

[0078] S702, fusing the low-level spatial features of the two pictures to obtain fused low-level spatial features, and fusing the high-level spatial features of the two pictures to obtain fused high-level spatial features;

[0079] S703, inputting the fused low-level spatial features into a first WDRB module, and using a first convolutional module to convolve the output of the first WDRB module to output local spatial features;

[0080] S704, input the fused low-level spatial features to the second WDRB module, learn the joint motion information of the high-level spatial features by using the second WDRB module, perform convolution on the output of the second WDRB module by using the second convolution module, and then perform upsampling output to obtain global spatial features;

[0081] S705, fuse the local spatial features and the global spatial features by using a channel splicing manner to obtain fused features.

[0082] This embodiment is a complete description of the high-low level spatial feature fusion process. As for the processing process of the first WDRB module and the second WDRB module, reference can be made to the content of the previous embodiments, and the present embodiment will not be repeated.

[0083] The processing flowchart of the high-low level spatial feature fusion process is as shown in Figure 8 The specific implementation can refer to the content of the previous embodiments, and the present embodiment will not be repeated. Figure 8 In the formula, Conv represents convolution operation, Upsample represents upsampling operation, and C represents fusion.

[0084] After obtaining the TDN model with enhanced spatial information, the obtained training set data and the corresponding label file are input into the TDN model with introduced spatial information enhancement for training, so as to obtain a weight model. The weight model is preferably trained for 30 rounds. After each round of training is completed, the data of the verification set is input into the weight model obtained by the current training for verification, so as to view the performance of the model after each round of training. In the training process, the model is optimized by continuously reducing the value of the loss function. With the increase of the training rounds, the predicted value gradually approaches the true value of the label, and the weight model is more accurate.

[0085] Since the present embodiment solves the classification problem, that is, whether to detect a fighting behavior or a non-fighting behavior, the loss function is set as a cross-entropy loss function, and the specific expression is as follows:

[0086] L = - [ylogy^ + (1-y)log(1-y)]

[0087] In the formula, L represents the value of the loss function, y represents the predicted value, and y represents the true value.

[0088] S103, real-time frame extraction is performed on the monitoring data by using a video stream to obtain a detection picture, a trained weight model is used to infer the obtained detection picture, and it is judged whether there is a fighting behavior. If yes, an alarm is triggered.

[0089] In the embodiment, the trained weight model is mainly used. The monitoring data of multiple scenes in a cell under a video stream is frame extracted in real time, a specified number (such as 100 frames) of detection pictures are extracted as a sequence through a set frame rate (such as 20 fps), and the trained weight model is used to infer the sequence to obtain an inference result.

[0090] The inference result R is:

[0091] R fight ,R no_fight ={c,g}

[0092] Wherein, c represents a category (fight for fighting and no_fight for not fighting), and g represents a confidence degree of the prediction.

[0093] It is checked whether there is information of fighting in the inference result. If there is, an alarm is triggered, and the alarm information is pushed to on-site staff through a short message or other manners, so that the related personnel of fighting can be quickly dissuaded.

[0094] In an embodiment, the trained weight model can be deployed in the cloud to realize real-time monitoring of multiple scenes, so that the occurrence of fighting events is automatically detected and an alarm is triggered, some possible safety hazards in the property industry are excluded, and the ability of community management is improved.

[0095] The embodiment of the application also provides a fighting detection device based on TDN, which is used to execute any embodiment of the foregoing fighting detection method based on TDN. Specifically, please refer to Figure 9 , Figure 9 is a schematic block diagram of the fighting detection device based on TDN provided by the embodiment of the application. The fighting detection device 900 comprises a collection unit 910, a training unit 920 and an inference unit 930.

[0096] The collection unit 910 is used to collect video data of fighting and non-fighting in different time periods under multiple scenes, segment and frame extract the video data to obtain a picture set, classify and label the obtained picture set, and divide the picture set into a training set and a verification set in proportion.

[0097] The training unit 920 is configured to input data in the training set into the TDN model with enhanced spatial information for training, obtain a weight model, and verify the weight model by using data in the verification set. The process of enhancing spatial information of the TDN model includes: extracting high-level spatial features and low-level spatial features from each of two continuous adjacent pictures by using a pre-trained VGG16 network, learning local spatial features by combining the low-level spatial features of the two adjacent pictures, learning global spatial features by combining the high-level spatial features of the two adjacent pictures, and obtaining fused features by combining the local spatial features and the global spatial features.

[0098] The reasoning unit 930 is configured to obtain detection pictures by real-time frame extraction on monitoring data in a video stream, perform reasoning on the detection pictures by using the trained weight model, determine whether there is a fighting behavior, and trigger an alarm if there is a fighting behavior.

[0099] In an embodiment, the fighting detection apparatus 900 further includes a deployment unit configured to deploy the trained weight model on a cloud to realize real-time monitoring on multiple scenes.

[0100] In an embodiment, as shown in FIG. 10, the collection unit 910 includes: Figure 10

[0101] The segmentation unit 911 is configured to perform equal-time segmentation on the video data by using a video segmentation tool to obtain a plurality of sub-videos, and perform frame extraction on each of the sub-videos at a specified frame rate to obtain a picture set.

[0102] The labeling unit 912 is configured to label data in the picture set to obtain a picture set containing a fighting label and a non-fighting label.

[0103] The division unit 913 is configured to divide the picture set according to a predetermined proportion to obtain a training set and a verification set.

[0104] In an embodiment, the training unit 920 includes:

[0105] The high-low level fusion unit is configured to fuse the low-level spatial features and the high-level spatial features of the two adjacent pictures according to the following formula:

[0106] C l (i+1) l (i)+F l (i+1)

[0107] C h (i+1) h (i)+F h (i+1)

[0108] wherein "+" represents fusion of feature maps, C l ​F represents the low-level spatial features after fusion. l (i) represents the low-level spatial features of the i-th image, F l (i+1) represents the low-level spatial features of the (i+1)th image, C h F represents the high-level spatial features after fusion. h (i) represents the high-level spatial features of the i-th image, F h (i+1) represents the high-level spatial features of the (i+1)th image.

[0109] In one embodiment, such as Figure 11 As shown, training unit 920 includes:

[0110] The local feature learning unit 921 is used to learn the joint motion information of the low-level spatial features using the first WDRB module, and to convolve the output of the first WDRB module using the first convolution module to obtain the local spatial features.

[0111] The global feature learning unit 922 is used to learn the joint motion information of the high-level spatial features using the second WDRB module, and to convolve the output of the second WDRB module using the second convolution module, and then upsample the output to obtain the global spatial features.

[0112] In one embodiment, such as Figure 12 As shown, training unit 920 includes:

[0113] The spatial fusion unit 923 is used to fuse the local spatial features and global spatial features by channel splicing to obtain fused features;

[0114] Addition unit 924 is used to add the fusion feature to the short-term network output in the TDN model.

[0115] In one embodiment, such as Figure 13 As shown, both the first WDRB module and the second WDRB module include:

[0116] The first output unit 925 is used to activate the input with the ReLU activation function, perform batch normalization, perform the first convolution operation, and then perform a probability-based discard operation to obtain the first output.

[0117] The second output unit 926 is used to activate the first output using the ReLU activation function, perform batch normalization, perform a second convolution operation, and perform a probability-based discard operation to obtain the second output.

[0118] The third output unit 927 is configured to activate the second output by using a ReLu activation function, perform batch normalization operation, and perform third convolution operation to obtain a final output.

[0119] In another embodiment of the present application, a computer device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the TDN-based fighting behavior detection method when executing the computer program.

[0120] In another embodiment of the present application, a computer readable storage medium is provided. The computer readable storage medium can be a non-volatile computer readable storage medium. The computer readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the following steps: collecting video data of fighting and non-fighting in different time periods in multiple scenes, segmenting and frame extracting the video data to obtain a picture set, classifying and labeling the obtained picture set, and dividing the picture set into a training set and a validation set according to a proportion; inputting data in the training set into a TDN model with enhanced spatial information for training to obtain a weight model and verifying the weight model by using data in the validation set; frame extracting detection pictures from monitoring data in real time by using a video stream, inferring the obtained detection pictures by using the trained weight model, judging whether there is fighting behavior, and if so, triggering an alarm.

[0121] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses and units can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein. Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in general terms in the foregoing description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0122] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other manners. For example, the embodiments of the apparatus described above are merely schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, or a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electric, mechanical or in other forms.

[0123] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0124] In addition, each functional unit in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0125] The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a storage medium. Based on such understanding, the technical solutions of the present application essentially or substantially or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0126] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for detecting fighting behavior based on TDN, characterized in that, include: Collect video data of fighting and non-fighting at different times in multiple scenarios, segment and extract frames from the video data to obtain an image set, classify and label the obtained image set, and divide it into a training set and a validation set according to the ratio; The data in the training set is input into the spatially augmented TDN model for training to obtain a weighted model, which is then validated using the data in the validation set. The spatial information augmentation process of the TDN model includes: using a pre-trained VGG16 network to extract low-level and high-level spatial features from two consecutive adjacent images, learning local spatial features by combining the low-level spatial features of the two adjacent images, learning global spatial features by combining the high-level spatial features of the two adjacent images, and combining the local and global spatial features to obtain fused features. The system extracts frames from the video stream in real time to obtain detection images. A trained weight model is then used to infer from the obtained detection images to determine whether fighting or brawling occurs. If so, an alarm is triggered.

2. The method for detecting fighting behavior based on TDN according to claim 1, characterized in that, Also includes: The trained weight model is deployed in the cloud to enable real-time monitoring of multiple scenarios.

3. The method for detecting fighting behavior based on TDN according to claim 1, characterized in that, The process of segmenting and extracting frames from the video data to obtain an image set, classifying and labeling the resulting image set, and dividing it into a training set and a validation set according to a set ratio includes: The video data is divided into multiple sub-videos by using a video segmentation tool at equal time intervals, and each sub-video is then frame-stripped at a specified frame rate to obtain an image set. The data in the image set is labeled to obtain an image set containing and not containing the "fighting" or "brawling" tags; The image set is divided into a training set and a validation set according to a predetermined ratio.

4. The method for detecting fighting behavior based on TDN according to claim 1, characterized in that, The process of using a pre-trained VGG16 network to extract high-level and low-level spatial features from two consecutive adjacent images, learning local spatial features by combining the low-level spatial features of the two adjacent images, and learning global spatial features by combining the low-level spatial features of the two adjacent images includes: The low-level and high-level spatial features of two adjacent images are fused using the following formula: C l =F l (i)+F l (i+1) C h =F h (i)+F h (i+1) Where "+" indicates feature map fusion, C l F represents the low-level spatial features after fusion. l (i) represents the low-level spatial features of the i-th image, F l (i+1) represents the low-level spatial features of the (i+1)th image, C h F represents the high-level spatial features after fusion. h (i) represents the high-level spatial features of the i-th image, F h (i+1) represents the high-level spatial features of the (i+1)th image.

5. The method for detecting fighting behavior based on TDN according to claim 4, characterized in that, The method uses a pre-trained VGG16 network to extract high-level and low-level spatial features from two consecutive adjacent images. It then combines the low-level spatial features of the two adjacent images to learn local spatial features and combines the high-level spatial features of the two adjacent images to learn global spatial features. Features also include: The joint motion information of the low-level spatial features is learned using the first WDRB module, and the output of the first WDRB module is convolved using the first convolution module to obtain local spatial features. The joint motion information of the high-level spatial features is learned using the second WDRB module, and the output of the second WDRB module is convolved using the second convolution module, and then upsampled to obtain the global spatial features.

6. The method for detecting fighting behavior based on TDN according to claim 1, characterized in that, The fused features obtained by combining the local spatial features and the global spatial features include: The local spatial features and global spatial features are fused by splicing channels to obtain fused features; The fusion feature is added to the short-term network output in the TDN model.

7. The method for detecting fighting behavior based on TDN according to claim 5, characterized in that, The processing procedures of the first WDRB module and the second WDRB module include: The input is activated using the ReLU activation function, and then batch normalized. After the first convolution operation, the first output is obtained by discarding the input according to probability. The first output is activated using the ReLU activation function, followed by batch normalization, then a second convolution operation, and finally a probability-based discard operation to obtain the second output. The second output is activated using the ReLU activation function, followed by batch normalization, and then a third convolution operation is performed to obtain the final output.

8. A fighting behavior detection device based on TDN, characterized in that, include: The collection unit is used to collect video data of fighting and non-fighting at different times in multiple scenarios, segment and extract frames from the video data to obtain an image set, classify and label the obtained image set, and divide it into a training set and a validation set according to the ratio. The training unit is used to input the data in the training set into the spatially augmented TDN model for training, obtain a weighted model, and validate it using the data in the validation set. The spatial information augmentation process of the TDN model includes: using a pre-trained VGG16 network to extract high-level spatial features and low-level spatial features from two consecutive adjacent images, learning local spatial features by combining the low-level spatial features of the two adjacent images, learning global spatial features by combining the high-level spatial features of the two adjacent images, and combining the local spatial features and global spatial features to obtain fused features. The inference unit is used to extract frames from the monitoring data in real time through the video stream to obtain detection images. It then uses a trained weight model to infer from the obtained detection images to determine whether fighting or brawling occurs. If so, an alarm is triggered.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the fighting behavior detection method based on TDN as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the fighting behavior detection method based on TDN as described in any one of claims 1 to 7.

Images