A fighting video detection method based on a bidirectional point multiplication residual structure

Abstract

The application provides a fighting video detection method based on a bidirectional point multiplication residual error structure in the technical field of video detection, and comprises the following steps: step S10, a large number of monitoring videos are acquired, and each monitoring video is subjected to differential enhancement preprocessing; step S20, each monitoring video after preprocessing is cut and labeled, and a video dataset is constructed; step S30, a fighting video detection model is created, and the fighting video detection model is trained by using the video dataset; and step S40, a monitoring video to be detected is input into the trained fighting video detection model to automatically detect a fighting event. The application has the advantages that the accuracy and generalization performance of fighting event recognition are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of video detection technology, and in particular to a method for detecting fighting videos based on a bidirectional dot product residual structure. Background Technology

[0002] Fighting disrupts social stability, threatens the lives and property of the people, and brings extremely negative social consequences.

[0003] However, in actual video detection, the accuracy of identifying fighting events is not satisfactory, for the following reasons: 1. The video quality captured by the camera is relatively blurry; 2. Many non-fighting videos contain interactions between people, and the interaction process is similar to the actions of fighting, so they are misclassified as fighting; 3. The number of non-fighting videos is far greater than the number of fighting videos, resulting in a lack of training samples for the model, which in turn leads to weak generalization performance of the model used to detect fighting events.

[0004] Therefore, how to provide a fight video detection method based on a bidirectional dot product residual structure to improve the accuracy and generalization performance of fight event recognition has become an urgent technical problem to be solved. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a fight video detection method based on a bidirectional dot product residual structure, thereby improving the accuracy and generalization performance of fight event recognition.

[0006] This invention is implemented as follows: a fight video detection method based on a bidirectional dot product residual structure, comprising the following steps:

[0007] Step S10: Acquire a large number of surveillance videos and perform preprocessing such as difference enhancement on each surveillance video;

[0008] Step S20: Cut and label each of the preprocessed surveillance videos to construct a video dataset;

[0009] Step S30: Create a fight video detection model and train the fight video detection model using the video dataset;

[0010] Step S40: Input the surveillance video to be detected into the trained fight video detection model to automatically detect fight events.

[0011] Further, step S10 specifically includes:

[0012] Step S11: Obtain a large amount of surveillance video, including both fighting and non-fighting videos;

[0013] Step S12: Decompose each of the monitoring videos into several image frames, and calculate the RGB difference of the pixels corresponding to adjacent image frames respectively.

[0014] Step S13: Construct a signal enhancement function, and perform RGB value difference enhancement on each image frame based on the signal enhancement function and the RGB difference;

[0015] Step S14: Merge the image frames after difference enhancement back into the monitoring video in their original order to complete the preprocessing of the monitoring video.

[0016] Further, in step S13, the formula for the signal enhancement function is: ;

[0017] Where x represents the RGB difference; α represents infinity; and e represents the natural constant.

[0018] Further, step S20 specifically includes:

[0019] Step S21: Divide the preprocessed surveillance videos into fighting videos and non-fighting videos, and set the cutting thresholds a and b; where a and b are both positive integers, and a > b;

[0020] Step S22: Locate the time range of the fighting event in each of the fighting videos, and randomly cut each of the fighting videos a times with the time range as the center to obtain a fighting sub-videos. Assign a soft tag to each of the fighting sub-videos and a hard tag to each of the fighting videos.

[0021] Step S23: Randomly cut each of the non-fighting videos b times to obtain b non-fighting sub-videos, and assign a non-fighting tag of type soft tag to each of the non-fighting videos.

[0022] Step S24: Construct a video dataset based on the fighting videos, videos of fighting, and videos of non-fighting.

[0023] Furthermore, both the fighting label and the non-fighting label are numerical labels;

[0024] The value range of the fighting tag of type soft tag is (0.8, 1); the value of the fighting tag of type hard tag is 1; the value range of the non-fighting tag of type soft tag is (0, 0.3).

[0025] Further, step S30 specifically includes:

[0026] Create a fighting video detection model, divide the video dataset into a training set and a validation set according to a preset ratio, and set a convergence condition and a detection accuracy threshold.

[0027] The fighting video detection model is trained using the training set until the convergence condition is met;

[0028] The trained fight video detection model is validated using the validation set to determine whether the detection accuracy is greater than the detection accuracy threshold. If yes, the training of the fight video detection model is completed; otherwise, the video dataset is expanded and training continues.

[0029] Furthermore, step S30 also includes:

[0030] A loss reduction threshold is set. During the training of the fight video detection model, it is monitored whether the degree of loss reduction of the loss function of the fight video detection model is greater than the loss reduction threshold. If so, proceed to step S20 and re-segment the corresponding monitoring video; otherwise, continue training.

[0031] Furthermore, in step S30, the fighting video detection model includes a first convolutional module, a second convolutional module, a first SM-Gather module, a second SM-Gather module, and a third SM-Gather module;

[0032] The first convolutional module, the first SM-Gather module, the second SM-Gather module, the second convolutional module, and the third SM-Gather module are connected in sequence.

[0033] The first convolutional module includes a 3x3 convolutional layer; the second convolutional module includes a 1x1 convolutional layer.

[0034] Furthermore, the first SM-Gather module, the second SM-Gather module, and the third SM-Gather module are all used to perform the following steps:

[0035] Step a: Analyze the input features Figure 1 Features are obtained by performing a 3x3 convolution. Figure 2 Let the feature Figure 2 Shape, number of channels and features Figure 1 Maintain consistency;

[0036] The feature is obtained by using the sigmoid function. Figure 2 The RGB values ​​are mapped to the range of 0 to 1, serving as the feature. Figure 1 The first weight;

[0037] Combine the first weight with the feature Figure 1 Performing a dot product yields an output of 1;

[0038] Step b: Use the sigmoid function to convert the features... Figure 1 The RGB values ​​are mapped to the range of 0 to 1, serving as the feature. Figure 2 The second weight;

[0039] Combine the second weight with the feature Figure 2 Performing a dot product yields an output of 2;

[0040] Step c: Add the outputs 1 and 2 using tensors to obtain the residual value, and then concatenate the outputs 1 and 2 using channels to obtain the concatenated value.

[0041] Step d: Perform a 1*1 convolution on the spliced ​​values ​​to ensure that the channels of the spliced ​​values ​​are consistent with those of the residual results.

[0042] Step e: Convert the residual result value into a third weight using the sigmoid function, and multiply the third weight by the concatenated value after convolution to obtain the final output.

[0043] Furthermore, the loss function of the fight video detection model is:

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] in, This represents the total loss value of the loss function; Indicates soft label loss; Represents cross-entropy loss; This indicates the percentage loss due to video segmentation. Indicates a soft tag; This represents the predicted value for the soft label; Indicates the true label; The predicted value represents the true label; This represents the sigmoid function, which maps values ​​to the range of 0 to 1.

[0049] The advantages of this invention are:

[0050] By decomposing surveillance video into image frames and calculating the RGB differences of pixels in adjacent image frames, the RGB values ​​of each image frame are enhanced using a constructed signal enhancement function and the RGB differences. The enhanced image frames are then merged back into the surveillance video in their original order, thus enhancing the data diversity of the surveillance video to overcome the problem of blurry image quality. Furthermore, by constructing a fight video detection model including the SM-Gather module, the model depth is reduced while simultaneously addressing the problem of poor image frame (feature map) representation capabilities, thereby enhancing the network information capture capability of the fight video detection model to cope with blurry image quality. By segmenting the surveillance video to increase the sample size and constructing soft labels to weaken the strong contrast between positive and negative samples (fighting videos and non-fighting videos), the label differences between fighting videos and non-fighting videos are reduced. The soft labels are then added to the reconstruction loss function to ensure that the model parameters of the fight video detection model are updated in a direction with stronger generalization ability, ultimately greatly improving the accuracy and generalization performance of fight event recognition. Attached Figure Description

[0051] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0052] Figure 1 This is a flowchart of a fight video detection method based on a bidirectional dot product residual structure according to the present invention.

[0053] Figure 2 This is a flowchart illustrating a fight video detection method based on a bidirectional dot product residual structure according to the present invention.

[0054] Figure 3 This is an architecture diagram of the fight video detection model of the present invention.

[0055] Figure 4 This is an architecture diagram of the SM-Gather module of this invention.

[0056] Figure 5 This is a schematic diagram of the process for cutting and tagging surveillance videos according to the present invention.

[0057] Figure 6 This is a schematic diagram of the cutting of the fighting video of the present invention.

[0058] Figure 7 This is a schematic diagram of the cutting of non-fighting video according to the present invention. Detailed Implementation

[0059] The overall approach of the technical solution in this application is as follows: The surveillance video is enhanced by interpolation, and a fight video detection model including the SM-Gather module is constructed to enhance network information capture capabilities and address the issue of blurry image quality. The surveillance video is segmented to increase the sample size, and soft labels are constructed to weaken the strong opposition between positive and negative samples. The soft labels are then added to the reconstructed loss function (reconstructed binary classification loss function) to ensure that the model parameters of the fight video detection model are updated in a direction with stronger generalization ability, thereby improving the accuracy and generalization performance of fight event recognition.

[0060] Please refer to Figures 1 to 7 As shown, a preferred embodiment of the fight video detection method based on a bidirectional dot product residual structure of the present invention includes the following steps:

[0061] Step S10: Acquire a large number of surveillance videos and perform preprocessing such as difference enhancement on each surveillance video;

[0062] Step S20: Cut and label each of the preprocessed surveillance videos to construct a video dataset;

[0063] Step S30: Create a fighting video detection model based on a neural network, and train the fighting video detection model using the video dataset; the fighting video detection model is created using the conv-lstm framework.

[0064] Step S40: Input the surveillance video to be detected into the trained fight video detection model to automatically detect fight events.

[0065] Step S10 specifically includes:

[0066] Step S11: Obtain a large amount of surveillance video, including both fighting and non-fighting videos;

[0067] Step S12: Decompose each of the monitoring videos into several image frames, and calculate the RGB difference of the pixels corresponding to adjacent image frames respectively.

[0068] Step S13: Construct a signal enhancement function, and perform RGB value difference enhancement on each image frame based on the signal enhancement function and the RGB difference;

[0069] The differences between the image frames are clearly visible to the naked eye, but the distribution of the RGB differences is relatively flat and cannot reflect these differences, which will lead to poor recognition performance of the fight video detection model. Therefore, it is necessary to amplify the larger RGB differences and suppress the smaller RGB differences.

[0070] Step S14: Merge the image frames after difference enhancement back into the monitoring video in their original order to complete the preprocessing of the monitoring video.

[0071] In step S13, the formula for the signal enhancement function is: ;

[0072] Where x represents the RGB difference; α represents infinity; e represents the natural constant; that is, if the RGB difference is greater than 0.5, then it is updated to... If the RGB difference is less than 0, update it to 0; otherwise, maintain the RGB difference.

[0073] Step S20 specifically includes:

[0074] Step S21: Divide the preprocessed surveillance videos into fighting videos and non-fighting videos, and set the cutting thresholds a and b; where a and b are both positive integers, and a > b;

[0075] Step S22: Locate the time range of the fighting event in each of the fighting videos, and randomly cut each of the fighting videos a times with the time range as the center to obtain a fighting sub-videos. Assign a soft tag to each of the fighting sub-videos and a hard tag to each of the fighting videos.

[0076] That is, the fight video is re-cut each time, and then several fight sub-videos with different cut intervals are saved;

[0077] Step S23: Randomly cut each of the non-fighting videos b times to obtain b non-fighting sub-videos, and assign a non-fighting tag of type soft tag to each of the non-fighting videos.

[0078] Step S24: Construct a video dataset based on the fighting videos, videos of fighting, and videos of non-fighting.

[0079] Both the "fighting" and "non-fighting" tags are numerical tags;

[0080] The value range of the fighting tag of type soft tag is (0.8, 1); the value of the fighting tag of type hard tag is 1; the value range of the non-fighting tag of type soft tag is (0, 0.3).

[0081] Step S30 specifically involves:

[0082] Create a fighting video detection model, divide the video dataset into a training set and a validation set according to a preset ratio, and set a convergence condition and a detection accuracy threshold.

[0083] The fighting video detection model is trained using the training set until the convergence condition is met;

[0084] The trained fight video detection model is validated using the validation set to determine whether the detection accuracy is greater than the detection accuracy threshold. If yes, the training of the fight video detection model is completed; otherwise, the video dataset is expanded and training continues.

[0085] Step S30 further includes:

[0086] A loss reduction threshold is set. During the training of the fight video detection model, it is monitored whether the decrease in the loss value of the model's loss function exceeds the loss reduction threshold. If so, step S20 is initiated to re-segment the corresponding surveillance video; otherwise, training continues. This method increases the diversity of negative samples, thereby greatly improving the generalization performance of the fight video detection model.

[0087] In step S30, the fighting video detection model includes a first convolutional module, a second convolutional module, a first SM-Gather module, a second SM-Gather module, and a third SM-Gather module; the first SM-Gather module, the second SM-Gather module, and the third SM-Gather module are all Sigmoid MultiplicationGather modules.

[0088] The first convolutional module, the first SM-Gather module, the second SM-Gather module, the second convolutional module, and the third SM-Gather module are connected in sequence; the first convolutional module is used as the input to the fight video detection model, and the third SM-Gather module is used as the output of the fight video detection model.

[0089] The first convolutional module includes a 3x3 convolutional layer; the second convolutional module includes a 1x1 convolutional layer.

[0090] Since low-resolution images contain less information in their feature maps after deep convolution, the number of convolutional layers cannot be too large. Therefore, the fighting video detection model with the above structure is adopted.

[0091] Both ResNet's residual structure and DenseNet's channel concatenation structure are methods to integrate convolutions between the previous and next layers. However, they have different focuses: the former adds channel information, while the latter concatenates channels. The SM-Gather module integrates the characteristics of both.

[0092] The first SM-Gather module, the second SM-Gather module, and the third SM-Gather module are all used to perform the following steps:

[0093] Step a: Analyze the input features Figure 1 (Feature Maps) are processed using 3x3 convolutions to obtain features. Figure 2 Let the feature Figure 2 Shape, number of channels and features Figure 1 Maintain consistency;

[0094] The feature is obtained by using the sigmoid function. Figure 2 The RGB values ​​are mapped to the range of 0 to 1, serving as the feature. Figure 1 The first weight;

[0095] Combine the first weight with the feature Figure 1 Performing a dot product yields an output of 1;

[0096] Step b: Use the sigmoid function to convert the features... Figure 1 The RGB values ​​are mapped to the range of 0 to 1, serving as the feature. Figure 2 The second weight;

[0097] Combine the second weight with the feature Figure 2 Performing a dot product yields an output of 2;

[0098] Step c: Add the outputs 1 and 2 using tensors to obtain the residual value, and then concatenate the outputs 1 and 2 using channels to obtain the concatenated value.

[0099] Step d: Perform a 1*1 convolution on the spliced ​​values ​​to ensure that the channels of the spliced ​​values ​​are consistent with those of the residual results.

[0100] Step e: Convert the residual result value into a third weight using the sigmoid function, and multiply the third weight by the concatenated value after convolution to obtain the final output.

[0101] The loss function of the fight video detection model is:

[0102] ;

[0103] ;

[0104] ;

[0105] ;

[0106] in, This represents the total loss value of the loss function; The loss of soft tags is expressed as the ratio of absolute errors. This represents the cross-entropy loss, also known as the hard-label loss. This represents the video segmentation loss, which is the difference loss between the segmented sample and the original sample. Indicates a soft tag; This represents the predicted value for the soft label; Indicates the true label; The predicted value represents the true label; This represents the sigmoid function, which maps values ​​to the range of 0 to 1.

[0107] In summary, the advantages of this invention are as follows:

[0108] By decomposing surveillance video into image frames and calculating the RGB differences of pixels in adjacent image frames, the RGB values ​​of each image frame are enhanced using a constructed signal enhancement function and the RGB differences. The enhanced image frames are then merged back into the surveillance video in their original order, thus enhancing the data diversity of the surveillance video to overcome the problem of blurry image quality. Furthermore, by constructing a fight video detection model including the SM-Gather module, the model depth is reduced while simultaneously addressing the problem of poor image frame (feature map) representation capabilities, thereby enhancing the network information capture capability of the fight video detection model to cope with blurry image quality. By segmenting the surveillance video to increase the sample size and constructing soft labels to weaken the strong contrast between positive and negative samples (fighting videos and non-fighting videos), the label differences between fighting videos and non-fighting videos are reduced. The soft labels are then added to the reconstruction loss function to ensure that the model parameters of the fight video detection model are updated in a direction with stronger generalization ability, ultimately greatly improving the accuracy and generalization performance of fight event recognition.

[0109] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for detecting fight videos based on a bidirectional dot product residual structure, characterized in that: Includes the following steps: Step S10: Acquire a large number of surveillance videos and perform preprocessing such as difference enhancement on each surveillance video; Step S20: Cut and label each of the preprocessed surveillance videos to construct a video dataset; Step S30: Create a fight video detection model and train the fight video detection model using the video dataset; The fight video detection model includes a first convolutional module, a second convolutional module, a first SM-Gather module, a second SM-Gather module, and a third SM-Gather module; The first convolutional module, the first SM-Gather module, the second SM-Gather module, the second convolutional module, and the third SM-Gather module are connected in sequence. The first convolutional module includes a 3*3 convolutional layer; the second convolutional module includes a 1*1 convolutional layer. Step S40: Input the surveillance video to be detected into the trained fight video detection model to automatically detect fight events.

2. The method for detecting fight videos based on a bidirectional dot product residual structure as described in claim 1, characterized in that: Step S10 specifically includes: Step S11: Obtain a large amount of surveillance video, including both fighting and non-fighting videos; Step S12: Decompose each of the monitoring videos into several image frames, and calculate the RGB difference of the pixels corresponding to adjacent image frames respectively. Step S13: Construct a signal enhancement function, and perform RGB value difference enhancement on each image frame based on the signal enhancement function and the RGB difference; Step S14: Merge the image frames after difference enhancement back into the monitoring video in their original order to complete the preprocessing of the monitoring video.

3. The method for detecting fight videos based on a bidirectional dot product residual structure as described in claim 2, characterized in that: In step S13, the formula for the signal enhancement function is: ; Where x represents the RGB difference; α represents infinity; and e represents the natural constant.

4. The method for detecting fight videos based on a bidirectional dot product residual structure as described in claim 1, characterized in that: Step S20 specifically includes: Step S21: Divide the preprocessed surveillance videos into fighting videos and non-fighting videos, and set the cutting thresholds a and b; where a and b are both positive integers, and a > b; Step S22: Locate the time range of the fighting event in each of the fighting videos, and randomly cut each of the fighting videos a times with the time range as the center to obtain a fighting sub-videos. Assign a soft tag to each of the fighting sub-videos and a hard tag to each of the fighting videos. Step S23: Randomly cut each of the non-fighting videos b times to obtain b non-fighting sub-videos, and assign a non-fighting tag of type soft tag to each of the non-fighting videos. Step S24: Construct a video dataset based on the fighting videos, videos of fighting, and videos of non-fighting.

5. The method for detecting fight videos based on a bidirectional dot product residual structure as described in claim 4, characterized in that: Both the "fighting" and "non-fighting" tags are numerical tags; The value range for the soft tag type of fighting tag is (0.8, 1); The value of a fighting tag of type hard tag is 1; the value range of a non-fighting tag of type soft tag is (0, 0.3).

6. The method for detecting fight videos based on a bidirectional dot product residual structure as described in claim 1, characterized in that: Step S30 specifically involves: Create a fighting video detection model, divide the video dataset into a training set and a validation set according to a preset ratio, and set a convergence condition and a detection accuracy threshold. The fighting video detection model is trained using the training set until the convergence condition is met; The trained fight video detection model is validated using the validation set to determine whether the detection accuracy is greater than the detection accuracy threshold. If yes, the training of the fight video detection model is completed; otherwise, the video dataset is expanded and training continues.

7. The method for detecting fight videos based on a bidirectional dot product residual structure as described in claim 1, characterized in that: Step S30 further includes: A loss reduction threshold is set. During the training of the fight video detection model, it is monitored whether the degree of loss reduction of the loss function of the fight video detection model is greater than the loss reduction threshold. If so, proceed to step S20 and re-segment the corresponding monitoring video; otherwise, continue training.

8. The method for detecting fight videos based on a bidirectional dot product residual structure as described in claim 1, characterized in that: The first SM-Gather module, the second SM-Gather module, and the third SM-Gather module are all used to perform the following steps: Step a: Perform a 3*3 convolution on the input feature map 1 to obtain feature map 2, ensuring that the shape and number of channels of feature map 2 are consistent with feature map 1; The RGB values ​​of feature map 2 are mapped to the range of 0 to 1 using the sigmoid function, and used as the first weight of feature map 1. The first weight is multiplied by the feature map 1 to obtain output 1; Step b: Map the RGB values ​​of feature map 1 to the range of 0 to 1 using the sigmoid function, and use them as the second weight of feature map 2; The second weight is multiplied by feature map 2 to obtain output 2; Step c: Add the outputs 1 and 2 using tensors to obtain the residual value, and then concatenate the outputs 1 and 2 using channels to obtain the concatenated value. Step d: Perform a 1*1 convolution on the spliced ​​values ​​to ensure that the channels of the spliced ​​values ​​are consistent with those of the residual results. Step e: Convert the residual result value into a third weight using the sigmoid function, and multiply the third weight by the concatenated value after convolution to obtain the final output.

9. The method for detecting fight videos based on a bidirectional dot product residual structure as described in claim 1, characterized in that: The loss function of the fight video detection model is: ； ； ； ； in, This represents the total loss value of the loss function; Indicates soft label loss; Represents cross-entropy loss; This indicates the percentage loss due to video segmentation. Indicates a soft tag; This represents the predicted value for the soft label; Indicates the true label; The predicted value represents the true label; This represents the sigmoid function, which maps values ​​to the range of 0 to 1.

Images