A method for detecting a rule violation video based on skeleton behavior recognition

Abstract

The application relates to a rule violation video detection method and system based on skeleton behavior recognition, which comprises a video acquisition module, a key frame extraction module, an image data set establishment module and a real-time video behavior detection module.The posture estimation algorithm is used in the video key frame extraction module to estimate the joint position, the rule violation behavior detection module adopts a multi-filter dynamic graph convolutional neural network model, the real-time video behavior detection module adopts a multi-filter dynamic graph convolutional neural network, and the connection relationship between different behavior nodes is adapted through a dynamic skeleton graph.The Inception structure and the dynamic skeleton graph method based on the graph convolutional neural network can realize real-time and accurate detection and identification of rule violation sensitive action behaviors in internet live broadcast, thereby providing an important guarantee for standardizing the anchor live broadcast behavior and providing a civilized internet live broadcast environment.

Description

Technical Field

[0001] This invention relates to image processing and detection, and more particularly to a method for detecting violations in videos based on skeleton behavior recognition. Background Technology

[0002] In recent years, behavior recognition has received increasing attention. It is not only crucial for understanding video content but also has wide applications in human-computer interaction and video surveillance. Due to the widespread use of online streaming media, the analysis of illegal, irregular, and unethical behaviors in online video streams has become urgent. Behavior recognition plays a vital role in detecting such scenes in videos. With the development of depth sensors and pose estimation algorithms, extracting human skeletal features from videos has become feasible and easy. The structure of joints in the human skeleton conveys important motion information and is more robust than RGB video formats; therefore, action recognition of skeleton sequences is of great significance.

[0003] Currently, the detection of violations in internet live streaming generally relies on manual screening or simple image recognition methods for overall image identification. However, these methods have the following problems in practical applications:

[0004] 1. Manual review involves a huge workload, making it difficult to guarantee effective review of every video;

[0005] 2. Manual review is highly subjective and makes it difficult to guarantee quality;

[0006] 3. Existing image recognition and classification methods are not accurate enough in identifying illegal, irregular, and unethical behaviors. Summary of the Invention

[0007] This invention provides a method and system for detecting violations in live-streamed videos based on skeleton behavior recognition. Utilizing deep neural network processing technology, the method first connects static human structures in the video to create dynamic connections, adding connections between previously unconnected nodes through learning. Secondly, it processes filters in the graph convolution using an Inception structure to enhance the influence information of each joint source point. Researching violations in live-streamed internet broadcasts using skeleton recognition methods has significant research value and importance. The method of this invention can detect and identify sensitive body movements in live-streamed internet videos in real time and accurately.

[0008] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0009] A method for detecting violations in videos based on skeleton behavior recognition includes the following steps:

[0010] Video capture steps: Obtain video stream data from internet live streaming platforms;

[0011] Keyframe extraction steps: Sparse sampling is used to obtain single-frame images from the video stream data; a pose estimation algorithm is used to estimate the joint positions of the single-frame images to obtain human multi-joint features; points with average joint confidence scores higher than a threshold are selected from the single-frame images as joint features to obtain keyframe images.

[0012] Steps to create an image dataset: Classify and label keyframe images as images of violations or non-violations, and remix the samples to create a sample image dataset;

[0013] The steps for establishing a graph convolutional network model for violation detection are as follows: Establish a multi-filter dynamic graph convolutional neural network model; repeatedly train the network using multiple sample image datasets in batches; iteratively optimize the model and set a violation detection probability threshold until an optimized graph convolutional network model based on human skeleton recognition for violation detection is obtained; this model is used to determine whether multiple consecutive frames of dynamic images contain violation actions.

[0014] Real-time video behavior detection steps: Real-time acquisition of video stream and extraction of keyframe images, input into an optimized convolutional network model for violation detection based on human skeleton recognition, and obtain violation judgment results.

[0015] The human joint features include the pixel positions of 15 joint point vectors and their confidence scores (x, y, acc).

[0016] The sample image dataset is divided into a training set, a test set, and a validation set.

[0017] The violation detection graph convolutional network model based on human skeleton recognition includes: a batch normalization (BN) layer, nine spatiotemporal residual blocks, a global average pooling layer, a fully connected layer, and a softmax classifier, all connected in sequence. The BN layer is used to normalize the input data. Each spatiotemporal residual block consists of a multi-filter dynamic graph convolutional network, a temporal convolutional network, and a residual structure. The global average pooling layer maps features from different samples to the same size, reducing the number of parameters while preserving useful features and improving the model's generalization ability. The fully connected layer maps the learned feature representations of human skeleton joints to the sample label space, extracting useful features. The softmax classifier is used for action classification and prediction.

[0018] The temporal convolutional network has a time window of 9, and the residual structure uses a dropout parameter with a probability of 0.5.

[0019] The multi-filtered graph dynamic convolutional structure includes: extending the Inception structure to a graph convolutional network, using 1-neighborhood and 2-neighborhood filters to extract features from the skeleton graph, and the nodes in each neighborhood follow a partitioning strategy: the convolutional kernels are divided into three categories, namely root node, centripetal set and eccentric set, for processing the input sample image, and finally the feature maps are connected together to form an adjacent pixel matrix for filtering joint positions.

[0020] During training, the training set data is input into the model in batches to obtain the dynamic human skeleton joint connections. The dynamic human skeleton joint connections and their corresponding behaviors are learned through a multi-filtered graph dynamic convolutional structure. A softmax classifier is used to give the probability of whether the current batch of input multi-frame image data is in violation.

[0021] The interval between the sparsely sampled video frames is 5 seconds.

[0022] A violation video detection system based on skeleton behavior recognition is used to identify violations in video stream data from internet live streaming platforms. The system includes the following program modules:

[0023] Video capture module: Acquires video stream data from internet live streaming platforms;

[0024] Keyframe extraction module: Sparse sampling is used to obtain single-frame images from video stream data; a pose estimation algorithm is used to estimate joint positions in the single-frame images to obtain human multi-joint features; points with average joint confidence scores higher than a threshold are selected as features in the single-frame images to obtain keyframe images.

[0025] The image dataset creation module classifies and labels keyframe images as either violation-related or non-violation-related action images, and then remixes the samples to create a sample image dataset.

[0026] A program module for establishing a graph convolutional network model for violation detection is provided: a multi-filter dynamic graph convolutional neural network model is established, and the network is repeatedly trained using a sample image dataset. The model is iteratively optimized and violation detection probability thresholds are set until an optimized graph convolutional network model for violation detection based on human skeleton recognition is obtained. This model is used to determine whether a single frame image is in violation.

[0027] Real-time video behavior detection module: Real-time acquisition of video stream and extraction of key frame images, input into the optimized convolutional network model of violation detection graph based on human skeleton recognition, and obtain the violation judgment results of single frame images one by one.

[0028] A violation video detection device based on skeleton behavior recognition includes: a processor and a computer-readable storage medium on a server side; the processor is used to implement various instructions, and the computer-readable storage medium is used to store multiple instructions, which are adapted to be loaded by the processor and executed by the violation video detection method based on skeleton behavior recognition according to any one of claims 1-8.

[0029] The present invention has the following beneficial effects and advantages:

[0030] 1. To address the issue that the source node's ability to acquire features from neighboring nodes and the static skeleton graph cannot meet adaptive requirements, this invention proposes an Inception structure and dynamic skeleton graph based on a graph convolutional neural network, namely a multi-filter dynamic graph convolutional network. This method not only enables the source node to acquire features from distant nodes but also utilizes a dynamic skeleton graph structure to generate different body associations for different actions, improving the model's adaptability and accuracy, and achieving better results.

[0031] 2. This invention addresses the static connection of the human body structure, transforming it into a dynamic connection method by learning to add corresponding connections between two nodes that were not originally connected.

[0032] 3. This invention extends the Inception structure to graph convolution and combines it with a partitioning strategy. It uses 1-neighborhood and 2-neighborhood filters to extract features from the skeleton graph, and the nodes within each neighborhood follow a partitioning strategy, dividing the convolution kernel into three categories: the root node, the centripetal set (containing nodes closer to the root node and nodes equidistant from the root node), and the eccentric set. Finally, the feature maps are concatenated. This approach solves the problem of not being able to capture the influence of distant nodes on the source node.

[0033] 4. This invention proposes a novel attention mechanism that can transform a static skeleton graph into a dynamic graph structure. Through learning, it can generate new connections between physically unconnected nodes, making the action recognition algorithm more reasonable and accurate.

[0034] 5. The method of the present invention can detect and identify sensitive body movements and behaviors in live Internet videos in real time and accurately, providing an important guarantee for regulating the behavior of live streamers and creating a civilized Internet live streaming environment.

[0035] 6. This system is easy to operate and has a high recognition accuracy rate, making it suitable for widespread application in today's era of popular live streaming. Attached Figure Description

[0036] Figure 1 This is a diagram of the model structure in this invention;

[0037] Figure 2 This is a convolutional structure diagram of multiple filtering graphs; Detailed Implementation

[0038] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0040] The method of this invention includes: first, transforming static human body structural connections into dynamic connections by learning to add connections between two nodes that were not originally connected; second, processing the filters in the graph convolution using an Inception structure to increase the influence information of each joint source point. Based on these two aspects, a multi-filter dynamic graph convolutional neural network action recognition model is proposed. Finally, action model recognition research is conducted on the Kinetics-Skeleton dataset, and the results verify the recognition accuracy of the multi-filter dynamic graph convolutional neural network model. The action recognition network model consists of 9 spatiotemporal residual blocks, each composed of a multi-filter dynamic graph convolutional network, a temporal convolutional network, and a residual structure. The temporal convolutional network has a time window of 9, and the residual structure uses Dropout with a probability of 0.5 to prevent overfitting. When data first enters the model, it passes through a Batch Normalization (BN) layer to normalize the input data. Finally, a global average pooling layer is used to map the features of different samples to the same size, which is then provided to a softmax classifier for action classification and prediction. This invention extends the Inception structure to graph convolution and incorporates a partitioning strategy. It uses 1-neighborhood and 2-neighborhood filters to extract features from the skeleton graph, and each neighborhood node follows a partitioning strategy, dividing the convolution kernel into three categories: the root node, the centripetal set (containing nodes closer to the root node and nodes equidistant from it), and the eccentric set. Finally, the feature maps are concatenated. This approach solves the problem of not being able to capture the influence of distant nodes on the source node.

[0041] The illegal videos referred to in this invention are, for example, videos containing sensitive physical actions or movements that are illegal, irregular, or morally offensive. The method of this invention uses skeleton-based behavioral recognition training to identify such actions and behaviors in videos.

[0042] The aforementioned illegal video detection system includes: a deep learning-based abnormal action recognition model and pre-trained model weight parameters; a video acquisition program that collects video data from both abnormal and legitimate live streaming platforms; a video parsing program that processes video streams into image data; and a live streaming and video abnormal behavior detection program. The recognition model installed in the illegal video detection system is an abnormal behavior recognition model based on a multi-filter dynamic graph convolutional neural network for skeleton behavior recognition.

[0043] The deep learning-based abnormal behavior recognition model employs a multi-filter dynamic graph convolutional network model and a pre-trained model based on a large-scale image dataset.

[0044] The model weight parameters installed in the aforementioned illegal video detection system are obtained by training the detection model using legitimate and abnormal video stream data from internet live streaming platforms.

[0045] The video acquisition program used in the violation video detection system collects video stream data publicly shared on internet platforms. The video parsing program within the system processes the video stream frame by frame into image data and compresses the resolution using sparse sampling. The video anomaly detection program uses key video frame images obtained from the video parsing program as input, employs a deep learning neural network model for identification and detection, and ultimately outputs whether the input video is abnormal.

[0046] like Figure 1As shown, this invention employs a video violation detection model based on a multi-filter dynamic graph convolutional network. A publicly available dataset of both violating and non-violating videos is obtained online using a video acquisition module. The model is then trained using the multi-filter dynamic graph convolutional network, and the trained model's weight parameters are stored on a server for use in detecting video violations. The model works from top to bottom. First, it uses the Openpose pose estimation algorithm to estimate joint positions for each frame of the video, obtaining the corresponding keypoints (finding the joints) and their confidence scores. Then, it uses the BatchNorm method to perform batch normalization on the data input to the model. Next, a 9-layer residual block consisting of a Multi-Filtered Dynamic Graph Convolutional Network (Mfd-GCN), a Temporal Convolutional Network (TCN), and a ResNet residual structure extracts features. After extraction, global pooling is used to map features from different samples to the same size, reducing the number of model parameters while preserving useful features and improving the model's generalization ability. Finally, a fully connected layer maps the learned feature representations of the human skeleton joints to the sample label space, extracting useful features (connecting the joints to obtain the human skeleton). Finally, a softmax classifier provides auxiliary decision results.

[0047] The entire network model consists of 9 spatiotemporal residual blocks. Each spatiotemporal residual block is composed of a multi-filter dynamic graph convolutional network, a temporal convolutional network, and a residual structure. The temporal convolutional network has a time window of 9, and the residual structure uses Dropout with a probability of 0.5 to prevent overfitting. When data first enters the model, it passes through a batch normalization (BN) layer to normalize the input data. Finally, a global average pooling layer is used to map the features of different samples to the same size, which is then provided to the softmax classifier for action classification and prediction.

[0048] The following is combined with Figure 2 ,right Figure 1 The multi-filter graph convolution structure is described in detail.

[0049] The filters are categorized into three classes using a graph partitioning strategy, representing centripetal motion, centrifugal motion, and stationary motion features, respectively. This paper proposes a network structure similar to Inception, namely a multi-filter structure, which uses filters of different dimensions to perform convolution operations on nodes, and finally fuses the feature maps together.

[0050] Extending the Inception structure to graph convolution and incorporating a partitioning strategy, 1-neighborhood (K1 in the diagram) and 2-neighborhood (K2 in the diagram) filters are used to extract features from the skeleton graph. Nodes within each neighborhood follow a partitioning strategy, dividing the convolutional kernel into three categories: the root node, the centripetal set (containing nodes closer to the root node and nodes equidistant from the root node), and the eccentric set. Finally, the feature maps are concatenated, solving the problem of not being able to capture the influence of distant nodes on the source node, thus improving feature extraction. The multi-filter dynamic graph convolutional neural network expands the neighborhood of the origin (current point) to capture the influence of distant nodes on the origin, and uses a dynamic skeleton graph to adapt to the connection relationships between nodes of different behaviors, thereby changing the original feature extraction scheme of the skeleton graph and achieving more efficient feature extraction.

[0051] The present invention provides a violation video detection method based on skeleton behavior recognition, comprising: a processor and a computer-readable storage medium on a server side; the processor is used to implement various instructions, and the computer-readable storage medium is used to store multiple instructions, the instructions being adapted to be loaded and executed by the processor to implement the multimodal violation video detection method based on text and video fusion.

[0052] The logical instructions in the computer-readable storage medium described in this invention can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0053] In addition, the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0054] The embodiments described above will help those skilled in the art to further understand the present invention, but do not limit the present invention in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

Claims

1. A method for detecting violations in videos based on skeleton behavior recognition, characterized in that, Includes the following steps: Video capture steps: Obtain video stream data from internet live streaming platforms; Keyframe extraction steps: Sparse sampling is used to obtain single-frame images from the video stream data; A pose estimation algorithm is used to estimate joint positions in a single frame image to obtain human multi-joint features; points with average joint confidence scores higher than a threshold are selected as joint features in a single frame image to obtain keyframe images. Steps to create an image dataset: Classify and label keyframe images as images of violations or non-violations, and remix the samples to create a sample image dataset; The steps for establishing a graph convolutional network model for violation detection are as follows: Establish a multi-filter dynamic graph convolutional neural network model; repeatedly train the network using multiple sample image datasets in batches; iteratively optimize the model and set a violation detection probability threshold until an optimized graph convolutional network model based on human skeleton recognition for violation detection is obtained; this model is used to determine whether multiple consecutive frames of dynamic images contain violation actions. The graph convolutional network model for violation detection based on human skeleton recognition includes: sequentially connected batch normalization (BN) layers, 9 spatiotemporal residual blocks, a global average pooling layer, a fully connected layer, and a softmax classifier. The BN layers are used to normalize the input data. Each spatiotemporal residual block consists of a multi-filter dynamic graph convolutional network, a temporal convolutional network, and a residual structure. The global average pooling layer maps features from different samples to the same size, reducing the number of parameters while preserving useful features and improving generalization ability. The fully connected layer stores learned data... The feature representations of human skeletal joints are mapped to the sample label space to extract useful features; the softmax classifier is used for action classification prediction; the multi-filter dynamic graph convolutional structure includes: extending the Inception structure to a graph convolutional network, using 1-neighborhood and 2-neighborhood filters to extract features from the skeleton map, and the nodes in each neighborhood follow a partitioning strategy: the convolutional kernels are divided into three categories, namely root node, centripetal set and eccentric set, for processing the input sample image, and finally the feature maps are connected together to form an adjacency pixel matrix for filtering joint positions; Real-time video behavior detection steps: Real-time acquisition of video stream and extraction of keyframe images, input into an optimized convolutional network model for violation detection based on human skeleton recognition, and obtain violation judgment results.

2. The method for detecting violations in videos based on skeleton behavior recognition according to claim 1, characterized in that, The human multi-joint features include the pixel positions of 15 joint vectors and their confidence scores (x, y, acc).

3. The method for detecting violations in videos based on skeleton behavior recognition according to claim 1, characterized in that, The sample image dataset is divided into a training set, a test set, and a validation set.

4. The method for detecting violations in videos based on skeleton behavior recognition according to claim 1, characterized in that, The temporal convolutional network has a time window of 9, and the residual structure uses a dropout parameter with a probability of 0.

5.

5. The method for detecting violations in videos based on skeleton behavior recognition according to claim 1, characterized in that, During training, the training set data is input into the model in batches to obtain the dynamic human skeleton joint connections. The dynamic human skeleton joint connections and their corresponding behaviors are learned through a multi-filtered dynamic graph convolutional structure. A softmax classifier is used to give the probability of whether the current batch of input multi-frame image data is in violation.

6. The method for detecting violations in videos based on skeleton behavior recognition according to claim 1, characterized in that, The interval between the sparsely sampled video frames is 5 seconds.

7. A violation video detection system based on skeleton behavior recognition, used to identify violations in video stream data from internet live streaming platforms, characterized in that, The system includes the following program modules: Video capture module: Acquires video stream data from internet live streaming platforms; Keyframe extraction module: Sparse sampling is used to obtain single-frame images from video stream data; a pose estimation algorithm is used to estimate joint positions in the single-frame images to obtain human multi-joint features; points with average joint confidence scores higher than a threshold are selected as features in the single-frame images to obtain keyframe images. The image dataset creation module classifies and labels keyframe images as either violation-related or non-violation-related action images, and then remixes the samples to create a sample image dataset. The program module for establishing a graph convolutional network model for violation detection is as follows: A multi-filter dynamic graph convolutional neural network model is established, and the network is repeatedly trained with a sample image dataset. The model is iteratively optimized and a violation discrimination probability threshold is set until an optimized graph convolutional network model for violation detection based on human skeleton recognition is obtained. The model is used to determine whether multiple consecutive frames of images violate regulations. The graph convolutional network model for violation detection based on human skeleton recognition includes: sequentially connected BN layers, 9 spatiotemporal residual blocks, a global average pooling layer, a fully connected layer, and a softmax classifier. The BN layers are batch normalization layers used to normalize the input data. Each spatiotemporal residual block consists of a multi-filter dynamic graph convolutional network, a temporal convolutional network, and a residual structure. The global average pooling layer maps the features of different samples to the same size, reducing the number of model parameters while retaining useful features and improving the model's generalization ability. The fully connected layer maps the learned feature representations of human skeletal joints to the sample label space, extracting useful features; the softmax classifier is used for action classification prediction; the multi-filter dynamic graph convolutional structure includes: extending the Inception structure to a graph convolutional network, using 1-neighborhood and 2-neighborhood filters to extract features from the skeleton map, and the nodes in each neighborhood follow a partitioning strategy: the convolutional kernels are divided into three categories, namely root node, centripetal set and eccentric set, for processing the input sample image, and finally the feature maps are connected together to form an adjacency pixel matrix for filtering joint positions; Real-time video behavior detection module: Real-time acquisition of video streams and extraction of keyframe images, input into an optimized convolutional network model for violation detection based on human skeleton recognition, and obtain violation judgment results.

8. A violation video detection device based on skeleton behavior recognition, comprising: The processor and computer-readable storage medium on the server side; The processor is used to implement various instructions, and the computer-readable storage medium is used to store multiple instructions, characterized in that the instructions are adapted to be loaded by the processor and executed by the illegal video detection method based on skeleton behavior recognition as described in any one of claims 1-6.

Images