Illegal action recognition method and device in video, medium and program product

Abstract

The invention discloses a method, a device and equipment for identifying illegal actions in a video and a storage medium, and relates to the technical field of computers. The method comprises the following steps: carrying out joint point detection on a target video stream to determine a joint point coordinate of each video frame, and generating a joint point coordinate sequence based on the joint point coordinates; inputting the articulation point coordinate sequence and the target video stream into a double-flow feature modeling network, and extracting articulation point spatial-temporal features of the articulation point coordinate sequence and image semantic features of the target video stream; and performing cross attention processing on the joint point spatial-temporal features and the image semantic features, and performing illegal action recognition according to the obtained fused features. It can be seen that the double-flow feature modeling network is utilized to extract the joint point spatio-temporal features and the image semantic features at the same time, the fused features rich in the context relation are generated through cross attention processing, illegal action recognition is conducted according to the fused features, and the accuracy of illegal action recognition in the video is improved through the multi-modal features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a method and device for identifying illegal action in video, equipment and storage medium. BACKGROUND

[0002] With the development of Internet technology, more and more social platforms build new entertainment ecology through multi-content such as live broadcast. The existing illegal detection method has defects such as high missed detection rate and weak real-time response, and is limited in complex background interference and multi-person interactive scene. SUMMARY

[0003] Therefore, the present application aims to provide a method and device for identifying illegal action in video, which can improve the accuracy of identifying illegal action in video. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a method for identifying illegal action in video, comprising:

[0005] Obtaining a target video stream, detecting the joint nodes of the target video stream to determine the joint node coordinates of each video frame, and generating a joint node coordinate sequence based on the joint node coordinates;

[0006] Inputting the joint node coordinate sequence and the target video stream into a double-stream feature modeling network, extracting the joint node spatio-temporal features of the joint node coordinate sequence by using a joint node time sequence subnetwork, and extracting the image semantic features of the target video stream by using an image semantic subnetwork;

[0007] Performing cross-attention processing on the joint node spatio-temporal features and the image semantic features to obtain fused features;

[0008] Identifying illegal action based on the fused features.

[0009] Optionally, the step of detecting the joint nodes of the target video stream to determine the joint node coordinates of each video frame comprises:

[0010] Performing cross-frame object matching on the target video stream by motion trajectory and appearance feature similarity;

[0011] For each matched object, determining the joint node coordinates of the object in the video frame by joint node detection.

[0012] Optionally, the step of generating a joint node coordinate sequence based on the joint node coordinates comprises:

[0013] Determining whether there is joint node occlusion based on dynamic confidence threshold and spatio-temporal consistency based on the joint node coordinates; the dynamic confidence threshold is determined based on the proportion of the occlusion area in the current frame;

[0014] For an abnormal joint node with continuous occlusion, the abnormal joint node is dynamically corrected according to a forward prediction and a backward correction of an object to which the abnormal joint node belongs;

[0015] Trajectory smoothing processing is performed on the joint nodes based on joint node coordinates in a sliding window to obtain a joint node coordinate sequence corresponding to the target video stream.

[0016] Optionally, the joint node time sequence subnetwork comprises an input layer, a multi-head attention mechanism module and a time sequence convolution module; the joint node spatio-temporal feature of the joint node coordinate sequence is extracted by using the joint node time sequence subnetwork, comprising:

[0017] The joint node coordinate sequence is mapped into a multi-dimensional position code through the input layer;

[0018] Based on the multi-dimensional position code, the multi-head attention mechanism module is used to learn the motion patterns of different limb parts;

[0019] Combined with the motion patterns of different limb parts, the time sequence convolution module is used to extract the action cycle features corresponding to the joint node coordinate sequence to obtain the joint node spatio-temporal feature.

[0020] Optionally, the image semantic subnetwork comprises a lightweight convolutional neural network, a region focusing module, a deformable convolution module and a multi-scale feature fusion module;

[0021] The image semantic feature of the target video stream is extracted by using the image semantic subnetwork, comprising:

[0022] The initial feature maps of different scales of the target video stream are extracted by the lightweight convolutional neural network;

[0023] The key regions in the feature maps are determined by the region focusing module, and the complex features of the key regions are extracted by the deformable convolution module, and the initial feature maps are optimized according to the load features;

[0024] The optimized feature maps of different scales are fused by the multi-scale feature fusion module to obtain the image semantic feature corresponding to the target video stream.

[0025] Optionally, the cross-attention processing is performed on the joint node spatio-temporal feature and the image semantic feature to obtain a fused feature, comprising:

[0026] The interaction feature of the joint node spatio-temporal feature and the image semantic feature is learned by using a feature interaction enhancement module, and the cross feature of the joint node spatio-temporal feature and the image semantic feature is learned by using a cross-attention module;

[0027] The feature fusion is performed by using a gated fusion mechanism based on the node spatio-temporal feature, the image semantic feature, the interaction feature, the cross feature, and the dynamic attention weight to obtain the fused feature.

[0028] Optionally, before the feature fusion is performed by using the gated fusion mechanism based on the node spatio-temporal feature, the image semantic feature, the interaction feature, the cross feature, and the dynamic attention weight to obtain the fused feature, the method further includes:

[0029] The motion intensity is determined according to the node speed, and the semantic abnormality degree is determined according to the difference between the image semantic feature and the normal sample feature.

[0030] The base attention weight is determined based on the motion intensity and the semantic abnormality degree.

[0031] It is determined whether there is a prop in the video frame, and if there is, the base attention weight is adjusted by a preset boost amplitude to obtain the dynamic attention weight.

[0032] Optionally, after the rule-violating action recognition is performed based on the fused feature, the method further includes:

[0033] According to the confidence of the rule-violating action recognition result and the corresponding relationship between the confidence and the disposal strategy, a corresponding disposal strategy is executed.

[0034] A video stream containing a node motion trajectory and an enlarged view of a rule-violating action region are displayed through a human-computer interaction interface.

[0035] A risk heat map is used to mark a rule-violating action risk value of the video at different time periods.

[0036] Optionally, the dual-flow feature modeling network is configured through knowledge distillation and adaptive feature selection.

[0037] The adaptive feature selection configuration includes an importance score configuration and a background feature truncation configuration.

[0038] The dual-flow feature modeling network evaluates the importance of each channel in the network for the current video frame according to the importance score configuration, and filters out non-key convolution channels and closes them according to the importance.

[0039] The dual-flow feature modeling network identifies the foreground region and the background region of the video frame according to the background feature truncation configuration, and performs feature truncation processing on the background region to ensure the calculation resources for the foreground region.

[0040] In a second aspect, the present application discloses an electronic device, comprising:

[0041] a memory for storing the computer program;

[0042] a processor for executing the computer program to implement the foregoing method for identifying a rule-violating action in a video.

[0043] In a third aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the foregoing method for identifying a rule-violating action in a video.

[0044] In a fourth aspect, the present application discloses a computer program product comprising a computer program, which, when executed by a processor, implements the foregoing method for identifying a rule-violating action in a video.

[0045] In the present application, a target video stream is obtained, joint point detection is performed on the target video stream to determine joint point coordinates of each video frame, and a joint point coordinate sequence is generated based on the joint point coordinates; the joint point coordinate sequence and the target video stream are input into a dual-stream feature modeling network, joint point spatio-temporal features of the joint point coordinate sequence are extracted by using a joint point time sequence subnetwork, and image semantic features of the target video stream are extracted by using an image semantic subnetwork; cross-attention processing is performed on the joint point spatio-temporal features and the image semantic features to obtain fused features; and rule-violating action recognition is performed based on the fused features. As can be seen, the dual-stream feature modeling network is used to simultaneously extract joint point spatio-temporal features and image semantic features, and cross-attention processing is used to generate fused features rich in context relationships, rule-violating action recognition is performed according to the fused features, and the accuracy of rule-violating action recognition in a video is improved by using multi-modal features. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0047] Figure 1 A flowchart of a rule-violating action recognition method provided by the present application is shown in FIG. 1;

[0048] Figure 2 A flowchart of a specific joint point correction method provided by the present application is shown in FIG. 2;

[0049] Figure 3 A flowchart of a specific feature extraction method provided by the present application is shown in FIG. 3;

[0050] Figure 4 A flowchart of a specific feature fusion method provided by the present application is shown in FIG. 4;

[0051] Figure 5 A specific violation action processing method flowchart is provided for the present application;

[0052] Figure 6 A video live broadcast auditing system architecture schematic diagram is provided for the present application;

[0053] Figure 7 An electronic device structure diagram is provided for the present application. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in a clear and complete manner with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0055] In the prior art, the violation detection method has defects such as high missed detection rate and weak real-time response, and has performance limitations in complex background interference and multi-person interactive scenes. To overcome the above technical problems, the present application proposes a violation action recognition method in a video, which can improve the accuracy of violation action recognition in a video.

[0056] The embodiments of the present application disclose a violation action recognition method in a video, referring to Figure 1 As shown in the figure, the method can include the following steps:

[0057] Step S11: Obtain a target video stream, detect the joint nodes of the target video stream to determine the joint node coordinates of each video frame, and generate a joint node coordinate sequence based on the joint node coordinates.

[0058] First, the video stream to be detected is obtained, which can be a live video stream, and can also be a complete video recorded. The target video stream can be processed by streaming media, that is, the live stream is decoded, key frames are sampled, and the resolution is adaptively adjusted. Specifically, the input live stream is fragmented using the HLS protocol, and then preprocessed by a low-performance node. The preprocessing includes but is not limited to: resolution downsampling, key frame extraction, and dynamic adjustment of the ROI region (such as focusing on the upper body region of the anchor).

[0059] In a specific embodiment, determining the joint coordinates of each video frame by performing joint point detection on the target video stream includes: performing cross-frame object matching on the target video stream based on motion trajectory and appearance feature similarity; and determining the joint coordinates of each matched object in the video frame through joint point detection. Utilizing motion trajectory and appearance feature similarity analysis to perform cross-frame object matching on the target video stream means matching identical objects (IDs) at different time periods within the video stream. By adding appearance feature similarity analysis, the problem of motion trajectory interruption caused by a target being briefly occluded or moving out of the frame and then reappearing can be solved. Appearance features can effectively re-identify and associate the interrupted target. Simultaneously, if multiple similar-looking targets are close together and crowded, relying solely on appearance can easily lead to confusion; motion trajectory can efficiently distinguish them. By combining motion trajectory and appearance feature similarity for cross-frame object matching of the target video stream, the two can be mutually verified, reducing false matching and making the tracking box more stable and accurate. This improves the accuracy and robustness of matching and effectively handles complex scenarios.

[0060] For example Figure 2 As shown, through target tracking algorithms (such as...) The tracking algorithm acquires the motion trajectory of objects within the video and combines appearance features for target tracking, i.e., cross-frame object matching. Multiple targets (i.e., multiple human bodies) are tracked simultaneously using the SORT algorithm to ensure cross-frame ID consistency. Appearance feature similarity calculation can use a lightweight ReID model to generate a 128-dimensional feature vector. Existing joint detection models are used to detect joints in each video frame of the target video stream. Joint detection locates semantically meaningful joints and outputs their coordinate positions. For example, human joint detection typically includes joints in multiple parts such as the hand, shoulder, elbow, wrist, hip, and knee. The hand includes wrist-metacarpophalangeal joints, metacarpophalangeal joints, and interphalangeal joints.

[0061] In a specific embodiment, generating a joint point coordinate sequence based on the joint point coordinates may include the following steps:

[0062] S111: Based on the key point coordinates, determine whether key point occlusion exists by using a dynamic confidence threshold and spatiotemporal consistency; the dynamic confidence threshold is determined based on the proportion of the occlusion area in the current frame.

[0063] Specifically, the dynamic confidence threshold and the spatio-temporal consistency can be used to determine whether the joint is occluded, that is, whether the joint is occluded is determined according to the trajectory of the joint in time and space and the dynamic confidence threshold; the dynamic confidence threshold is determined based on the proportion of the occlusion area in the current frame; the dynamic confidence threshold is calculated by setting a dynamic confidence threshold for each joint for local confidence evaluation, and the dynamic confidence threshold calculation formula can be Ct = 0.7-0.2 x (proportion of occlusion area in current frame). Considering that the probability of different joints being occluded under normal circumstances is different, the dynamic confidence threshold is set for each joint for targeted evaluation, thereby improving the accuracy of the determination.

[0064] S112: For the abnormal joint with continuous occlusion, the abnormal joint is dynamically corrected by forward prediction and backward correction according to the object to which the abnormal joint belongs.

[0065] The joint with continuous occlusion is selected as an abnormal joint, for example, the joint is corrected when the joint is lost for more than 10 consecutive frames, and the joint that is temporarily lost and then appears does not need to be concerned, thereby avoiding resource waste. The forward prediction and backward correction are used for dynamic correction of the abnormal joint in the time dimension, and the correction operation can be realized by constructing a motion trajectory prediction model based on a long short-term memory network. The forward prediction is used to temporarily fill in the predicted value when the joint is occluded, for example, the Kalman filter is used to predict the joint position in the next 3 frames; the backward filling is used to backfill and correct the real data after the occlusion ends, that is, the motion trajectory is reconstructed by using cubic spline interpolation for the lost frame. At the same time, the joint can be spatially corrected by using bilinear interpolation, and the constraint between different joints in the same frame (such as the limb length constraint), so as to ensure that the joint conforms to the human body constraint.

[0066] S113: The joint trajectory is smoothed based on the joint coordinates in the sliding window, and a joint coordinate sequence corresponding to the target video stream is obtained.

[0067] Weighted average or low-pass filtering can be used to realize the smoothing, so as to suppress high-frequency jitter; for example, the sliding window takes the joint positions in the last several frames (such as 5 frames), the average value or trend line of the joint positions is calculated, and the smoothed trajectory is used to replace the original jitter data, thereby eliminating the coordinate mutation caused by the shaking of the camera.

[0068] By tracking the stable target ID first and then optimizing the joint based on the tracking result, not only the time jitter is processed, but also the coordinates can be corrected by using the spatial relationship (human body topology). The joint detection mechanism optimized by the multi-target tracking described above uses an improved multi-target tracking algorithm (Deep SORT) ) and spatiotemporal interpolation technology, aiming at the unstable problem of joint point detection caused by human body occlusion and motion blur in live broadcast, combining target tracking algorithm and joint point correction method of spatiotemporal interpolation, through cross-frame target ID matching and spatiotemporal interpolation optimization of abnormal coordinates, the detection jitter and drift problem caused by occlusion in live broadcast scene, and the joint point detection jitter problem caused by rapid action switching and camera jitter are solved, the stability of dynamic behavior modeling is improved, and the trajectory prediction error is reduced to 2.1 pixels.

[0069] Step S12: input the joint point coordinate sequence and the target video stream into a double-stream feature modeling network, extract the joint point spatiotemporal feature of the joint point coordinate sequence by using a joint point time sequence subnetwork, and extract the image semantic feature of the target video stream by using an image semantic subnetwork.

[0070] The joint point coordinate sequence and the target video stream are input into a double-stream feature modeling network, and the double-stream feature modeling network includes a joint point time sequence subnetwork and an image semantic subnetwork. The joint point time sequence subnetwork is used to extract the joint point spatiotemporal feature of the joint point coordinate sequence, and the image semantic subnetwork is used to extract the image semantic feature of the target video stream. Specifically, the joint point time sequence subnetwork can be constructed based on the Transformer architecture to extract continuous action patterns, and the image semantic subnetwork can be constructed based on a lightweight convolutional neural network (CNN) to extract visual semantic features of human body regions and key parts (such as hands and props). By capturing dynamic evolution rules and static visual clues in parallel through double streams, the misjudgment rate of semantically similar behaviors (such as suggestive dance and normal performance) is reduced. By fusing joint point motion features and image semantic features, the interference of prop occlusion, multi-person interaction and dynamic background is effectively dealt with.

[0071] The double-stream feature modeling network is designed in parallel based on the joint point time sequence modeling branch of the Transformer and the lightweight CNN image semantic branch, and combines a cross-modal attention mechanism to dynamically allocate feature weights, which significantly distinguishes between similar low-level behaviors (such as suggestive dance and normal performance), and the false alarm rate of semantically similar behaviors is reduced to 1.3%, which is 40% less than existing methods.

[0072] In specific embodiments, the joint point time sequence subnetwork includes an input layer, a multi-head attention mechanism module, and a time sequence convolution module; and the extraction of the joint point spatiotemporal feature of the joint point coordinate sequence by using the joint point time sequence subnetwork can include: mapping the joint point coordinate sequence to multi-dimensional position encoding through the input layer; learning the motion patterns of different limb parts based on the multi-dimensional position encoding by using the multi-head attention mechanism module; and combining the motion patterns of the different limb parts, extracting the action period features corresponding to the joint point coordinate sequence by using the time sequence convolution module to obtain the joint point spatiotemporal feature.

[0073] For example Figure 3 As shown, the input layer maps the joint coordinate sequence into a 256-dimensional position encoding, which can preserve the spatiotemporal information of the joints; 8 attention heads are used to capture the motion patterns of different body parts (upper limbs / lower limbs / trunk), such as high-frequency pelvic swing and hand-sensitive area stay; the parameters are shared between the attention heads to implicitly associate them, and the final features need to be spliced or weighted. A temporal convolutional network (TCN) layer is added at the back end of the encoder to extract the action cycle features from the high-order features output by the attention module, and to extract the action periodicity and long-term temporal dependence features. Through dynamic joint trajectory analysis, the motion trajectories of multiple joints of the human body are tracked in real time, and abnormal limb motion patterns are identified,

[0074] In specific embodiments, the image semantic sub-network includes a lightweight convolutional neural network, a region focus module, a deformable convolution module, and a multi-scale feature fusion module; the image semantic features of the target video stream can be extracted using the image semantic sub-network, which can include: extracting initial feature maps of different scales of the target video stream through the lightweight convolutional neural network; determining key regions in the feature maps through the region focus module, and extracting complex features of the key regions using the deformable convolution module, and optimizing the initial feature maps according to the load features; using a multi-scale feature fusion module to fuse the optimized feature maps of different scales to obtain the image semantic features corresponding to the target video stream.

[0075] For example Figure 3 As shown, the backbone network uses a lightweight convolutional neural network (such as MobileNetV3-Small network removing the last 3 convolutional blocks), and adds a region focus module to determine the key regions in the feature maps (such as hand region / prosthetic region), which can also be combined with head pose to determine the gaze focus region; then through deformable convolution (Deformable Conv) to strengthen the feature extraction of the hand / prosthetic region; finally through a multi-scale feature fusion module to fuse the feature maps of shallow details, middle semantics, and high-level abstractions. Through local semantic enhancement detection by the region focus module and deformable convolution, the use of props is analyzed according to the image features of the hand region, further improving the accuracy and comprehensiveness of the violation behavior recognition.

[0076] Further, the above-mentioned double-flow feature modeling network is configured through knowledge distillation and adaptive feature selection. For example, a teacher model (e.g., a ResNet50+Transformer double-flow network with a parameter amount of 98M) is first trained to reach stable performance; a student model (e.g., a MobileNetV3+lightweight Transformer) is trained again under the guidance of the teacher model, and the parameter amount is reduced to 12M; the student model is constrained by a distillation loss function during the training process. The distillation loss function can be a KL divergence constraint output distribution combined with an intermediate layer feature similarity constraint.

[0077] The adaptive feature selection configuration includes importance score configuration and background feature truncation configuration. According to the importance score configuration, the double-flow feature modeling network evaluates the importance of each channel in the network for the current video frame, and filters out non-key convolutional channels and closes them. According to the background feature truncation configuration, the double-flow feature modeling network identifies the foreground and background regions of the video frame, and performs feature truncation processing on the background region to ensure the calculation resources for the foreground region. That is, the double-flow feature modeling network includes an adaptive feature selection module to reduce computational redundancy by dynamically selecting. For example, 30% of non-key convolutional channels are dynamically closed, and feature truncation is performed on the background region to reserve at least 40% of the calculation resources for processing the foreground region.

[0078] For example, for an input picture, the adaptive feature selection module quickly analyzes the feature map and generates an importance score for each channel, and closes those channels with scores below a threshold or in the lower ranks. Different input pictures have different channels closed, that is, this decision is made in real time during the model running process. Feature truncation is performed on the background region. The model can first run a very lightweight prelude network (such as target detection or saliency detection) to quickly identify the foreground and background in the input picture. The background region performs feature truncation to allocate the main calculation resources to the foreground region for fine processing, and uses much less resources to process the background region roughly. Since the foreground and background of different video frames are different, the resource allocation is also different. This decision is also made in real time during the model running process.

[0079] The distilled model is deployed. Specifically, the model can be converted into an FP16 precision engine, and dynamic shape optimization is enabled to support different resolution inputs. Dynamic shape optimization enables the optimized TensorRT engine to efficiently process inputs of different sizes without the need to rebuild a separate engine for each size. The model can run in pipeline parallelism, such as node detection and image feature extraction belonging to different GPU nodes; and inter-frame interleaving processing is implemented, that is, the current frame feature extraction is parallel with the next frame data preprocessing to improve processing speed.

[0080] Introducing knowledge distillation to compress the model parameter amount to obtain a lightweight real-time inference system, combined with an adaptive feature selection module to reduce computational redundancy, while ensuring accuracy in complex scenes (occlusion, multi-person interaction) (recognition accuracy in complex occlusion scenes can be improved to 92.7%, more than 15% higher than traditional single-mode methods), Real-time low vulgar behavior interception and early warning, false positive rate reduced to 1.3%, meet the business needs of live streaming platforms with high concurrency and low latency.

[0081] Step S13: Cross attention processing is performed on the joint node spatio-temporal features and the image semantic features to obtain fused features.

[0082] By dynamically allocating feature weights through cross-modal attention mechanisms, the discrimination ability for semantically similar behaviors and key frames is enhanced, and the sensitivity to low vulgar behavior key frames (specific gestures, prop use) is enhanced, solving the problem of insufficient discrimination ability for semantically ambiguous behaviors in traditional weighted fusion. Deeply coupled human motion temporal features and visual semantic static features, combined with optimization algorithm design, effectively solved the dynamic avoidance problem of hidden vulgar behavior, significantly improved the accuracy and scene adaptability of the recognition system,

[0083] In specific embodiments, before the feature fusion using a gated fusion mechanism based on the joint node spatio-temporal features, the image semantic features, the interaction features, the cross features, and the dynamic attention weights to obtain the fused features, it further includes: determining motion intensity according to joint node speed, and determining semantic abnormality according to the difference between image semantic features and normal sample features; determining a basic attention weight based on the motion intensity and the semantic abnormality; determining whether there is a prop in the video frame, if there is, adjusting the basic attention weight by a preset increase, to obtain a dynamic attention weight. That is, when feature fusion is performed, a dynamic attention weight is used to improve the sensitivity to illegal behavior, solve the problem of insufficient discrimination ability for semantically ambiguous behaviors, and improve the accuracy of illegal action recognition in videos.

[0084] For example Figure 4 As shown, the basis weight is generated according to the motion intensity and the semantic abnormality. Specifically, the motion intensity selects the joint node speed variance, the formula is: ; wherein, represents the speed of joint node i, wherein, represents the average value of the joint speed, N represents the total number of joints, and the data dispersion degree is measured by the speed variance. The cosine distance of the image feature and the normal sample is taken as a semantic anomaly degree index. The base attention weight W_base=0.5×(motion intensity)+0.5×(semantic anomaly degree) is calculated; when the prop is detected, the base attention weight is dynamically adjusted according to a preset increase amplitude, for example, the image feature weight is increased by 0.3. Violation behaviors are often accompanied by the use of props, and the attention weight is dynamically adjusted according to the props, which further improves the recognition ability of the violation actions.

[0085] In specific embodiments, the cross-attention processing of the joint spatio-temporal feature and the image semantic feature to obtain the fused feature comprises: learning the interaction feature of the joint spatio-temporal feature and the image semantic feature by using a feature interaction enhancement module, and learning the cross feature of the joint spatio-temporal feature and the image semantic feature by using a cross-attention module; based on the joint spatio-temporal feature, the image semantic feature, the interaction feature, the cross feature, and the dynamic attention weight, the feature fusion is performed by using a gated fusion mechanism to obtain the fused feature. The feature interaction enhancement module learns the interaction feature of the joint spatio-temporal feature and the image semantic feature, the cross-attention module establishes the association matrix of the joint motion vector and the image local feature, and the association analysis of the hand and the pelvic region is highlighted and enhanced; the gated fusion mechanism adopts a Sigmoid function to control the contribution proportion of different modal features, and the feature fusion formula is:

[0086] ; wherein, F_fusion is the fused feature, F_joint is the joint spatio-temporal feature, F_image is the image semantic feature, and W_j is the dynamic weight. is a Sigmoid function.

[0087] Step S14: performing violation action recognition based on the fused feature.

[0088] Finally, the fused feature obtained by the cross-attention processing is used for violation action recognition. The whole process fuses the human joint time sequence motion feature and the image local semantic feature, constructs a cross-modal collaborative analysis framework, that is, the complex scene recognition ability is improved through multi-modal feature fusion, and the feature extraction failure problem caused by prop shielding, dynamic background interference and multi-person interaction in the live streaming scene is effectively solved. By using the scheme of the present application, the obscene behaviors such as suggestive actions and violation prop use can be recognized in real time in the live streaming platform, the violation content interception efficiency is improved by 60%, and the artificial auditing workload is reduced by at least 35%.

[0089] From the above, in the embodiment, the target video stream is matched across frames by combining the motion trajectory and the appearance feature similarity, the accuracy of target matching is improved, the spatial and temporal features of the key nodes and the image semantic features are extracted by using the dual-stream feature modeling network, the accuracy of action recognition is improved by using the fused features of the spatial and temporal features of the key nodes and the image semantic features, and the sensitivity to illegal behavior is improved by using dynamic attention weight during feature fusion, so that the problem of poor discrimination of semantic ambiguous behavior is solved, and the accuracy of illegal action recognition in the video is improved.

[0090] On the basis of the above embodiment, the cross-attention processing of the spatial and temporal features of the key nodes and the image semantic features to obtain the fused features comprises: based on the spatial and temporal features of the key nodes, the image semantic features, the dynamic attention weight, and the audio features, the fused features are obtained by using a gating fusion mechanism, that is, on the basis of the current key nodes and image features, the audio feature analysis (such as speech keyword detection and background sound effect recognition) is added, and the vulgar behavior is judged by multi-modal alignment, for example, the sexual suggestion action is combined with the specific voice content. In addition, it should be noted that when the audio feature analysis is combined, the audio-visual synchronization alignment processing needs to be performed, that is, the spatial and temporal features of the key nodes, the image semantic features, and the audio features corresponding to the same video frame are fused.

[0091] On the basis of the above embodiment, after the illegal action recognition based on the fused features, it further comprises: according to the confidence of the illegal action recognition result and the corresponding relationship between the confidence and the disposal strategy, the corresponding disposal strategy is executed; the video stream containing the motion trajectory of the key nodes and the enlarged view of the illegal action region are displayed through the man-machine interaction interface; and the risk heat map is used to mark the illegal action risk value of the video in different time periods.

[0092] For example Figure 5 As shown, by pre-configuring a hierarchical disposal strategy, different confidence corresponds to different processing strategies, for example: first-level interception, triggering automatic blocking for explicit illegal behavior (such as exposing the lower body) with a confidence of >90%, such as triggering real-time warning, live interruption or content shielding; second-level warning, live picture coding and manual review for suspected behavior (such as simulated action) with a confidence of 70%-90%; third-level record, generating an audit log for a risk behavior with a confidence of 50%-70% for subsequent user behavior portrait analysis. At the same time, a multi-modal feature visualization panel is provided for manual auditors, and the visual interface includes a key node motion trajectory heat map and a local image highlight marker, for example, the left interface displays the original live picture superimposed with the key node motion trajectory, and the right interface displays the enlarged view of the local high-risk area, such as the close-up of the hand and the prop. The risk heat map refers to marking the behavior risk value in different time periods with a color gradient, and supports jumping to the corresponding video segment by clicking the hot area, so that the administrator can have a global control.

[0093] The identified action recognition result also supports manual review in this embodiment, which can specifically sort and review a queue according to a behavior risk level; an administrator manually confirms a disposal mode by viewing a multi-modal evidence chain. For misjudged videos, unblock and code are released and released, and for manually confirmed illegal content, live interruption or shielding processing is performed.

[0094] For example Figure 6 A specific live broadcast review system architecture diagram is shown, and the system adopts a distributed processing cluster form deployed in the cloud. The stream media processing layer is responsible for live stream decoding, key frame sampling, and resolution adaptive adjustment; the feature extraction layer runs a joint node detection engine and an image feature extraction engine in parallel, and outputs joint node spatio-temporal features and image semantic features; the decision fusion layer implements cross-modal feature fusion and behavior classification decision; the disposal execution layer calls platform open API to perform blocking, warning and other operations according to the risk level. Through adaptive detection and lightweight deployment, while ensuring the accuracy of vulgar behavior recognition, the daily processing capacity reaches millions of live streams, helping to build a healthy and compliant live broadcast ecology. And through real-time interception and early warning mechanism, the user report quantity decreases by 45%, and the platform content safety score increases by 20%. In some embodiments, by deploying the above system on multiple live broadcast platforms, and then updating the global model by aggregating the data features of each node through federated learning, the problems of data privacy and uneven sample distribution can be solved.

[0095] In some embodiments, feature extraction can also use a 3D convolutional neural network, that is, the target video stream is input into a 3D convolutional neural network, and action evolution features and visual semantic information are extracted and used for illegal action recognition, avoiding the complexity of multi-modal feature separation extraction. However, 3D convolution has a large amount of calculation and large processing delay, and the anti-interference ability of action evolution features is lower than that of joint node features.

[0096] Correspondingly, the embodiment of the application also discloses a video illegal action recognition device, which comprises:

[0097] The cross-frame object matching module is configured to obtain a target video stream, perform joint node detection on the target video stream to determine joint node coordinates of each video frame, and generate a joint node coordinate sequence based on the joint node coordinates.

[0098] The feature extraction module is configured to input the joint node coordinate sequence and the target video stream into a double-stream feature modeling network, extract joint node spatio-temporal features of the joint node coordinate sequence by using a joint node time sequence subnetwork, and extract image semantic features of the target video stream by using an image semantic subnetwork.

[0099] The feature fusion module is configured to perform cross-attention processing on the joint node spatio-temporal features and the image semantic features to obtain fused features.

[0100] a rule violation action recognition module configured to recognize a rule violation action based on the fused features.

[0101] As can be seen, in the embodiment, the joint node detection is performed on the target video stream to determine the joint node coordinates of each video frame, and the joint node coordinate sequence is generated based on the joint node coordinates; the joint node coordinate sequence and the target video stream are input into the dual-stream feature modeling network to extract the joint node spatio-temporal features of the joint node coordinate sequence and the image semantic features of the target video stream; the cross attention processing is performed on the joint node spatio-temporal features and the image semantic features, and the rule violation action recognition is performed based on the obtained fused features. As can be seen, the dual-stream feature modeling network is used to simultaneously extract the joint node spatio-temporal features and the image semantic features, and the cross attention processing is used to generate the fused features rich in context relationship, and the rule violation action recognition is performed based on the fused features, so that the multi-modal features are used to improve the accuracy of the rule violation action recognition in the video.

[0102] In some specific embodiments, the cross-frame object matching module can specifically include:

[0103] an object matching unit configured to perform cross-frame object matching on the target video stream by motion trajectory and appearance feature similarity;

[0104] a joint node detection unit configured to determine joint node coordinates of each matched object in a video frame by joint node detection.

[0105] In some specific embodiments, the cross-frame object matching module can specifically include:

[0106] a proportion determination unit configured to determine whether there is joint node occlusion based on the joint node coordinates by dynamic confidence threshold and spatio-temporal consistency; the dynamic confidence threshold is determined based on the proportion of the occlusion area in the current frame;

[0107] a joint node correction unit configured to dynamically correct an abnormal joint node with forward prediction and backward correction according to the object to which the abnormal joint node belongs, for the abnormal joint node with continuous occlusion;

[0108] a trajectory smoothing processing unit configured to perform trajectory smoothing processing on the joint nodes based on the joint node coordinates in the sliding window to obtain the joint node coordinate sequence corresponding to the target video stream.

[0109] In some specific embodiments, the joint node time sequence sub-network includes an input layer, a multi-head attention mechanism module and a time sequence convolution module; and the feature extraction module can specifically include:

[0110] an encoding mapping unit configured to map the joint node coordinate sequence into multi-dimensional position encoding by the input layer;

[0111] a motion pattern learning unit configured to learn motion patterns of different limb parts based on the multi-dimensional position encoding and using the multi-head attention mechanism module;

[0112] a spatio-temporal feature determination unit configured to extract motion cycle features corresponding to the joint coordinate sequence using the temporal convolution module in combination with the motion patterns of the different limb parts, to obtain joint spatio-temporal features.

[0113] In some embodiments, the image semantic subnetwork includes a lightweight convolutional neural network, a region focusing module, a deformable convolution module, and a multi-scale feature fusion module. The feature extraction module can specifically include:

[0114] an initial feature map extraction unit configured to extract initial feature maps of different scales of the target video stream by using the lightweight convolutional neural network;

[0115] a feature map optimization unit configured to determine key regions in the feature maps by using the region focusing module, extract complex features of the key regions by using the deformable convolution module, and optimize the initial feature maps according to the load features;

[0116] an image semantic feature determination unit configured to fuse the optimized feature maps of different scales by using the multi-scale feature fusion module, to obtain image semantic features corresponding to the target video stream.

[0117] In some embodiments, the video rule violation action recognition device can specifically include:

[0118] an evaluation unit configured to determine motion intensity according to joint velocity and semantic abnormality according to differences between image semantic features and normal sample features before feature fusion using a gated fusion mechanism to obtain the fused features based on the joint spatio-temporal features, the image semantic features, the interaction features, the cross features, and the dynamic attention weight.

[0119] a basic attention weight determination unit configured to determine a basic attention weight based on the motion intensity and the semantic abnormality;

[0120] a dynamic attention weight determination unit configured to determine whether a prop exists in a video frame, and if so, adjust the basic attention weight by a preset increase amplitude to obtain a dynamic attention weight.

[0121] In some embodiments, the feature fusion module can specifically include:

[0122] The feature interaction enhancement unit is configured to learn interaction features of the node spatio-temporal features and the image semantic features by using a feature interaction enhancement module, and learn cross features of the node spatio-temporal features and the image semantic features by using a cross attention module.

[0123] The feature fusion unit is configured to perform feature fusion by using a gated fusion mechanism based on the node spatio-temporal features, the image semantic features, the interaction features, the cross features, and the dynamic attention weights to obtain the fused features.

[0124] In some specific embodiments, the video rule violation action recognition device can specifically include:

[0125] The treatment execution unit is configured to execute a corresponding treatment strategy according to a confidence of a rule violation action recognition result and a corresponding relationship between the confidence and the treatment strategy after rule violation action recognition based on the fused features.

[0126] The display unit is configured to display a video stream containing node motion trajectories and an enlarged view of a rule violation action region through a human-computer interaction interface.

[0127] The risk labeling unit is configured to label rule violation action risk values of the video at different time periods by using a risk heat map.

[0128] In some specific embodiments, the dual-stream feature modeling network is configured by knowledge distillation and adaptive feature selection; the adaptive feature selection configuration includes importance score configuration and background feature truncation configuration.

[0129] The dual-stream feature modeling network evaluates the importance of each channel in the network for a current video frame according to the importance score configuration, and filters out non-key convolution channels and closes them according to the importance.

[0130] The dual-stream feature modeling network identifies a foreground region and a background region of a video frame according to the background feature truncation configuration, and performs feature truncation processing on the background region to ensure the calculation resources for the foreground region.

[0131] Further, the embodiments of the present application also disclose an electronic device, referring to Figure 7 The content in the figure cannot be considered as any limitation on the use range of the present application.

[0132] Figure 7A structural schematic diagram of an electronic device 20 is provided in the embodiments of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is configured to store a computer program, and the processor 21 is configured to load and execute the computer program to implement the related steps in the method for identifying a rule violation action in a video disclosed in any of the foregoing embodiments.

[0133] In the embodiments, the power supply 23 is configured to provide working voltage for each hardware device on the electronic device 20, the communication interface 24 is capable of creating a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not specifically limited herein. The input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not specifically limited herein.

[0134] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon include an operating system 221, a computer program 222 and data 223 including the node coordinate sequence, etc., and the storage mode can be temporary storage or permanent storage.

[0135] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, so as to implement the operation and processing of the processor 21 on the mass data 223 in the memory 22, and the operating system 221 can be Windows Server, Netware, Unix, Linux, etc. The computer program 222 can further include computer programs for completing other specific work in addition to the computer programs for completing the method for identifying a rule violation action in a video executed by the electronic device 20 disclosed in any of the foregoing embodiments.

[0136] Further, the embodiments of the present application further disclose a computer storage medium, and the computer storage medium stores computer executable instructions. When the computer executable instructions are loaded and executed by a processor, the steps of the method for identifying a rule violation action in a video disclosed in any of the foregoing embodiments are implemented.

[0137] Further, the embodiments of the present application further disclose a computer program product, and the computer program product includes a computer program. When the computer program is executed by a processor, the steps of the method for identifying a rule violation action in a video disclosed in any of the foregoing embodiments are implemented.

[0138] The various embodiments described in this specification are presented for the purpose of illustration and description. Each of the embodiments highlights a different aspect of the application. The embodiments are not mutually exclusive, and some of the embodiments can be combined. The same or similar elements are denoted by the same or similar reference signs.

[0139] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The storage medium can be tangible and a tangible storage medium can be used in the implementation of the application.

[0140] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but rather are used to distinguish one element from another, and do not imply singular or plural. Moreover, the terms "include", "have", and the like, are intended to encompass non-exclusive inclusions, such that processes, methods, articles, or apparatuses that include a series of elements are not required to include only those elements specifically listed, but can include other elements not expressly listed or inherent to such processes, methods, articles, or apparatuses. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0141] The above describes in detail a method, device, and storage medium for identifying a violation action in a video. The principles and implementation modes of the application are described using specific examples. The above description of the embodiments is only intended to help understand the method and core idea of the application. For those skilled in the art, the specific implementation mode and application range can be changed according to the idea of the application. In summary, the content of the specification should not be understood as limiting the application.

Claims

1. A method for identifying inappropriate actions in a video, characterized in that, include: The target video stream is acquired, and joint point detection is performed on the target video stream to determine the joint point coordinates of each video frame. A joint point coordinate sequence is generated based on the joint point coordinates. The joint coordinate sequence and the target video stream are input into a dual-stream feature modeling network. The joint coordinate sequence is extracted using a joint temporal sub-network, and the target video stream is extracted using an image semantic sub-network. Cross-attention processing is performed on the spatiotemporal features of the key points and the semantic features of the image to obtain fused features; Identification of illegal actions is performed based on the fused features.

2. The method for identifying illegal actions in videos according to claim 1, characterized in that, The step of performing joint point detection on the target video stream to determine the joint point coordinates of each video frame includes: Cross-frame object matching of the target video stream is performed based on the similarity between motion trajectory and appearance features; For each matched object, the joint coordinates of the object in the video frame are determined by joint detection.

3. The method for identifying illegal actions in videos according to claim 1, characterized in that, The generation of the joint coordinate sequence based on the joint coordinates includes: Based on the keypoint coordinates, a dynamic confidence threshold and spatiotemporal consistency are used to determine whether keypoint occlusion exists; the dynamic confidence threshold is determined based on the proportion of the occlusion area in the current frame; For abnormal joints with continuous occlusion, the abnormal joints are dynamically corrected using forward prediction and backward correction based on the object to which they belong. The trajectory of the joints is smoothed based on the joint coordinates within the sliding window to obtain the joint coordinate sequence corresponding to the target video stream.

4. The method for identifying illegal actions in videos according to claim 1, characterized in that, The joint point temporal sub-network includes an input layer, a multi-head attention mechanism module, and a temporal convolution module; the extraction of joint point spatiotemporal features from the joint point coordinate sequence using the joint point temporal sub-network includes: The input layer maps the key point coordinate sequence into a multi-dimensional position code. Based on the multi-dimensional position encoding, the multi-head attention mechanism module is used to learn the movement patterns of different limb parts; By combining the movement patterns of the different limb parts, the temporal convolution module is used to extract the motion cycle features corresponding to the joint coordinate sequence to obtain the spatiotemporal features of the joints.

5. The method for identifying illegal actions in videos according to claim 1, characterized in that, The image semantic subnetwork includes a lightweight convolutional neural network, a region focusing module, a deformable convolutional module, and a multi-scale feature fusion module; The step of extracting image semantic features of the target video stream using an image semantic subnetwork includes: The lightweight convolutional neural network is used to extract initial feature maps of different scales from the target video stream; The region focusing module determines the key regions in the feature map, and the deformable convolution module extracts the complex features of the key regions. The initial feature map is then optimized based on the load features. The multi-scale feature fusion module is used to fuse the optimized feature maps at different scales to obtain the image semantic features corresponding to the target video stream.

6. The method for identifying illegal actions in videos according to claim 1, characterized in that, The cross-attention processing of the spatiotemporal features of the key points and the semantic features of the image to obtain the fused features includes: The interaction features between the spatiotemporal features of the key points and the semantic features of the image are learned using the feature interaction enhancement module, and the cross-attention module is used to learn the cross features between the spatiotemporal features of the key points and the semantic features of the image. Based on the spatiotemporal features of the key points, the semantic features of the images, the interaction features, the cross-diagnostic features, and the dynamic attention weights, the fused features are obtained by using a gating fusion mechanism.

7. The method for identifying illegal actions in videos according to claim 6, characterized in that, Before obtaining the fused features by performing feature fusion using a gating fusion mechanism based on the spatiotemporal features of the key points, the semantic features of the image, the interaction features, the cross-diagnostic features, and the dynamic attention weights, the following steps are included: The intensity of motion is determined by the speed of the joints, and the degree of semantic anomaly is determined by the difference between the semantic features of the image and the features of normal samples. The basic attention weights are determined based on the intensity of the motion and the degree of semantic anomaly. Determine whether there are props in the video frame. If so, adjust the basic attention weights according to a preset increase to obtain dynamic attention weights.

8. The method for identifying illegal actions in videos according to claim 1, characterized in that, After identifying the violation based on the fused features, the method further includes: Based on the confidence level of the violation identification results and the correspondence between the confidence level and the handling strategy, the corresponding handling strategy is implemented; The video stream containing the motion trajectory of the joints is displayed through a human-computer interaction interface, as well as a magnified view of the area of ​​violation. The risk heatmap is used to indicate the risk value of the violations in the video at different time periods.

9. The method for identifying illegal actions in videos according to any one of claims 1 to 8, characterized in that, The dual-stream feature modeling network is configured with knowledge distillation and adaptive feature selection; the adaptive feature selection configuration includes an importance scoring configuration and a background feature truncation configuration. The dual-stream feature modeling network evaluates the importance of each channel in the network for the current video frame based on the importance score configuration, and filters out non-critical convolutional channels and turns them off based on the importance. The dual-stream feature modeling network identifies the foreground and background regions of a video frame based on the background feature truncation configuration, and performs feature truncation processing on the background region to ensure the computational resources available for the foreground region.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the video violation action recognition method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein the computer programs, when executed by a processor, implement the video violation action recognition method as described in any one of claims 1 to 9.

12. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements the video violation action recognition method as described in any one of claims 1 to 9.

Images