Intelligent bad behavior auditing method based on attitude estimation and time sequence prediction

Abstract

The invention relates to the technical field of intelligent auditing, and discloses an intelligent bad behavior auditing method based on attitude estimation and time sequence prediction, which comprises the following steps of: S1, preprocessing an input image: carrying out resolution adjustment, RGB (Red, Green, Blue) color channel conversion and pixel normalization on the image; s2, improved human body target detection and posture estimation: recognizing and positioning a human body, and detecting key points of a human body posture; s3, target association: generating a corresponding human body motion track key point sequence; s4, human body motion trail key point sequence interpolation: complementing a missing human body key point sequence; s5, time sequence behavior prediction: identifying whether the action sequence is a bad behavior; and S6, carrying out bad behavior aggregation: fusing prediction results of the space-time diagram convolutional network.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent auditing, and in particular to an intelligent auditing method for bad behavior based on posture estimation and time series prediction. Background Art

[0002] With the rapid development of the Internet, a large amount of bad content such as those that violate public order and morality has appeared on the Internet, which has a negative impact on young people and children. Therefore, it is necessary to review Internet content. However, traditional image and video review usually requires manual review to identify various violations, especially to determine whether the video contains bad behavior, and then take corresponding measures such as warnings. However, this traditional method has many problems: (1) It is time-consuming and labor-intensive. It requires multiple workers to watch the video at the same time. When the video clip is too long, long-term viewing of the video can easily cause fatigue of the workers. Moreover, when there are many users and many videos, it will waste a lot of manpower and material resources. (2) It is less effective. When users upload a large number of videos, it takes a long time for users to get the review results, which leads to poor user experience. (3) It is difficult to handle concurrency. When a large number of users provide a large number of videos during special periods, the number of reviewers cannot be expanded in time, which will lead to a huge workload for the reviewers, and it is difficult to supplement the relevant reviewers in a short time. Therefore, these problems urgently need to be handled by new technologies.

[0003] With the continuous advancement of science and technology, intelligent auditing, as a key security measure, has been widely used in various network spaces. This intelligent detection technology does not require human intervention. It analyzes and processes bad behaviors in surveillance videos by applying computer vision technology, thereby realizing automatic detection and reporting of bad behaviors. Therefore, there are bad behavior recognition methods based on posture estimation and bad behavior recognition based on time series prediction. Among them, the bad behavior recognition method based on posture estimation can effectively extract the posture features of the human body, but this method lacks the extraction of time series information, resulting in its low recognition accuracy and easy false detection and repeated detection. The method based on time series prediction can effectively capture the time series features of the image and has good recognition performance for detection scenes with prominent targets. However, in security scenes, the background of video surveillance images is often very complex and the human targets are relatively small, resulting in the inability of the method based on time series prediction to extract the time series features of moving targets well.

[0004] A Chinese patent discloses an edge node content review and filtering system and method (Announcement No. CN112364216A). This patented technology uses edge nodes to perform content review and filtering, improves review speed, reduces hardware requirements, reduces bandwidth requirements, and meets user needs in a timely manner. Compared with previous content review and filtering systems, it has great advantages; however, its detection accuracy of bad behavior is not high, which can easily lead to false detection and missed detection. Summary of the invention

[0005] The purpose of the present invention is to provide an intelligent audit method for bad behavior based on posture estimation and timing prediction to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] An intelligent audit method for bad behavior based on posture estimation and time series prediction includes the following steps:

[0008] S1. Input image preprocessing: convert the video to be detected into a sequence of images, and then use the proportional bilinear interpolation algorithm to unify the input RGB three-channel image to a fixed size; then convert the RGB three-channel image to a BGR image; and normalize the pixels of the image;

[0009] S2. Improved human target detection and posture estimation: Use the improved YOLOv8 human target detection and key point estimation model to identify human targets in the input image, locate the largest outer rectangular box of the human target area, and detect the key point coordinate information of the human posture;

[0010] S3, target association: For the detected human key points, use the multi-target IOU matching algorithm to perform trajectory matching on the human targets detected in each frame, and assign a unique identifier to each human target in the image sequence, thereby generating a corresponding human motion trajectory key point sequence;

[0011] S4, interpolation of key point sequence of human motion trajectory: extract the key point position of each ID corresponding to the target in the current video screen, and use the nearest neighbor interpolation algorithm to complete the missing human key point sequence where the key point information is lost;

[0012] S5. Sequential behavior prediction: Create a spatiotemporal graph for each action sequence, and classify the key point sequence actions into two categories: bad behavior and normal behavior through the spatiotemporal graph convolution algorithm, and identify whether the action sequence is a bad behavior;

[0013] S6. Bad behavior aggregation: Use the behavior aggregation algorithm to fuse the prediction results of the spatiotemporal graph convolutional network to accurately locate the time period when bad behavior occurs.

[0014] As a further solution of the present invention: in the S2 step, the improved YOLOv8 human detection and key point estimation model is an expanded spatial pyramid pooling model, which adopts grouped expanded convolution; wherein, the expanded convolution changes the receptive field by controlling the convolution kernel sampling interval on the basis of ordinary convolution, and under the same computing conditions, the expanded convolution can obtain a larger receptive field than the ordinary convolution; and the grouped convolution divides the input feature map into multiple groups, each group of features is independently convolved, and then the convolution results of each group are spliced ​​together to form an output.

[0015] As a further solution of the present invention: in the step S3, the specific steps of target association are as follows:

[0016] S31, using the human coordinate frame detected in the current frame and the tracking target frame in the existing human tracker to perform IOU calculation;

[0017] S32, setting the IOU matching threshold to c. When the IOU matching threshold between the detection frame and the tracking frame is greater than c, the match is successful, and the current frame human body coordinate frame information is added to the tracker trajectory sequence.

[0018] S33, if there are any remaining unmatched trackers, the number of tracker re-searches increases. When the number of tracker re-searches exceeds the re-search threshold, the human target tracked by the tracker has been lost or no longer appears in the picture, and the tracker is deleted;

[0019] S34. If there is a matched human body detection frame, it is determined that a new human body has entered the video screen or a human body that was not in the screen before has reappeared. At this time, a tracker is added to track the trajectory of the human body.

[0020] As a further solution of the present invention: in the step S4, the missing human body key points are supplemented by the average value of the sliding window through the nearest neighbor interpolation algorithm, and the calculation formula is as follows:

[0021] In the above formula (1), it is assumed that Frame is the missing frame in the current human key point sequence, where For the Frame No. The key point coordinates and the window size are .

[0022] As a further solution of the present invention: in the step S6, the specific steps of the behavior aggregation algorithm are as follows:

[0023] S61, perform equidistant sampling on a set of sequence images, and set the sampling interval to ;

[0024] S62, dividing a set of sequence images into Group subsequences, and The model prediction is performed on the group subsequences, the majority voting algorithm is used to make decisions, the output action category is determined, and the post-processing algorithm is used for behavior aggregation.

[0025] As a further solution of the present invention: a system implementing the method includes a base layer, a platform layer and a service layer;

[0026] The platform layer is used to build face recognition, target detection, image classification, OCR recognition, speech recognition, text classification and text intelligent proofreading;

[0027] The service layer is used to support the execution of the system's review capabilities for different business scenarios; including video review, audio review, image review, text review, template management, manual review, and sensitive word and sensitive image management.

[0028] As a further solution of the present invention: the video review is based on the comprehensive review capabilities of image, text and voice technology to accurately filter illegal content in the video, and integrates the image and text review capabilities to simultaneously review the pictures and titles in the short video, thereby improving the efficiency of video review;

[0029] The audio audit is based on deep learning algorithms and massive data accumulation, supporting accurate recognition of voice content in the presence of background noise; audio recognition is combined with voiceprint detection and text audit capabilities to effectively identify illegal voice content;

[0030] The image review provides a standard image review library and a custom image review library; wherein the standard image review library provides a training set of common sensitive images; the custom image review library provides users with the functions of custom classification, adding, deleting, enabling, disabling, and modifying sensitive images, as well as custom review character management functions;

[0031] The text audit provides a standard text audit library and a custom text audit library; wherein the standard text audit library provides a training set of common sensitive texts; the custom text audit library provides users with the functions of custom classification, adding, deleting, enabling, disabling, and modifying sensitive texts;

[0032] The sensitive word and sensitive image management supports users to enable, disable, add, delete, and modify sensitive words and sensitive images, mark violation types, and apply sensitive words or sensitive images to content review business scenarios.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention sequentially performs input image preprocessing, improved human target detection and posture estimation, target association, human motion trajectory key point sequence interpolation, time series behavior prediction and bad behavior aggregation; combining human posture information and time series information, the prediction accuracy of bad behavior recognition algorithms in complex environments can be improved, so that whether bad behavior exists can be accurately detected from continuous video image frames, thereby reducing the false detection rate of bad event reporting. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flowchart of an intelligent audit method for bad behavior based on posture estimation and time series prediction;

[0036] Figure 2 A schematic diagram of the network structure of the expanded space pyramid pooling in a method of intelligent auditing of bad behaviors based on posture estimation and time series prediction;

[0037] Figure 3 It is a flowchart of a multi-target IOU matching algorithm in an intelligent audit method for bad behavior based on posture estimation and time series prediction;

[0038] Figure 4 A schematic diagram of the system framework in a method for intelligent auditing of bad behaviors based on posture estimation and time series prediction;

[0039] Figure 5 Schematic diagram of the effect of an intelligent review method for bad behavior based on posture estimation and time series prediction on video review. DETAILED DESCRIPTION

[0040] See also Figure 1-Figure 5 In an embodiment of the present invention, an intelligent audit method for bad behavior based on posture estimation and timing prediction is provided. Figure 1 As shown, the following steps are included:

[0041] S1. Input image preprocessing: convert the video to be detected into a sequence of images, and then use the proportional bilinear interpolation algorithm to unify the input RGB three-channel image to a fixed size; then convert the RGB three-channel image to a BGR image; and normalize the image pixels; thereby solving the problem of large differences in monitoring video encoding formats and image resolutions in security scenarios to meet the requirements of model input;

[0042] S2. Improved human target detection and posture estimation: Using the improved YOLOv8 human target detection and key point estimation model, the human targets in the input image are identified, and the maximum outer rectangular box of the human target area is located. At the same time, the key point coordinate information of the human posture is detected; such as nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle, a total of 17 key points;

[0043] S3, target association: For the detected human key points, use the multi-target IOU matching algorithm to perform trajectory matching on the human targets detected in each frame, and assign a unique identifier to each human target in the image sequence, thereby generating a corresponding human motion trajectory key point sequence;

[0044] S4, interpolation of key point sequence of human motion trajectory: extract the key point position of each ID corresponding to the target in the current video screen. During the occurrence of bad behavior, due to the excessive human behavior, the target will be frequently occluded for a short time, and the motion key point sequence detected by the tracker will be missing, resulting in the inability to align the key point sequences of multiple targets in time. Therefore, at the point where the key point information is lost, the nearest neighbor interpolation algorithm is used to supplement the missing human key point sequence;

[0045] S5. Sequential behavior prediction: Create a spatiotemporal graph for each action sequence, and classify the key point sequence actions into two categories: bad behavior and normal behavior through the spatiotemporal graph convolution algorithm, and identify whether the action sequence is a bad behavior;

[0046] S6. Aggregation of bad behaviors: Since the temporal feature extraction of the spatiotemporal graph convolutional action sequence classification algorithm is based on the temporal convolutional network (TCN) model, it lacks the accuracy in predicting longer sequence actions.

[0047] Preferably, in step S2, Figure 2 As shown in the figure, the improved YOLOv8 human detection and key point estimation model is an expanded spatial pyramid pooling model, which uses grouped expanded convolution instead of maximum pooling. Under the premise of almost no increase in the number of parameters, the semantic information of each receptive field branch is retained to the greatest extent. Among them, the expanded convolution changes the receptive field by controlling the convolution kernel sampling interval on the basis of ordinary convolution. Under the same computing conditions, the expanded convolution can obtain a larger receptive field than the ordinary convolution. The grouped convolution divides the input feature map into multiple groups, each group of features is independently convolved, and then the convolution results of each group are spliced ​​together to form the output, thereby effectively reducing the number of model parameters.

[0048] Preferably, in step S3, Figure 3 As shown in the figure, the specific steps of target association are as follows:

[0049] S31, using the human coordinate frame detected in the current frame and the tracking target frame in the existing human tracker to perform IOU calculation;

[0050] S32, setting the IOU matching threshold to c. When the IOU matching threshold between the detection frame and the tracking frame is greater than c, the match is successful, and the current frame human body coordinate frame information is added to the tracker trajectory sequence.

[0051] S33, if there are any remaining unmatched trackers, the number of tracker re-searches increases. When the number of tracker re-searches exceeds the re-search threshold, the human target tracked by the tracker has been lost or no longer appears in the picture, and the tracker is deleted;

[0052] S34. If there is a matched human body detection frame, it is determined that a new human body has entered the video screen or a human body that was not in the screen before has reappeared. At this time, a tracker is added to track the trajectory of the human body.

[0053] Preferably, in step S4, the missing human body key points are completed by using the nearest neighbor interpolation algorithm on the average value of the sliding window, and the calculation formula is as follows:

[0054]

[0055] In the above formula (1), it is assumed that Frame is the missing frame in the current human key point sequence, where For the Frame No. The key point coordinates and the window size are .

[0056] Preferably, in step S6, the specific steps of the behavior aggregation algorithm are as follows:

[0057] S61, perform equidistant sampling on a set of sequence images, and set the sampling interval to ;

[0058] S62, dividing a set of sequence images into Group subsequences, and The model prediction is performed on the group subsequences, the majority voting algorithm is used to make decisions, the output action category is determined, and the post-processing algorithm is used for behavior aggregation.

[0059] Preferably, the system implementing the method includes a base layer, a platform layer and a service layer;

[0060] The basic layer system runs on the open source application container engine docker, which can adapt to hardware physical computing resources and cloud computing resources, greatly improving the deployment efficiency and adaptability of the product; and uses the object-relational database management system PgSQL to store core data, including audit templates, audit records, and statistical data; uses the remote dictionary service Redis to store cache data, which can provide better performance; and uses the search analysis engine Elasticsearch to store vector feature data;

[0061] The platform layer is used to build face recognition, object detection, image classification, OCR recognition, speech recognition, text classification, and text intelligent proofreading. It uses mainstream training frameworks such as Pytorch, TensorFlow, Caffe, and MXNet, and provides advertising detection, prohibited detection, and audio recognition to improve the adaptability and development efficiency of product model training.

[0062] The service layer is used to support the execution of the system's review capabilities for different business scenarios; including video review, audio review, image review, text review, template management, manual review, and sensitive word and image management; among them, video review is based on the comprehensive review capabilities of image, text, and voice technology to accurately filter illegal content in the video, and integrate image and text review capabilities to simultaneously review the images and titles in short videos, thereby improving video review efficiency;

[0063] Audio review is based on deep learning algorithms and massive data accumulation, and supports accurate recognition of voice content in the presence of background noise. Audio recognition combines voiceprint detection and text review capabilities to effectively identify illegal voice content, and supports multiple methods such as real-time detection of short audio, long audio, and asynchronous detection of audio streams.

[0064] Image review provides a standard image review library and a custom image review library. The standard image review library provides a training set of common sensitive images, and the applicable sensitive image detection types include prohibited, advertising, and disgusting images. The custom image review library provides users with the functions of custom classification, adding, deleting, enabling, disabling, and modifying sensitive images, as well as the management of custom review characters. It also provides a strong rule management function for blacklists and whitelists. Users can manage custom image libraries and blacklists based on their own experience and habits.

[0065] Text auditing provides a standard text auditing library and a custom text auditing library. The standard text auditing library provides a common training set of sensitive texts, and the applicable sensitive text detection types include behaviors that violate public order and good morals and hinder public interests. The custom text auditing library provides users with the functions of custom classification, adding, deleting, enabling, disabling, and modifying sensitive texts, and provides a strong rule management function for blacklists and whitelists. Users can manage custom text libraries and blacklists and whitelists based on their own experience and habits.

[0066] Sensitive word and sensitive image management supports users to enable, disable, add, delete, and modify sensitive words and sensitive images, mark violation types, and apply sensitive words or sensitive images to content review business scenarios.

[0067] In order to better illustrate the technical effect of the present invention, the following experiments are used to illustrate:

[0068] like Figure 5As shown, the method of the present invention is used to sequentially perform input image preprocessing, improved human target detection and posture estimation, target association, temporal behavior prediction and bad behavior aggregation on a piece of bad video on the Internet, and it is concluded that the video is an illegal video;

[0069] Therefore, it can be verified that the method adopted by the present invention can accurately detect whether there is any bad behavior.

[0070] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technical person familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An intelligent audit method for bad behavior based on posture estimation and time series prediction, characterized in that: The following steps are involved: S1. Input image preprocessing: convert the video to be detected into a sequence of images, and then use the proportional bilinear interpolation algorithm to unify the input RGB three-channel image to a fixed size; then convert the RGB three-channel image to a BGR image; and normalize the pixels of the image; S2. Improved human target detection and posture estimation: Use the improved YOLOv8 human target detection and key point estimation model to identify human targets in the input image, locate the largest outer rectangular box of the human target area, and detect the key point coordinate information of the human posture; S3, target association: For the detected human key points, use the multi-target IOU matching algorithm to perform trajectory matching on the human targets detected in each frame, and assign a unique identifier to each human target in the image sequence, thereby generating a corresponding human motion trajectory key point sequence; S4, interpolation of key point sequence of human motion trajectory: extract the key point position of each ID corresponding to the target in the current video screen, and use the nearest neighbor interpolation algorithm to complete the missing human key point sequence where the key point information is lost; S5. Sequential behavior prediction: Create a spatiotemporal graph for each action sequence, and classify the key point sequence actions into two categories: bad behavior and normal behavior through the spatiotemporal graph convolution algorithm, and identify whether the action sequence is a bad behavior; S6. Bad behavior aggregation: Use the behavior aggregation algorithm to fuse the prediction results of the spatiotemporal graph convolutional network to accurately locate the time period when bad behavior occurs.

2. According to claim 1, a method for intelligent auditing of bad behavior based on posture estimation and time series prediction is characterized in that: In the step S2: the improved YOLOv8 human detection and key point estimation model is an expanded spatial pyramid pooling model, which adopts grouped expanded convolution; wherein, the expanded convolution changes the receptive field by controlling the convolution kernel sampling interval on the basis of ordinary convolution. Under the same computing conditions, the expanded convolution can obtain a larger receptive field than the ordinary convolution; and the grouped convolution divides the input feature map into multiple groups, each group of features is independently convolved, and then the convolution results of each group are spliced ​​together to form an output.

3. According to claim 1, a method for intelligent auditing of bad behavior based on posture estimation and time series prediction is characterized in that: In the S3 step, the specific steps of target association are as follows: S31, using the human coordinate frame detected in the current frame and the tracking target frame in the existing human tracker to perform IOU calculation; S32, setting the IOU matching threshold to c. When the IOU matching threshold between the detection frame and the tracking frame is greater than c, the match is successful, and the human body coordinate frame information of the current frame is added to the tracker trajectory sequence. S33, if there are any remaining unmatched trackers, the number of tracker re-searches increases. When the number of tracker re-searches exceeds the re-search threshold, the human target tracked by the tracker has been lost or no longer appears in the picture, and the tracker is deleted; S34. If there is a matched human body detection frame, it is determined that a new human body has entered the video screen or a human body that was not in the screen before has reappeared. At this time, a tracker is added to track the trajectory of the human body.

4. According to claim 1, a method for intelligent auditing of bad behavior based on posture estimation and time series prediction is characterized in that: In the step S4, the missing human body key points are supplemented by the average value of the sliding window through the nearest neighbor interpolation algorithm, and the calculation formula is as follows: In the above formula (1), it is assumed that Frame is the missing frame in the current human key point sequence, where For the Frame No. The key point coordinates and the window size are .

5. According to claim 1, a method for intelligent auditing of bad behavior based on posture estimation and time series prediction is characterized in that: In step S6, the specific steps of the behavior aggregation algorithm are as follows: S61, perform equidistant sampling on a set of sequence images, and set the sampling interval to ; S62, dividing a set of sequence images into Group subsequences, and The model prediction is performed on the group subsequences, the majority voting algorithm is used to make decisions, the output action category is determined, and the post-processing algorithm is used for behavior aggregation.

6. The method for intelligent auditing of bad behavior based on posture estimation and time series prediction according to any one of claims 1 to 5, characterized in that: The system implementing the method includes a base layer, a platform layer and a service layer; The platform layer is used to build face recognition, object detection, image classification, OCR recognition, speech recognition, text classification and text intelligent proofreading; The service layer is used to support the execution of the system's review capabilities for different business scenarios; including video review, audio review, image review, text review, template management, manual review, and sensitive word and sensitive image management.

7. According to claim 6, a method for intelligent auditing of bad behavior based on posture estimation and time series prediction is characterized in that: The video review is based on the comprehensive review capabilities of image, text and voice technology to accurately filter illegal content in the video, and integrate image and text review capabilities to simultaneously review the pictures and titles in the short video to improve the efficiency of video review; The audio audit is based on deep learning algorithms and massive data accumulation, supporting accurate recognition of voice content in the presence of background noise; audio recognition is combined with voiceprint detection and text audit capabilities to effectively identify illegal voice content; The image review provides a standard image review library and a custom image review library; wherein the standard image review library provides a training set of common sensitive images; the custom image review library provides users with the functions of custom classification, adding, deleting, enabling, disabling, and modifying sensitive images, as well as custom review character management functions; The text audit provides a standard text audit library and a custom text audit library; wherein the standard text audit library provides a training set of common sensitive texts; the custom text audit library provides users with the functions of custom classification, adding, deleting, enabling, disabling, and modifying sensitive texts; The sensitive word and sensitive image management supports users to enable, disable, add, delete, and modify sensitive words and sensitive images, mark violation types, and apply sensitive words or sensitive images to content review business scenarios.

Images