Multi-person behavior recognition method and system based on posture estimation and binary classification

Abstract

The present invention relates to a multi-person behavior recognition method and system based on posture estimation and dual classification, which belongs to the field of deep learning and behavior recognition technology, and particularly relates to a behavior recognition method performed by an intelligent device, which solves the problem that the prior art is difficult to obtain accurate target positions and their joint point coordinates. The present invention includes preprocessing and labeling video image data sets; extracting external features of images based on an improved OpenPose network model; extracting skeleton key points from the external features of images; predicting and supplementing missing skeleton key points; extracting spatial features of skeleton key points; integrating human posture features of continuous frame images into a feature matrix, and then classifying and visualizing the feature matrix based on dual classification. The present invention is used for real-time recognition of multi-person interaction scenes.

Description

Technical Field

[0001] A multi-person behavior recognition method and system based on posture estimation and dual classification, which are used to recognize multi-person interaction scenes in real time, belong to the technical field of deep learning and behavior recognition, and particularly relate to a behavior recognition method performed by an intelligent device. Background Art

[0002] Human behavior recognition technology is a research hotspot in the field of computer vision. Computers analyze images to identify the ongoing behaviors of targets in different scenarios, and obtain more abundant information through the specific scenarios. This technology has a wide range of applications in public intelligent video surveillance, patient monitoring systems, smart homes, and smart security. At present, single-person behavior recognition systems have been rapidly developed and widely used, but in real life scenarios, multi-person scenarios are more common, such as large gatherings in public places, group performances, etc., so there is an urgent practical application for multi-person behavior recognition technology.

[0003] However, there is still room for improvement in the recognition of multi-person behaviors. The steps are cumbersome, the efficiency is low, the accuracy is low, and the system is imperfect. These problems have always been solved. At present, most multi-person behavior recognition systems use multi-person detection methods, and use target detection algorithms such as the YOLO series detection algorithm to mark and detect targets in videos or images, determine the target position and the number of targets, and then perform target segmentation processing. A top-down approach is used to extract target key points, and on this basis, external features and local features of the image are extracted respectively. Finally, classification and recognition are performed by analyzing the features. However, this method is easily affected by the detection results, and the entire model is carried out in a multi-stage serial manner, which is inefficient and difficult to transplant and deploy. The amount of calculation will increase exponentially with the increase in the number of targets in the field of view, which seriously affects the calculation efficiency and may even cause memory explosion. The detection of the number of people and the recognition of a single person belong to serial calculations, which greatly reduces the operating efficiency. The failure to handle the false detection and missed detection in a timely manner has a great impact on the subsequent recognition processing. In order to solve the problem of excessive data volume, the image field of view can be reduced by cropping the picture. However, it is difficult to obtain the accurate target position and its joint coordinates when there are multiple targets in the video or image sequence. This situation is more suitable for single-person behavior recognition. Based on images with fixed pixel size, the movement of multiple people in the same field of view will not reduce the amount of calculation much.

[0004] In summary, the prior art has the following technical problems:

[0005] 1. It is difficult to obtain the accurate target position and its joint point coordinates;

[0006] 2. The amount of calculation is too large, which requires high hardware resources and seriously affects the calculation efficiency. Summary of the invention

[0007] In response to the above research problems, the purpose of the present invention is to provide a multi-person behavior recognition method and system based on posture estimation and dual classification to solve the problem that it is difficult to obtain accurate target positions and their joint point coordinates in the prior art.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A multi-person behavior recognition method based on posture estimation and dual classification includes the following steps:

[0010] Step 1: Obtain a video image dataset, and preprocess and annotate the videos in the video image dataset to obtain an annotated image frame dataset, wherein the annotation is an action category annotation;

[0011] Step 2: Use the improved OpenPose network model after label correction to extract external features of the images in the labeled image frame dataset to obtain external features, where the external features include human face, hands, limbs, and body trunk;

[0012] Step 3: Based on the improved OpenPose network model, the skeleton key points of each image are extracted in a bottom-up manner and represented in the form of tuples, which include the coordinates and visible status of the skeleton key points;

[0013] Step 4: Based on the key points of two adjacent frames and the prediction and supplement calculation method, the missing skeleton key points are predicted and supplemented. After the prediction and supplement, all the skeleton key points are obtained, and the head key points are removed to obtain the filtered skeleton key points;

[0014] Step 5: Extracting the spatial features of the human skeleton key points based on the filtered human spatial coordinates of the skeleton key points, wherein the spatial features refer to the human posture features;

[0015] Step 6: Integrate the human body posture features of continuous frame images into a feature matrix, and then classify and visualize the feature matrix based on binary classification.

[0016] Furthermore, the specific steps of preprocessing the videos in the video image dataset in step 1 are:

[0017] The FFmpeg tool is used to convert the videos in the video image dataset into video data with a fixed frame rate, that is, into image frames;

[0018] The image frames are then filtered and denoised in turn to obtain a preprocessed image frame data set.

[0019] Furthermore, the OpenPose network model in step 2 is a VGG-19 network, including a skeleton key point prediction branch and a skeleton key point connection branch;

[0020] The improved OpenPose network model is as follows:

[0021] The score value of the skeleton key point prediction branch of the OpenPose network model directly takes the larger value between the true value and the predicted value, and the score value of the skeleton key point connection branch takes the larger absolute value between the true value and the predicted value. The calculation formula is:

[0022]

[0023]

[0024] Among them, L i (p) represents the score value of the skeleton key point prediction branch of the i-th frame image, Represents the true value of the skeleton key point prediction branch of the i-th frame image, represents the prediction value of the skeleton key point prediction branch of the i-th frame image, D i (p) represents the score value of the skeleton key point connection branch of the i-th frame image, Represents the true value of the skeleton key point connection branch of the i-th frame image, Represents the predicted value of the skeleton key point connection branch of the i-th frame image, gt is the true value label, t is the predicted value label, gt and t are interpreted as a whole with the variable L or D.

[0025] Further, the calculation formula for the predicted supplement in step 4 is:

[0026] (x i ,y i )=((x i-1 +x i+1 ) / 2,(y i-1 +y i+1 ) / 2)

[0027] Among them, x i-1 ,y i-1 Indicates the coordinates of the bone key points in the predicted supplementary i-th frame image corresponding to the bone key points in the previous frame image position, x i+1 ,y i+1 Indicates the coordinates of the skeleton key points in the i-th frame image corresponding to the position of the skeleton key points in the next frame image.

[0028] Further, the specific steps of step 5 are:

[0029] Calculate the distance difference between the skeleton key points of each target in each frame image and the center point of the skeleton key points of the human neck, that is, the offset, and then extract the human posture features, that is, the spatial features, based on the offset of the images between adjacent frames. The extraction formula is:

[0030] (x offset ,y offset )=(x m -x center ,y m -y center )

[0031] Among them, x center ,y center The coordinate position of the center point of the key bone points of the neck is calculated by the posture estimation algorithm, x i ,y i Represents the coordinates of the mth bone key point, x offset ,y offset Indicates the calculated offset.

[0032] Further, the specific steps of step 6 are:

[0033] The feature matrix is ​​classified and visualized based on the method combining support vector machine and Bayesian network, specifically:

[0034] The extracted feature matrix is ​​input into the support vector machine to obtain the output result, and then the output result is sent to the trained Bayesian network to obtain the category classification;

[0035] After category classification, the probability value of each behavior posture is obtained, and the probability values ​​are sorted in descending order. The top-1 probability value, that is, the value with the highest probability value, is selected as the final classification and visualized.

[0036] A multi-person behavior recognition system based on posture estimation and binary classification, comprising:

[0037] Data processing module: obtains a video image data set, and preprocesses and annotates the videos in the video image data set to obtain an annotated image frame data set, wherein the annotation is an action category annotation;

[0038] External feature extraction module: The OpenPose network model improved by label correction is used to extract external features of the images in the labeled image frame dataset to obtain external features, where the external features include human face, hands, limbs, and body trunk;

[0039] Key point extraction module: Based on the improved OpenPose network model, the skeleton key points are extracted from the external features of each image in a bottom-up manner and represented in the form of tuples, which include the coordinates and visible status of the skeleton key points;

[0040] Prediction and supplement module: predict and supplement the missing skeleton key points based on the key points of two adjacent frames and the prediction and supplement calculation method. After prediction and supplement, all skeleton key points are obtained, and the head key points are removed to obtain the filtered skeleton key points.

[0041] Spatial feature extraction module: extracts the spatial features of human skeleton key points based on the filtered skeleton key points’ spatial coordinates, where the spatial features refer to human posture features;

[0042] Classification and visualization operation module: Integrate the human posture features of continuous frame images into a feature matrix, and then classify and visualize the feature matrix based on binary classification.

[0043] Furthermore, the specific steps of preprocessing the video in the video image data set in the data processing module are:

[0044] The FFmpeg tool is used to convert the videos in the video image dataset into video data with a fixed frame rate, that is, into image frames;

[0045] The image frames are then filtered and denoised in turn to obtain a preprocessed image frame data set.

[0046] Furthermore, the OpenPose network model in the external feature extraction module is a VGG-19 network, including a skeleton key point prediction branch and a skeleton key point connection branch;

[0047] The improved OpenPose network model is as follows:

[0048] The score value of the skeleton key point prediction branch of the OpenPose network model directly takes the larger value between the true value and the predicted value, and the score value of the skeleton key point connection branch takes the larger absolute value between the true value and the predicted value. The calculation formula is:

[0049]

[0050]

[0051] Among them, L i (p) represents the score value of the skeleton key point prediction branch of the i-th frame image, Represents the true value of the skeleton key point prediction branch of the i-th frame image, represents the prediction value of the skeleton key point prediction branch of the i-th frame image, Di (p) represents the score value of the skeleton key point connection branch of the i-th frame image, Represents the true value of the skeleton key point connection branch of the i-th frame image, Represents the predicted value of the skeleton key point connection branch of the i-th frame image, gt is the true value label, t is the predicted value label, gt and t are interpreted as a whole with the variable L or D.

[0052] Furthermore, the calculation formula of the prediction supplement in the prediction supplement module is:

[0053] (x i ,y i )=((x i-1 +x i+1 ) / 2,(y i-1 +y i+1 ) / 2)

[0054] Among them, x i-1 ,y i-1 Indicates the coordinates of the bone key points in the predicted supplementary i-th frame image corresponding to the bone key points in the previous frame image position, x i+1 ,y i+1 Indicates the coordinates of the skeleton key points in the i-th frame image corresponding to the position of the skeleton key points in the next frame image;

[0055] The specific implementation steps of the spatial feature extraction module are:

[0056] Calculate the distance difference between the skeleton key points of each target in each frame image and the center point of the skeleton key points of the human neck, that is, the offset, and then extract the human posture features, that is, the spatial features, based on the offset of the images between adjacent frames. The extraction formula is:

[0057] (x offset ,y offset )=(x m -x center ,y m -y center )

[0058] Among them, x center ,y center The coordinate position of the center point of the key bone points of the neck is calculated by the posture estimation algorithm, x i ,y i Represents the coordinates of the mth bone key point, x offset ,y offset Indicates the calculated offset;

[0059] The specific implementation steps of the classification and visualization operation module are:

[0060] The feature matrix is ​​classified and visualized based on the method combining support vector machine and Bayesian network, specifically:

[0061] The extracted feature matrix is ​​input into the support vector machine to obtain the output result, and then the output result is sent to the trained Bayesian network to obtain the category classification;

[0062] After category classification, the probability value of each behavior posture is obtained, and the probability values ​​are sorted in descending order. The top-1 probability value, that is, the value with the highest probability value, is selected as the final classification and visualized.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] 1. The present invention performs image preprocessing on the video, ensures the validity of the data and ensures the integrity of the data features in the image data by filtering, ensures the continuous judgement of multi-person behaviors in continuous frames, and retains key frames for certain action values ​​to reduce the amount of subsequent calculations and ensure efficiency;

[0065] 2. The present invention proposes to detect and extract key points of human skeletons in a bottom-up manner based on posture estimation, and then to comb and optimize, that is, to extract key points of skeletons based on the improved OpenPose network model, thereby improving the accuracy of key point extraction, ensuring that the behavior differences of the same target in consecutive frames are small, and no unrelated situations will occur, and supplementing the key points of skeletons that fail to be extracted due to missed detection or wrong detection, which can effectively provide more key information for subsequent human feature extraction, especially the positions of key points of skeletons of some human legs and hands, which may cause a large number of observable actions to be lost, thereby causing misjudgment. Therefore, the present invention effectively improves the model accuracy by pre-estimating the missing skeleton information;

[0066] 3. The present invention adopts a feature description method to describe the human body posture features. By calculating the offset between the positions of different skeletal key points and the center of the skeletal key points in the neck, and calculating the skeletal offset speed through the offset, it avoids obtaining features from the texture in the image, but directly calculates from the skeletal key points, which can improve processing efficiency and ensure a certain degree of real-time performance. In summary, the multi-person behavior recognition system in the present invention is an end-to-end encapsulation program from image input to final category classification, which can be convenient for practical application. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a schematic diagram of the overall framework of the present invention;

[0068] Figure 2 It is a schematic diagram of displaying the classification results on the interface in the present invention. DETAILED DESCRIPTION

[0069] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0070] The present invention proposes a multi-person behavior recognition method based on posture estimation and binary classification, which retains key video information while reducing the number of video frames and gaining key skeleton positions, aiming to ensure the accuracy and real-time performance of neural network prediction of video behavior.

[0071] The main process of the present invention includes: 1) preprocessing and annotating the video image data set; 2) extracting external features of the image based on the improved OpenPose network model; 3) extracting skeleton key points from the external features of the image; 4) predicting and supplementing the missing skeleton key points; 5) extracting the spatial features of the skeleton key points; 6) integrating the human body posture features of continuous frame images into a feature matrix, and then classifying and visualizing the feature matrix based on dual classification. The specific implementation steps are as follows:

[0072] 1. Preprocessing and labeling of video image datasets

[0073] The present invention processes images in a video image data set by calling a python-based FFmpeg library. First, the video is uniformly processed into a playback speed of 25fps, and RGB images of fixed-interval frames of the video are obtained. For some actions that take a long time to complete, frames with a larger image frame interval length can be selected. For example, walking behavior has a relatively simple action, and it is not necessary to collect every frame of the image. Image frames with a certain amount of data can be spaced apart, which can reduce a certain amount of calculation and reduce data redundancy. Some video segments have noise added to the video due to the unstable camera data stream. Therefore, in order to ensure better recognition of the target human body's actions in the image, filtering and noise reduction are used for denoising, and the corresponding image frame data set is labeled after denoising.

[0074] 2. Extract external features of images based on the improved OpenPose network model

[0075] Improvements have been made to the label correction problem in the OpenPose network model (here, a pre-trained deep neural network model), and the appearance features of multiple targets in the image are extracted to classify subsequent behavior categories. The pre-trained deep neural network model already has a good image feature extraction capability, and only needs to be fine-tuned for a specific task to obtain a model that meets the task requirements. The present invention uses the VGG-19 network to build a deep neural network model to extract features, and can easily and quickly obtain a pre-trained model and fine-tune model parameters. That is, when the VGG-19 network is iteratively trained, the external features extracted are used as the input of two branch convolutional layers, which are respectively a skeleton key point prediction branch and a skeleton key point connection branch, and the skeleton key point prediction branch predicts the skeleton key points, and the skeleton key point connection branch connects the skeleton key points. Finally, the image features, the skeleton point positions, and the connection information between the skeletons are connected and sent to the next stage of training, and the skeleton key points of the human body are repeatedly predicted through t stages. However, some visible key point areas and occluded parts of the image are not labeled with key points, resulting in low accuracy of feature extraction and low recognition rate. Therefore, the strategy of label supplementation and correction is adopted to improve the model, which can accelerate the convergence of the model and improve the model accuracy.

[0076] The improved OpenPose network model is as follows:

[0077] The score value of the skeleton key point prediction branch of the OpenPose network model directly takes the larger value between the true value and the predicted value, and the score value of the skeleton key point connection branch takes the larger absolute value between the true value and the predicted value. The calculation formula is:

[0078]

[0079]

[0080] Among them, L i (p) represents the score value of the skeleton key point prediction branch of the i-th frame image, Represents the true value of the skeleton key point prediction branch of the i-th frame image, represents the prediction value of the skeleton key point prediction branch of the i-th frame image, D i (p) represents the score value of the skeleton key point connection branch of the i-th frame image, Represents the true value of the skeleton key point connection branch of the i-th frame image, Represents the predicted value of the skeleton key point connection branch of the i-th frame image, gt is the true value mark, t is the predicted value mark, gt and t are interpreted as a whole with the variable L or D. 3. Extract skeleton key points from the external features of the image

[0081] The human skeleton position feature will not be affected by external environmental factors, and is the best description of the human body's ongoing actions and behaviors. The posture estimation algorithm OpenPose (the improved OpenPose network model) is used to detect the key points of the human skeleton in a bottom-up manner. It can accurately estimate the key points of the human face, hands, limbs, and body trunk. It is suitable for multi-person scenes and has strong robustness. Most of the human body's actions can be represented by the results of indirect calculations of the relative positions and relative angles of the above key points. The position information of each key point is predicted and repeatedly trained to obtain the skeleton key points of the human body in the image. The skeleton key point position information extracted by the improved OpenPose network model is represented in the form of a tuple (x, y, n), where x, x represents the coordinates of the skeleton key point, and n represents whether the skeleton key point is visible. n = 1 means visible, n = 2 means invisible, and n = 3 means it cannot be determined. At the same time, in order to facilitate subsequent operations, the human body key points are numbered from the head to the ankles for better data processing in the future.

[0082] 4. Predict and supplement missing bone key points

[0083] The skeleton key point data of different target individuals in the image are sorted and optimized. Since the extracted original skeleton key point data is relatively messy, sometimes there will be missed detection and wrong detection between adjacent data frames of the same behavior. Therefore, it is necessary to sort and optimize the key point data again, mainly to predict and supplement some missing key points, and calculate and estimate based on the detection positions of the two frames before and after. Supplementing the missing joint points can ensure the integrity of the key point information, increase the judgment factors for the subsequent feature extraction, and use more information for classification. For the key points that cannot be detected due to occlusion, 0 is added.

[0084] The calculation formula for predicted replenishment is:

[0085] (x i ,y i )=((x i-1 +x i+1 ) / 2,(y i-1 +y i+1 ) / 2)

[0086] Among them, x i-1 ,y i-1 Indicates the coordinates of the bone key points in the predicted supplementary i-th frame image corresponding to the bone key points in the previous frame image position, x i+1 ,y i+1 Indicates the coordinates of the skeleton key points in the i-th frame image corresponding to the position of the skeleton key points in the next frame image.

[0087] 5. Extract the spatial features of skeleton key points

[0088] The number of people in the image is confirmed by the center point of the skeleton key points of the neck. Since the joint points of the head have no direct relationship with the subsequent behavior recognition and will not play a key role, removing the relevant joint points of the head can reduce the amount of feature extraction calculation. The extraction of skeleton key points requires the description of human posture features. By calculating the distance difference between the skeleton key points and the center point of each target human body, and calculating the offset between adjacent data frames, the degree of change of the action can be estimated. At the same time, the angular velocity of the action can be calculated.

[0089] The formula for extracting human posture features is:

[0090] (x offset ,y offset )=(x m -x center ,y m -y center )

[0091] Among them, x center ,y center The coordinate position of the center point of the key bone points of the neck is calculated by the posture estimation algorithm (that is, calculated by the improved OpenPose network model), x i ,y i Represents the coordinates of the mth bone key point, x offset ,y offset Indicates the calculated offset;

[0092] 6. Integrate the human posture features of continuous frame images into a feature matrix, and then classify and visualize the feature matrix based on binary classification

[0093] The skeleton key point data are discrete and relatively independent. After extracting the human posture data (human posture features), the adjacent data are still discontinuous. To analyze the human body's action behavior in the image, multiple continuous image frames are needed as a basis for judgment in order to accurately classify. All the skeleton key points in a frame of image are represented as a one-dimensional array. Then, for continuous image frames, the input can be expanded into a multi-dimensional matrix, where a row represents a human posture feature extracted from a skeleton key point, and a column represents the number of frames to complete the action. After processing according to such a data structure, the continuity of a human action can be guaranteed, providing more key information for subsequent action classification and recognition.

[0094] A human action is represented as a collection of human posture features of continuous frame images. After the collection, a feature matrix is ​​obtained. The feature matrix is ​​used as input, and a combination of machine learning algorithms support vector machine and Bayesian is used as classifier training parameters to achieve multi-person action recognition. Through multiple iterations of training, the classifier has the ability to recognize multi-person actions. For any video input, the above steps can be used to process and identify the behavior category in real time. Specifically:

[0095] The extracted feature matrix is ​​input into the support vector machine to obtain the output result, and then the output result is sent to the trained Bayesian network to obtain the category classification;

[0096] After category classification, the probability value of each behavior posture is obtained, the probability values ​​are sorted in descending order, the top-1 probability value is selected as the final classification, and it is visualized.

[0097] The above are only representative embodiments of the present invention in many specific application scopes, and do not constitute any limitation on the protection scope of the present invention. Any technical solutions formed by transformation or equivalent replacement fall within the protection scope of the present invention.

Claims

1. A multi-person behavior recognition method based on posture estimation and dual classification, characterized in that: The steps include: Step 1: Obtain a video image dataset, and preprocess and annotate the videos in the video image dataset to obtain an annotated image frame dataset, wherein the annotation is an action category annotation; Step 2: Use the OpenPose network model improved by label correction to extract external features of the images in the labeled image frame dataset to obtain external features, where the external features include human face, hands, limbs, and body trunk; Step 3: Based on the improved OpenPose network model, the skeleton key points of each image are extracted in a bottom-up manner and represented in the form of tuples, which include the coordinates and visible status of the skeleton key points; Step 4: Based on the key points of two adjacent frames and the prediction and supplement calculation method, the missing skeleton key points are predicted and supplemented. After the prediction and supplement, all the skeleton key points are obtained, and the head key points are removed to obtain the filtered skeleton key points; Step 5: Extracting the spatial features of the human skeleton key points based on the filtered human spatial coordinates of the skeleton key points, wherein the spatial features refer to the human posture features; Step 6: Integrate the human body posture features of continuous frame images into a feature matrix, and then classify and visualize the feature matrix based on binary classification.

2. The method for multi-person behavior recognition based on posture estimation and dual classification according to claim 1, characterized in that: The specific steps of preprocessing the videos in the video image dataset in step 1 are: The FFmpeg tool is used to convert the videos in the video image dataset into video data with a fixed frame rate, that is, into image frames; The image frames are then filtered and denoised in sequence to obtain a preprocessed image frame data set.

3. The method for multi-person behavior recognition based on posture estimation and dual classification according to claim 2, characterized in that: The OpenPose network model in step 2 is a VGG-19 network, including a skeleton key point prediction branch and a skeleton key point connection branch; The improved OpenPose network model is as follows: The score value of the skeleton key point prediction branch of the OpenPose network model directly takes the larger value between the true value and the predicted value, and the score value of the skeleton key point connection branch takes the larger absolute value between the true value and the predicted value. The calculation formula is: in, Indicates The score value of the skeleton key point prediction branch of the frame image, Indicates The true value of the skeleton key point prediction branch of the frame image, Indicates The prediction value of the skeleton key point prediction branch of the frame image, Indicates The score value of the skeleton key point connection branch of the frame image, Indicates The true value of the skeleton key point connection branch of the frame image, Indicates The predicted value of the skeleton key point connection branch of the frame image, is the true value mark, is the predicted value identifier, and variables or Explain as a whole.

4. The method for multi-person behavior recognition based on posture estimation and dual classification according to claim 3, characterized in that: The calculation formula for the predicted supplement in step 4 is: in, , Indicates the predicted supplement The coordinates of the skeleton key points in the frame image correspond to the skeleton key points in the previous frame image position. , Indicates The skeleton key points in a frame image correspond to the coordinates of the skeleton key points at the position of the next frame image.

5. The method for multi-person behavior recognition based on posture estimation and dual classification according to claim 4, characterized in that: The specific steps of step 5 are: Calculate the distance difference between the skeleton key points of each target in each frame image and the center point of the skeleton key points of the human neck, that is, the offset, and then extract the human posture features, that is, the spatial features, based on the offset of the images between adjacent frames. The extraction formula is: in, , The coordinate position of the center point of the key bone points of the neck is calculated by the posture estimation algorithm. , Indicates The coordinates of the key points of the skeleton, Indicates the calculated offset.

6. The method for multi-person behavior recognition based on posture estimation and dual classification according to claim 5, characterized in that: The specific steps of step 6 are: The feature matrix is ​​classified and visualized based on the method combining support vector machine and Bayesian network, specifically: The extracted feature matrix is ​​input into the support vector machine to obtain the output result, and then the output result is sent to the trained Bayesian network to obtain the category classification; After category classification, the probability value of each behavior posture is obtained, and the probability values ​​are sorted in descending order. The top-1 probability value, that is, the value with the highest probability value, is selected as the final classification and visualized.

7. A multi-person behavior recognition system based on posture estimation and dual classification, characterized in that: include: Data processing module: obtains a video image data set, and preprocesses and annotates the videos in the video image data set to obtain an annotated image frame data set, wherein the annotation is an action category annotation; External feature extraction module: The OpenPose network model improved by label correction is used to extract external features of the images in the labeled image frame dataset to obtain external features, where the external features include human face, hands, limbs, and body trunk; Key point extraction module: Based on the improved OpenPose network model, the skeleton key points are extracted from the external features of each image in a bottom-up manner and represented in the form of tuples, which include the coordinates and visible status of the skeleton key points; Prediction and supplement module: predict and supplement the missing skeleton key points based on the key points of two adjacent frames and the prediction and supplement calculation method. After prediction and supplement, all skeleton key points are obtained, and the head key points are removed to obtain the filtered skeleton key points. Spatial feature extraction module: extracts the spatial features of human skeleton key points based on the filtered skeleton key points’ spatial coordinates, where the spatial features refer to human posture features; Classification and visualization operation module: Integrate the human posture features of continuous frame images into a feature matrix, and then classify and visualize the feature matrix based on binary classification.

8. The multi-person behavior recognition system based on posture estimation and dual classification according to claim 7, characterized in that: The specific steps of preprocessing the video in the video image data set in the data processing module are: The FFmpeg tool is used to convert the videos in the video image dataset into video data with a fixed frame rate, that is, into image frames; The image frames are then filtered and denoised in sequence to obtain a preprocessed image frame data set.

9. The multi-person behavior recognition system based on posture estimation and dual classification according to claim 8, characterized in that: The OpenPose network model in the external feature extraction module is a VGG-19 network, including a skeleton key point prediction branch and a skeleton key point connection branch; The improved OpenPose network model is as follows: The score value of the skeleton key point prediction branch of the OpenPose network model directly takes the larger value between the true value and the predicted value, and the score value of the skeleton key point connection branch takes the larger absolute value between the true value and the predicted value. The calculation formula is: in, Indicates The score value of the skeleton key point prediction branch of the frame image, Indicates The true value of the skeleton key point prediction branch of the frame image, Indicates The prediction value of the skeleton key point prediction branch of the frame image, Indicates The score value of the skeleton key point connection branch of the frame image, Indicates The true value of the skeleton key point connection branch of the frame image, Indicates The predicted value of the skeleton key point connection branch of the frame image, is the true value mark, is the predicted value identifier, and With variables or Explain as a whole.

10. The multi-person behavior recognition system based on posture estimation and dual classification according to claim 9, characterized in that: The calculation formula of the prediction supplement in the prediction supplement module is: in, , Indicates the predicted supplement The coordinates of the skeleton key points in the frame image correspond to the skeleton key points in the previous frame image position. , Indicates The coordinates of the skeleton key points in the frame image corresponding to the skeleton key points at the position of the next frame image; The specific implementation steps of the spatial feature extraction module are: Calculate the distance difference between the skeleton key points of each target in each frame image and the center point of the skeleton key points of the human neck, that is, the offset, and then extract the human posture features, that is, the spatial features, based on the offset of the images between adjacent frames. The extraction formula is: in, , The coordinate position of the center point of the key bone points of the neck is calculated by the posture estimation algorithm. , Indicates The coordinates of the key points of the skeleton, Indicates the calculated offset; The specific implementation steps of the classification and visualization operation module are: The feature matrix is ​​classified and visualized based on the method combining support vector machine and Bayesian network, specifically: The extracted feature matrix is ​​input into the support vector machine to obtain the output result, and then the output result is sent to the trained Bayesian network to obtain the category classification; After category classification, the probability value of each behavior posture is obtained, and the probability values ​​are sorted in descending order. The top-1 probability value, that is, the value with the highest probability value, is selected as the final classification and visualized.

Images