Multi-feature fusion human-object interaction action recognition method

Abstract

The present application relates to a kind of multi-feature fusion human-object interaction action recognition method, by extracting video sequence frame to carry out data processing.First, object recognition is carried out using the target detection algorithm based on deep learning, and the boundary box and position information of object are extracted.Then, cross fusion is carried out to skeletal data and object recognition result, and the angle feature between skeletal segment and the relative distance feature between joint are extracted according to skeletal information, and the relative distance and angle feature between object and joint are multi-feature fusion.Finally, the feature vector between human and object and the feature vector between human joints are integrated by the classification method of multilayer learning in the method of the present application, so that the action classification of video can be more accurately completed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of human-object action recognition, in particular to a multi-feature fusion human-object interaction action recognition method. BACKGROUND

[0002] Human action recognition is an important research field of computer vision, and the human-object interaction part in the action recognition plays an important role in many practical applications. However, the interactive action is different from the ordinary action recognition, which is more complex and requires more accurate features. It not only depends on the detection result of the object, but also needs to recognize the position relationship between the human and the object, so the recognition result often has the problem of inaccuracy.

[0003] The current human-object interaction recognition method generally uses deep learning. The conventional extraction method of human-object interaction in deep learning is to pair two proposals according to the target detection proposal of RPN. In recent years, although deep learning has achieved some good results in the research of interactive action, there are still some problems. On the one hand, it excessively depends on a large amount of data set, and on the other hand, it ignores the indirect feature connection between the detected targets. Moreover, for RGB images, some complex backgrounds, light, angles and other factors will still affect the target recognition result. The traditional algorithm has the problems of low recognition accuracy and complex recognition process. Therefore, it is urgent to research a multi-feature fusion human-object interaction action recognition method, so as to have more accurate judgment on the classification result. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application provides a multi-feature fusion human-object interaction action recognition method, which fuses the posture recognition network and the target detection network, extracts the feature information of human joints and objects from multiple directions in the video for feature fusion, so as to have more accurate judgment on the classification result. The method of the present application uses a target detection algorithm and a multi-layer classification learning method for object recognition and action classification, so as to more accurately describe the interaction relationship between the human and the object. The multi-layer classification learning method proposed in the present application directly produces a voting result based on the classification result of the picture, which has more advantages than the current single deep learning method in recognition effect, and has higher accuracy of classification result.

[0005] To achieve the above-mentioned purpose, the present application discloses the following technical scheme:

[0006] The present application provides a multi-feature fusion human-object interaction action recognition method, which comprises the following steps:

[0007] S1, extracting bone data video from the database, pre-processing the video sequence frame, and using a target detection algorithm based on deep learning to recognize the object and obtain the object recognition result;

[0008] S2, cross-fuse the skeleton data and the object recognition result to construct a fusion model, and the specific steps include the following steps:

[0009] S21, judge the interactive frame and the non-interactive frame, and the specific method is to define E hi , i∈[1, N] is the set of key points in the i-th frame, E hi The calculation formula is as follows:

[0010] E hi = {(x ik ,y ik ) | (x ik ,y ik ) ∈ F hi and score ik <τ},

[0011] Wherein, τ is a very small threshold value, used to judge the reliability of human joint key point extraction, if It means that there is interaction between the human and the object in the i-th frame, and the i-th frame is set as an interactive frame, and vice versa.

[0012] S22, define the object closest to a wrist joint of each frame of the interactive frame in the skeleton data video as the interactive object in the interactive frame, and represent it as B i = {(x io1 ,y io1 ,x io2 ,y io2 ,w i )}, w i is the type label of the interactive object; if all the interactive objects in the interactive frames are the same, it is considered that the video contains single-task HOI, otherwise, it is considered that the video contains multi-task HOI;

[0013] S23, network fusion and construction of fusion feature set: put the sequence frames saved in the original video into the set, extract the skeleton data and object recognition result from the starting frame, and put the sequence number of the original video frame corresponding to the extracted skeleton data and object recognition result into the set for comparison, the intersection key frame in the set is the fusion part of the skeleton data and object recognition result, and finally generate a fusion feature set based on the intersection joint frame, and the specific steps include the following steps:

[0014] S231, define the feature sets generated by the pose estimation network and the target detection network as <F human ,f object >, wherein F human = {F h1 ,F h2 ,…,Fhi}, i ∈ [1, N] and F object = {F o1 , F o2 , …, F oi}, i ∈ [1, N] are the feature sets of human joints or objects in the ith frame, assuming that c objects are detected in the ith frame, the output is defined as F oi = {O i1 , O i2 , …, O ic} and O ic = {(x ic , y ic ) | x ic ∈ [x ic1 , x ic2 ], y ic ∈ [y ic1 , y ic2 ], F oi and O ic are the upper left corner (x ic1 , y ic1 ) and the lower right corner (x ic1 , y ic1 ) of the cth box, respectively;

[0015] S232, according to the values in the interactive frames, mark the continuous frames with the same object category as sub-videos with labels from N1 to N2, N1, N2 ∈ [1, N], N1 < N2, then perform network fusion to obtain the intersection of human joints and objects <F human , F object >

[0016]

[0017]

[0018] where H om is the feature set of the sub-video extracted from the interactive frames from N1 to N2, representing the two-dimensional coordinate combination of human joints and objects after network fusion;

[0019] S233, redefine the key points and line segments in the interactive frames as {K i0 , K i1 , …, K i24 , K i25 , K i26}, i ∈ [N1, N2] and {L i0 , l i1 , …, L i24 , L i25}, where K ia = (x ia , y ia), i ∈ [0, 24] represents 25 joint coordinates, K i24 = (x io1 , y io1 ) and K i25 = (x io2 , y io2 ) are the corners B i , L i0 , L i1 , …, L i24 are the skeleton vectors, and L i25 is the target vector;

[0020] S234, the following features are extracted from human joint and object key frames:

[0021] H i = {(D iab , A iab ) | D iab = |K ia K ib |, A iab = L ia gL ib , a = 0, 1L24, b = 0, 1L24, a ≠ b}

[0022] O i = {(D iob , A iob ) | D iob = |K io K ib |, A iob = L io gL ib , o = 25, b = 0, 1L24}

[0023] wherein H i , O i represent the feature sets of human and object in spatial dimensions, D iab is the distance between two joints; A iab is the angle between two skeleton vectors;

[0024] Then a matrix integral feature is defined in the space-time dimension:

[0025]

[0026] wherein D ija is the distance between the corresponding key points in the i-th frame and the j-th frame of the interaction frame, and is represented as D ija = |K ja - K ia |, a ∈ [0, 26];

[0027] S3, extracting the feature set H of the sub-video from the N1 to N2 interactive frames obtained in step S2 om The human joint features and the object features are fused to obtain fused features.

[0028] S4, classifying the video by the multi-layer learning classification method, inputting the feature vectors between the human and the object and the feature vectors between the human joints into the multi-layer classification model for training, establishing a feedback channel, finding and outputting the highest level classifier result, and completing the action classification of the skeletal data video.

[0029] The multi-layer learning classification method is performed by using a multi-layer classification learning algorithm. The multi-layer classification learning algorithm is a method of cascading two or more classifiers to achieve higher classification accuracy. The specific process is as follows:

[0030] S41, inputting the fused features into the loop body of k-fold cross-entropy verification for training by using the multi-layer learning method, randomly dividing the original training set into k parts without repetition, selecting one part as a verification set, and using the remaining k-1 parts as a training set for model training, inputting the training data into the first layer classifier for training, further obtaining a predicted class vector, combining the predicted vector with the n-dimensional feature vector extracted from the data set as new features, inputting the combined feature vector into the second layer, the third layer, and the kth layer classifier for training, obtaining a multi-layer learning network model, inputting the verification set into the multi-layer learning network model for testing, and saving the test result as an evaluation index of the model.

[0031] S42, repeating step S41 for k times to obtain k test results.

[0032] S43, calculating the average value of the k test results as an evaluation performance index, establishing a feedback channel, and obtaining the final classification result by voting.

[0033] Preferably, the specific method of establishing a feedback channel in step S43 is to input the feature vectors between the human joints and the object and the feature vectors between the human joints into the multi-layer learning network model for multi-layer training. The feature vectors start from the second layer classifier. Each layer will obtain a classification feedback result, and the classification feedback result will be returned to the previous layer classifier and the next layer classifier respectively, until the classification result of a certain layer classifier is greater than that of the previous and next two layer classifiers. The classification result of the layer classifier with the highest accuracy is output as the final classification result.

[0034] Preferably, before judging the interactive action of the human and the object in step S2, it is first necessary to judge whether the human and the object are in contact, i.e., contact is 1 and non-contact is 0.

[0035] Preferably, in step S1, after the skeleton data is extracted from the original video, the output video is saved as a corresponding sequence frame, and the 2D coordinates of the joint points of each frame are saved to the local.

[0036] Preferably, the features in the intersection key frame include the distance of the object to the human body joint, the angle, the distance and the angle, and the distance and the angle between the human body joints.

[0037] Preferably, in step S2, the left top and right bottom two vertices of the bounding box of the object are m p , m q , m p and m q , and the spatial coordinates are:

[0038] m p =(x io1 ,y io1 ), m q =(x io2 ,y io2 )

[0039] Preferably, in step S3, the human body joint features and the object features are fused by feature addition to obtain the fused features.

[0040] Compared with the prior art, the present application has the following advantages:

[0041] (1) The present application combines deep learning and traditional algorithms to solve the speed problem and the recognition accuracy problem. The multi-layer classification learning method proposed in the present application better fuses the relative relationship between the human and the object in the actual motion interaction process, can more accurately and quickly complete motion recognition, and is a complete human-object interaction motion recognition method. Compared with the current deep learning method alone, the recognition effect is improved and more interpretable.

[0042] (2) In the present application, when the motion recognition involves the interaction between the human and the object, the interaction relationship between the human and the object must be considered according to the actual situation. In the human-object interaction process, relative position changes, angle changes, etc. will occur. Therefore, the intersection key frame is extracted for multi-feature extraction of the interactive motion. Including the distance of the object to the human body joint, the angle, the distance+angle, the distance and the angle between the human body joints. The extracted features such as angles and distances of the skeleton data are tested for motion classification, and the weighted features and the human-object features are fused, so as to increase the accuracy of feature fusion and ensure the classification precision.

[0043] (3) The application adopts a feedback channel to put the feature vectors between the human and the object and the feature vectors between the human joints into a classification model for multi-layer training. The feature vectors start from the second layer classifier, and each layer gives a classification feedback result, which is returned to the previous layer and the next layer respectively, until the classification result of a certain layer is greater than the previous and the next two layers, then the highest accuracy corresponding layer is output, so as to complete the action classification of the video by the classification model, thereby ensuring that the classification result is the classification with the highest accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 The flowchart of the application is shown in the figure;

[0045] Figure 2 The specific human joint point diagram of the application is shown in the figure;

[0046] Figure 3 The action recognition algorithm flow of human-object interaction of the application is shown in the figure;

[0047] Figure 4 The openpose extraction skeleton data process diagram of the application is shown in the figure;

[0048] Figure 5 The network cross-fusion process of the application is shown in the figure;

[0049] Figure 6 The flowchart of the feedback channel of the application is shown in the figure. DETAILED DESCRIPTION

[0050] Hereinafter, the embodiments of the application will be described with reference to the accompanying drawings.

[0051] The application provides a multi-feature fusion human-object interaction action recognition method, as shown in Figure 1 and Figure 3 , which includes the following steps:

[0052] S1, extract the skeleton data video from the database, pre-process the video sequence frames, and use the target detection algorithm based on deep learning to identify the object and obtain the object identification result; in this embodiment, after the skeleton data is extracted from the original video, the output video is saved as the corresponding sequence frame, and the 2D coordinates of each frame joint point are saved to the local, the position information of the joint point. As shown in Figure 2 The database in this step refers to the interactive image of the human body and the object photographed by the camera device, which is stored in the database and extracted directly from the database during subsequent processing.

[0053] The model adopted by the application is a 25-point model, that is, the position information of 25 joint points of the extracted skeleton data, and each joint point information includes (x, y, score) three information. The 25 joint points include:

[0054] nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, mid hip, right hip, right knee, right ankle, left hip, left knee, left knee, left ankle, right eye, left eye, right ear, left thumb, left pinky, left heel, right thumb, right pinky, right heel. The skeleton map is as shown in Figure 2 .

[0055] The original video in step S1 uses the ORGBD dataset, which is a small sample human-object interaction action dataset. The video contains seven common human-object interaction actions. For example, common actions such as drinking water, eating, using a notebook computer, making a phone call, playing a mobile phone, reading a book, and using a remote control. Preprocess the ORGBD dataset, i.e. after extracting the skeleton data by openpose, since the number of saved sequence frames of each video is 500-1000 frames, the ORGBD dataset contains 224 videos, including seven actions, performed by 16 different people twice.

[0056] In the specific implementation process, the data volume is slightly large, and further frame extraction is adopted, such as Figure 4 drinking action, frame extraction from the key frame of the interaction action of picking up the water cup, every 4 frames, because many consecutive sequence frames have no obvious change in the front and back pictures, so it is not necessary to process all frames of data.

[0057] The target detection algorithm used in this step is the pp-yolov2 detection model, which uses a pose estimation network and a lightweight target detection network. The lightweight target detection network has better detection effect on small target objects. In the saved sequence frame, 80-120 frames are selected for key frame extraction and labeling. The labeled object information is saved to an XML file named after the action sequence.

[0058] In the specific implementation process, the input size is increased to expand the target area. It is easier to capture the information of small scale targets, thereby greatly improving the detection performance. In actual operation, a larger image input size will occupy more memory, so the Batchsize is reduced from Gpu 24 pictures to 12 pictures, and the maximum input size is increased from 608 to 768.

[0059] The specific image input size is selected from [320, 352, 384, 416, 448, 480, 512, 544, 576, 608, 640, 672, 704, 736, 768]. Then the labeled image is reloaded into pp-yolov2 for training to obtain the training model.

[0060] S2, cross-fuse the skeleton data and the object recognition result to construct a fusion model, and the specific process is: put the sequence frames saved by the original video into a set, extract the skeleton data and the object recognition result from the starting frame, and put the sequence numbers of the original video frames corresponding to the extracted skeleton data and the object recognition result into a set for comparison, the intersection part in the set is the fusion part of the skeleton data and the object recognition result, and multi-feature extraction of interactive actions is performed according to the extracted intersection key frame, and the intersection key frame includes the distance, angle, distance + angle of the object to the human body joint, and the distance and angle between the joints of the human body itself.

[0061] The skeleton data and the object recognition result are cross-fused to construct a fusion model, which specifically includes the following sub-steps:

[0062] S21, interactive and non-interactive judgment, and the interactive and non-interactive judgment specifically includes: defining E hi , i ∈ [1, N] is the set of key points in the i-th frame, E hi The calculation formula is as follows:

[0063] E hi = {(x ik , y ik ) | (x ik , y ik ) ∈ F hi and score ik <τ},

[0064] Wherein, τ is a very small threshold value, used to judge the reliability of human key point extraction, if It is indicated that there is interaction between the human and the object in the i-th frame, and the i-th frame is set as an interactive frame, and vice versa.

[0065] S22, define the object closest to the left wrist joint or the right wrist joint of each frame in the video as the interactive object in the frame, and represent it as the type label of the interactive object, if all the interactive objects in the interactive frames are the same, it is considered that the video contains single-task HOI, otherwise, it is considered that the video contains multi-task HOI. Single task refers to single object recognition of a single person, and multi-task refers to multi-object recognition of a single person or multi-object recognition of multiple persons.

[0066] S23, network fusion and construction of a fusion model: put the sequence frames saved by the original video into a set, extract the skeleton data and the object recognition result from the starting frame, and put the sequence numbers of the original video frames corresponding to the extracted skeleton data and the object recognition result into a set for comparison, the intersection key frame in the set is the fusion part of the skeleton data and the object recognition result, and the specific steps include the following sub-steps:

[0067] S231, define the feature sets generated by the pose estimation network and the target detection network as <Fhuman F object}, where F human = {F h1 , F h2 ,..., F hi}, i ∈ [1, N] and F object = {F o1 , F o2 ,..., F oi}, i ∈ [1, N] are the feature sets of human joints or objects in the i-th frame, assuming that c objects are detected in the i-th frame, the output is defined as F oi = {O i1 , O i2 ,..., O ic} and O ic = {(x ic , y ic )|x ic ∈ [x ic1 , x ic2 ], y ic ∈ [y ic1 , y ic2 ], F oi and O ic represent the c-th box between the top-left corner (x ic1 , y ic1 ) and the bottom-right corner (x ic1 , y ic1 ). The pose estimation network in this step refers to extracting the human skeleton, i.e., openpose, and the object detection network refers to extracting object information, i.e., yolo.

[0068] S232, according to the values in the interactive frames, mark the consecutive frames with the same object class as sub-videos with labels from N1 to N2, N1, N2 ∈ [1, N], N1 < N2, then perform network fusion to obtain the intersection of human joints and objects <F human ,F object >

[0069]

[0070]

[0071] where H om is the feature set of the sub-video extracted from the interactive frames from N1 to N2, and represents the two-dimensional coordinate combination of human joints and objects after network fusion.

[0072] S233, in order to describe the spatio-temporal features of the interactive frames, the key points and line segments in the interactive frames are redefined as {K i0 ,K i1 ,..., K i24 ,Ki25 ,K i26}, i∈[N1,N2] and {L i0 ,L i1 ,…,L i24 ,L i25}, where K ia =(x ia ,y ia ), i∈[0,24] represents the coordinates of 25 key points, K i24 =(x io1 ,y io1 ) and K i25 =(x io2 ,y io2 ) represents the corner B of the detected bounding box. i L i0 ,L i1 ,…,L i24 L is the skeleton vector. i25 The target vector is defined. Then, the following features are extracted from the human body and objects.

[0073] H i ={(D iab A iab )|D iab =|K ia K ib |,A iab =L ia gL ib a = 0, 1L24, b = 0, 1L24, a ≠ b

[0074] O i ={(D iob A iob )|D iob =|K io K ib |,A iob =L io gL ib ,o=25,b=0,1L24}

[0075] In the formula, H i O i D represents the feature set of the human body and objects in spatial dimensions. iab It is the distance between two joints. A iab It is the angle between two skeleton vectors, where skeleton vector L refers to the line segment connecting the two key points.

[0076] A matrix integral feature is defined in the space-time dimension.

[0077]

[0078] Among them, Dija is the distance between the corresponding key points in the i-th frame and the j-th frame in the interactive frame. Denoted as D ija = |K ja - K ia |, a [0, 26].

[0079] S234, for a single task sub-video, the corresponding interactive frame ST i may be calculated. However, a large number of interactive frames may cause more computational burden, in order to solve this problem, we consider the strategy of selecting appropriate interactive frames from the entire sub-video. That is, the label set I m = {i1, i2, L, i m}, i1, i2, L, i m [ N1, N2] satisfies the condition Here, is the moving distance of the person and the object in the time dimension. After the interactive frame selection, a fusion feature set F

[0080] In this step, the left top and right bottom two vertices of the bounding box of the object in the above step are m p , m q , m p and m q , the spatial coordinates are:

[0081] m p = (x io1 , y io1 ), m q = (x io2 , y io2 )

[0082] The number of features between the body's own key points is calculated by the following formula:

[0083]

[0084] That is, the angle feature of the own key point is 300, the distance feature is 300, and the distance feature of the object to each key point. As shown in S13, the object frame extracts the coordinates of the left top corner and the right bottom corner, that is, the number of distance and angle features is 25*2=50.

[0085] In the embodiment of the application, as shown in step S21, before judging the specific action of the interaction between the person and the object, a contact judgment needs to be made first, that is, a contact two-classification algorithm, to judge whether there is interaction. Specifically, the distance of the two points of the extracted object to the 25 key points of the human body is calculated, and then the extracted distance is put into the classification algorithm for two-classification. That is, the contact is 1, and the non-contact is 0.

[0086] In this embodiment, the specific network fusion model is as shown in Figure 5 The black numbers represent the original video sequence frames, the yellow numbers represent the original video sequence frames corresponding to the extracted skeleton data part, the blue numbers represent the original video sequence frames corresponding to the extracted object information part, and the green numbers represent the fusion part of the two networks.

[0087] S3, based on the cross-fusion feature of step S2, the human joint feature and the object feature are fused, and the human joint feature and the object feature are fused by feature addition in step S3 to obtain the fusion feature.

[0088] S4, the fusion feature is used to classify the video by a multi-layer learning classification method, the feature vectors between the person and the object and the feature vectors between the human joints are input into a multi-layer classification model for training, a feedback channel is established, the highest level classifier result is found and output, and the accurate classification of the action of the video is completed.

[0089] The multi-layer learning classification method is performed by using a multi-layer classification learning algorithm, and the multi-layer classification learning algorithm is a method of using ensemble learning to cascade two or more layers of classifiers to achieve higher classification accuracy. The specific process is as follows:

[0090] S41, the fusion feature is input into the loop body of k-fold cross-entropy verification and trained by using a multi-layer learning method, the original training set is randomly divided into k parts without repetition, one part is selected as a verification set, and the remaining (k-1) parts are used as a training set for model training, the training data is put into the first layer classifier for training, a predicted class vector is further obtained, then the predicted vector is combined with the n-dimensional feature vector extracted from the data set as new features, the combined feature vector is input into the second layer and the third layer classifier for training again, a multi-layer learning network model is obtained by training, the verification set is input into the multi-layer learning network model for testing, and the evaluation index of the model is saved.

[0091] S42, repeat step S41 for k times to obtain k test results.

[0092] S43, calculate the average value of the k test results as the evaluation performance index, establish a feedback channel, and obtain the final classification result by voting.

[0093] As shown in Figure 6As shown, the specific method for establishing the feedback channel in step S43 is to put the feature vectors between the human joints and the object and the feature vectors between the human joints into the multi-layer learning network model for multi-layer training, the feature vectors start from the second layer classifier, each layer will obtain a classification feedback result, and the classification feedback result is returned to the previous layer classifier and the next layer classifier respectively, until the classification result of a certain layer classifier is greater than the classification results of the previous and next layer classifiers, then the classification result of the layer classifier corresponding to the highest accuracy is output as the final classification result.

[0094] In order to more comprehensively and accurately extract the features in the interactive action process, the application analyzes the interactive relationship between the human and the object, and the relative position change and angle change generated in the interactive process between the human and the object, and proposes the distance, angle, distance+angle of the object to the human body joint, and the distance and angle between the human joints. In the remaining embodiments, the method of the application can also be applied to feature extraction and classification of multiple actions, and the accuracy of the classification result can be ensured.

[0095] The multi-layer classification learning method proposed by the application better integrates the relative relationship between the human and the object in the actual action interactive process, can more accurately and quickly complete action recognition, is a complete character interactive action recognition method, and has improved recognition effect and better interpretability compared with the current single deep learning method.

[0096] The above-described embodiments only describe the preferred embodiments of the application, and do not limit the scope of the application, and various modifications and improvements to the technical solutions of the application made by those skilled in the art without departing from the design spirit of the application shall fall within the protection scope of the claims of the application.

Claims

1. A multi-feature fusion method for human-object interaction action recognition, characterized in that: It includes the following steps: S1. Extract skeletal data video from the database, preprocess the video sequence frames, and use a deep learning-based object detection algorithm to identify objects and obtain the object recognition results. S2. Cross-fuse the skeletal data and object recognition results to construct a fusion model, which includes the following sub-steps: S21. Perform interaction frame and non-interaction frame determination. The specific method for determining interaction frames and non-interaction frames is as follows: Define E hi Let i∈[1,N] be the set of keypoints in the i-th frame, and E hi The calculation formula is as follows: E hi ={(x ik ,y ik )|(x ik ,y ik )∈F hi and score ik <τ}, Where τ is a very small threshold used to determine the reliability of human joint key point extraction; if If there is interaction between the person and the object in the i-th frame, then the i-th frame is set as an interactive frame; otherwise, it is a non-interactive frame. S22. Define the object closest to a wrist joint in each interactive frame of the skeletal data video as the interactive object in that interactive frame, denoted as B. i ={(x io1 y io1 x io2 y io2 w i )},w i The type label for the interactive object; if the interactive object is the same in all interactive frames, the video is considered to contain a single-task HOI; otherwise, the video is considered to contain a multi-task HOI. S23. Perform network fusion and construct a fusion feature set: Place the original video's saved sequence frames into a set, extract skeletal data and object recognition results starting from the starting frame, and compare the sequence numbers of the original video frames corresponding to the extracted skeletal data and object recognition results into the set. The intersection keyframes in the set represent the fused part of the skeletal data and object recognition results. Finally, generate a fusion feature set based on the intersection keyframes, which specifically includes the following sub-steps: S231. Define the feature set generated by the pose estimation network and the object detection network as... <F human F object >, where F human ={F h1 ,F h2 F hi }, i∈[1,N] and F object ={F o1 F o2 F oi Let i ∈ [1, N] be the feature sets of human joints or objects in the i-th frame. Assuming c objects are detected in the i-th frame, its output is defined as F. oi ={O i1 O i2 , ..., O ic } and O ic ={(x ic y ic )|x ic ∈[x ic1 x ic2 ], y ic ∈[y ic1 y ic2 ]}, F oi and O ic The top left corner (x) ic1 y ic1 ) and bottom right corner (x ic1 y ic1 The c-th box between ); S232. Based on the values ​​in the interactive frames, consecutive frames with the same object category are labeled as sub-videos with labels from N1 to N2, where N1, N2 ∈ [1, N], and N1 < N2. Then, network fusion is performed to obtain the intersection of human joints and objects. <F human F object > Among them, H om The feature set of the sub-video extracted from the N1 to N2 interactive frames represents the combination of two-dimensional coordinates of human joints and objects after network fusion. S233. Redefine the key points and line segments in the interactive frame as {K} i0 K i1 , ..., K i24 K i25 K i26 }, i∈[N1, N2] and {L i0 L i1 , ..., L i24 L i25 }, where K ia =(x ia y ia ), i∈[0,24] represents the coordinates of 25 key points, K i24 =(x io1 y io1 ) and K i25 =(x io2 y io2 ) are the corners B of the detected bounding box. i L i0 L i1 , ..., L i24 L is the skeleton vector. i25 The target vector; S234. Extract the following features from human joints and object keyframes: H i ={(D iab ,A iab )|D iab =|K ia K ib |,A iab =L ia gL ib ,a=0,1L 24,b=0,1L 24,a≠b} O i ={(D iob ,A iob )|D iob =|K io K ib |,A iob =L io gL ib ,o=25,b=0,1L 24} Among them, H i O i D represents the feature set of the human body and objects in spatial dimensions. iab The distance between the two joints; A iab The angle between the two skeleton vectors; The distance between any two joints is: Then, a matrix integral feature is defined in the spacetime dimension: a,b∈[0,24],c=25,j<i,j∈[N1,N2-1] Among them, D ija Let D be the distance between corresponding keypoints in the i-th and j-th interactive frames. ija =|K ja -K ia |, a∈[0,26]; S235. Select the interactive frame and generate the fused feature set, so that the label set I of the interactive frame... m ={i1, i2, ..., i m }, i1, i2, L, i m ∈[N1, N2] satisfies in, After selecting interaction frames based on the distance traveled by people and objects over time, a fused feature set is finally generated. S3. The fusion feature set HO of the sub-videos extracted from the interaction frames N1 to N2 obtained in step S2. m The fused features are obtained by fusing human joint features and object features. S4. The fused features are used to classify the video action by a multi-level learning classification method. The feature vectors between people and objects and the feature vectors between human joints are input into the multi-level classification model for training. A feedback channel is established to find and output the classification result of the highest level classifier, thus completing the action classification of the skeletal data video. Multi-level learning classification methods employ multi-level classification learning algorithms, which utilize ensemble learning to cascade two or more classifiers to achieve higher classification accuracy. The specific process is as follows: S41. Input the fused features into the loop of k-fold cross-entropy validation and train using a multi-layer learning method. The original training set is randomly divided into k parts without repetition. One part is selected as the validation set, and the remaining k-1 parts are used as the training set for model training. The training data is put into the first layer classifier for training to obtain the predicted class vector. Then, the predicted vector is used as a new feature and combined with the n-dimensional feature vector extracted from the dataset. The combined feature vector is then input into the second, third, ... k-th layer classifiers for training to obtain a multi-layer learning network model. The validation set is input into the multi-layer learning network model for testing, and the test results are saved as the model's evaluation metric. S42. Repeat step S41 a total of k times to obtain k test results; S43. Calculate the average value of k test results as the performance evaluation index, establish a feedback channel, and obtain the final classification result through voting.

2. The human-object interaction action recognition method based on multi-feature fusion according to claim 1, characterized in that: The specific method for establishing the feedback channel in step S43 is to put the feature vectors between the human joints and the object and the feature vectors between the human joints into a multi-layer learning network model for multi-layer training. Starting from the second layer classifier, each layer will obtain a classification feedback result, and the classification feedback result will be returned to the previous layer classifier and the next layer classifier respectively, until the classification result of a certain layer classifier is greater than the classification results of the previous and next layer classifiers. Then, the classification result of the layer classifier corresponding to the highest accuracy is output as the final classification result.

3. The human-object interaction action recognition method based on multi-feature fusion according to claim 1, characterized in that: In step S2, before judging the interaction between the human body and the object, it is first necessary to judge whether the human body and the object are in contact, i.e., contact is 1 and no contact is 0.

4. The human-object interaction action recognition method based on multi-feature fusion according to claim 1, characterized in that: In step S1, after extracting the skeletal data from the original video, the output video is saved as the corresponding sequence of frames, and the 2D coordinates of the joints of each frame are saved locally.

5. The multi-feature fusion human-object interaction action recognition method according to claim 1, characterized in that: The features in the intersection keyframes in step S2 include the distance and angle from the object to the joints of the human body, as well as the distance and angle between the joints of the human body itself.

6. The human-object interaction action recognition method based on multi-feature fusion according to claim 5, characterized in that: In step S2, the top-left and bottom-right vertices of the object's bounding box are m respectively. p m q m p and m q The spatial coordinates are: m p =(x io1 ,y io1 ),m q =(x io2 ,y io2 )。 7. The human-object interaction action recognition method based on multi-feature fusion according to claim 5, characterized in that: In step S3, the human joint features and object features are fused by adding the features together to obtain the fused features.

Images