Two-person interaction behavior identification method based on MSA-CLSTM body local activeness

Abstract

The invention provides a two-person interaction behavior recognition method based on MSA-CLSTM body local activeness, and the method comprises the steps: firstly, extracting a human body skeleton key point sequence of an interaction person from an RGB video through employing a posture estimation technology; performing activeness evaluation on the key points of the human body skeleton sequence through a human body local activeness evaluator, generating a human body local weight, and weighting the original skeleton sequence to obtain a key point coordinate feature flow; calculating a relative distance between key points of the human skeleton, and generating a relative position feature flow and a mutual feature flow of the key points through two-dimensional discrete wavelet transform and a space attention mechanism; calculating a local skeleton direction according to the original data of the human skeleton, calculating an included angle between skeletons according to a natural connection mode of the human skeleton, and generating a local skeleton direction feature flow; and finally, realizing fusion of four feature flows through a multi-feature flow fusion model, calculating according to the fused features to obtain a double-person interaction action category, and completing double-person interaction behavior recognition.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method for identifying two-person interactive behavior based on MSA-CLSTM local body activity. Background Art

[0002] Due to the growing demand for analysis of two-person interaction activities in daily production, life, and management, two-person interaction behavior recognition has gradually become an important research direction in the field of pattern recognition. Among them, the recognition method based on RGB video is widely used in intelligent monitoring, smart medical care, human-computer interaction, etc. In these scenarios, two-person interaction behavior is common.

[0003] Although a large number of studies have been conducted on the recognition and analysis of interactive behaviors in two-person scenarios, the recognition of two-person interactive behaviors still faces many challenges. Two-person interaction is not just a combination of independent actions of two individuals, it involves complex behavioral patterns and changing interactive scenarios. For example, the body movements and posture changes between two people may be affected by multiple factors such as the environment, contextual semantics, and interactive objects. These factors make the modeling of interactive behaviors more complex, and the model needs to have sufficient feature expression capabilities to capture and understand these complex interactive actions. When paying attention to more interactive features, it also means that there will be more invalid features, which will increase the model burden and even affect the recognition performance of the model. Therefore, it is necessary to pay attention to valid features consciously. At the same time, in the scene, when another person who is not related to the interaction appears in the video, the posture estimation technology may assign the skeleton of the irrelevant person to the interactor, which will greatly weaken the reliability of the human skeleton data. This situation often exists in real videos, which makes the optimization of posture estimation technology also very important.

[0004] Existing methods for identifying two-person interactive behaviors do not pay attention to the problem of feature redundancy, and mainly rely on deep learning models to extract a large number of features, which makes the feature extraction process affected by invalid features. At the same time, current methods for identifying two-person interactive behaviors cannot take into account both individual action details and overall interactive properties, which causes the model to pay too much attention to individuals and ignore the whole, or pay too much attention to the whole and ignore individuals, which will lead to insufficient feature representation and affect model performance. Summary of the invention

[0005] In view of the shortcomings of the above background technology, the present invention proposes a two-person interactive behavior recognition method based on MSA-CLSTM body local activity, and the specific steps are as follows:

[0006] S1. Continuously track interactive characters through frame optimization method;

[0007] S2. Supplement frames of the action sequence by frame optimization method;

[0008] S3 establishes the MSA activity evaluation module, which highlights individual actions and local active limbs with interactive semantics;

[0009] S4. Based on the relative position of the key points at each moment, the relative distance matrix of the key points of the individual is regarded as an image. The image is transferred from the spatial domain to the spatial-frequency domain by using a two-dimensional discrete wavelet transform, and the individual action details are extracted at multiple scales to construct the individual action feature stream;

[0010] S5. Select each pair of adjacent key points as a local bone, calculate the direction of the local bone unit vector, and calculate the angle between adjacent local bones according to the natural connection mode of the human body to describe the movement posture of the human body;

[0011] S6. Obtaining two-person action interaction features;

[0012] S7. A multi-feature stream fusion model MSA-CLSTM for two-person interactive behavior recognition is established. Each individual feature stream is given a learnable coefficient to dynamically adjust the weight of each feature stream. Multiple feature streams are fused through a neural network, and the fused features are used as the basis for classifying two-person interactive actions to achieve two-person interactive action recognition.

[0013] Furthermore, step S1 is specifically as follows:

[0014] S1.1 obtains the bounding box of the person in the video through the existing AlphaPose posture estimation technology, and sets the number of bounding boxes to 2. The offset threshold is set to 200 pixels, which is consistent with the natural movement speed of the human body between two frames.

[0015] S1.2 When an unrelated person appears in the picture, AlphaPose will capture his or her bounding box, so that the offset distance between the two frames exceeds the offset threshold. At this time, the optimization mechanism is triggered, and the bounding box position tracked in the previous frame covers the bounding box position of this frame, so that the two frames continue to track the same person.

[0016] Furthermore, step S2 is specifically as follows:

[0017] S2.1 uses the existing AlphaPose posture estimation technology to extract the human skeleton of the person in the human bounding box and obtain the human skeleton action sequence.

[0018] S2.2 records the key point positions of the human skeleton in each frame. When the human body overlaps in the video, the key point positions cannot be obtained and are automatically set to the position of the pixel origin (0,0) of the screen. This causes the key point position offset between the two frames to exceed the offset threshold, triggering the optimization mechanism.

[0019] When the S2.3 optimization mechanism is triggered, the current frame is overwritten according to the key point position of the previous frame, so that when the key point position is abnormal, the key points can still form a complete human skeleton and will not destroy the contextual relevance of the action sequence, thereby obtaining the skeleton action sequence of the two interactors. The key points form the skeleton P according to the natural connection method of the human body joints. sub =(p sub (x 0 ,y 0 ),…,p sub (x n-1 ,y n-1 )),sub∈{A,B}.

[0020] Furthermore, step S3 is specifically as follows:

[0021] S3.1 Obtaining the relative position change matrix

[0022] The minimum joint connection method is adopted, each key point and adjacent points form several joints, and adjacent joints are connected to form a skeleton. The Euclidean distance between the two key points i and j of individual A and individual B at time t is taken as the relative distance.

[0023]

[0024] The interactive activity evaluator calculates the mutual activity of the n key points of individual A and individual B. After determining the relative distance of each pair of key points, the relative position change between time t and time t-1 is calculated as follows:

[0025]

[0026] It reflects the motion state of each pair of key points, and the relative position changes of the n key points of individual A to the n key points of individual B constitute a relative position change matrix The calculation is as follows:

[0027]

[0028] S3.2 Calculation of Mutual Activity

[0029] right The L1 norm of the row vector and column vector is calculated to obtain the preliminary Mutual activity score of each key point. It is the sum of the absolute values ​​of the relative position changes of key point i and other key points of individual A, calculated as follows:

[0030]

[0031] Matrix column vector Represents the relative position change relationship between the i-th key point of individual A and all the key points of individual B. The row vector represents the relative position change relationship between the jth key point of individual B and all key points of individual A. The preliminary Mutual activity score of each person at the current moment is calculated by formula (4): The mutual activity is obtained by normalizing them respectively using formula (5).

[0032]

[0033] Where α is a minimum constant and δ is the equilibrium constant. To splice the row vectors in the vertical direction, splice the mutual activity of the key points at each moment to obtain the mutual activity matrix of the entire time period t end For the last time point.

[0034] S3.3 Secondary weighting of Mutual activity through spatial attention mechanism

[0035] Will and Splice into a feature map t is the total time step, 2n is the total number of key points of individual A and individual B, and the key points of the two people are weighted twice through the SE Attention module. The specific calculation is as follows:

[0036] s=σ(W 2 δ(W 1 z))

[0037]

[0038] W 1 ,W 2 is the weight of the fully connected layer, σ is the Sigmoid activation function, δ is the ReLU activation function, and ⊙ represents element-by-element multiplication. A ,s B are the quadratic weights of the key points belonging to individual A and individual B respectively. The obtained quadratic weighted feature map Split into and They are the mutual activities of individual A and individual B respectively.

[0039] S3.4 Get Self activity

[0040] The calculation of the Self-activity evaluator for Self-activity is similar to that of the interactive activity evaluator. The difference is that the self-activity evaluator only focuses on the individual's own actions.

[0041] Get the relative position change matrix of A and B Let B = A in the interactive activity evaluator, and we can get the Self activity of individual A. Let A=B in the interactive activity evaluator, and we can get the Self activity of individual B.

[0042] Furthermore, step S4 is specifically as follows:

[0043] S4.1 Obtaining the relative distance matrix of key points of the individual

[0044] Make A and B equal in formula (1), and calculate the relative distance between the key point i of each individual and its own n key points at time t Get the relative distance matrix of the individual's own key points as follows:

[0045]

[0046] S4.2 The relative distance matrix of the key points of the individual Extract high-frequency and low-frequency components as an image

[0047] The scale functions and wavelet functions of different scales and positions are defined as follows:

[0048]

[0049] In the formula is the scaling function, ψ l is a wavelet function, H, V, D represent the horizontal, vertical and diagonal directions respectively,

[0050] h, k correspond to the positions of the scale function and wavelet function in the horizontal and vertical directions respectively. The two-dimensional discrete wavelet transform is used to extract the low-frequency and high-frequency components of the image as follows:

[0051]

[0052] The approximate coefficient That is, low-frequency components. Detail coefficient That is, the high-frequency component is divided into high-frequency information in the vertical, horizontal, and diagonal directions of the image. Definition is the horizontal splicing matrix, and each individual is obtained based on the relative position feature flow of key points

[0053] Furthermore, step S5 is specifically as follows:

[0054] S5.1 Get the local bone unit vector direction

[0055] The unit vector direction of a bone at time t is calculated by formula (10):

[0056]

[0057] Where i and j are the numbers of two adjacent key points. Get the individual bone direction at all times t end is the last time step.

[0058] S5.2 Calculation of angles between adjacent bones

[0059] According to the bone combination method of posture estimation, calculate the angle between the two bones formed by the three key points i, j, and k:

[0060]

[0061] The bone angle of an individual at all times is expressed as Horizontally concatenate the bone direction and the bone angle as the local bone direction feature flow

[0062] Further, step S6 is specifically as follows:

[0063] The distance between each key point of individual A and individual B at time t is calculated by formula (1): Get the mutual feature space at time t:

[0064]

[0065] The mutual feature space for all moments Generate spatial scale weights:

[0066]

[0067] Where T is the total number of time steps, σ is the sigmoid function, h,w are spatial position indexes, and W t is a 1*1 convolution kernel, and is a convolution operation. The generated spatial weight matrix S is combined with the mutual feature space Multiply element by element to get the mutual feature flow after spatial scale weighting

[0068] Further, step S7 is specifically as follows:

[0069] Assign a learnable coefficient Θ to each individual feature stream, set four CNNs as feature extractors, and each feature extractor is responsible for processing one feature stream. Use the same feature extractor to extract the individual features of two people. The spliced ​​output feature is the fused individual feature stream F sub =[f P,sub ·Θ P ,f T,sub·Θ T ,f V,sub ·Θ V ]. sub Input each individual sub-LSTM and pass it through the fully connected layer to get the individual action prediction label. Model output prediction label Where c is the number of interactive action categories, n is the sample number, Predict the probability distribution of category k for the nth sample, and the true label of the nth sample is t (n) ∈{0,…,c-1}, the predicted label and the true label are calculated through the cross entropy loss function to obtain the individual action loss. sub The calculation is as follows:

[0070]

[0071] Mutual characteristic flow After inputting into CNN and F A and F B The two are input into LSTM together and the mutual action prediction labels are obtained through full connection. Where c is the number of interactive action categories, n is the sample number, The probability distribution of the nth sample predicting category k is obtained by formula (15): mutual .

[0072]

[0073] The total loss of the model MSA-CLSTM is Loss = Loss mutual +Loss A +Loss B According to the total loss, the parameter weights are back-propagated to update the MSA-CLSTM parameters, so that MSA-CLSTM can automatically learn the recognition of two-person interaction behaviors.

[0074] Compared with the prior art, the advantages of the present invention are:

[0075] 1) When tracking interactive characters in the video, the problem of abnormal key point positions caused by body occlusion is solved, reducing the negative impact of abnormal key points on the context of the action sequence and the integrity of the action sequence.

[0076] 2) In the task of identifying two-person interactive behaviors with a large number of features, there are some invalid features that increase the model burden and even affect the model recognition performance. The MSA activity evaluation module proposed in the present invention enables the model to consciously focus on valid features and give smaller weights to invalid features, thereby reducing the impact of invalid features on the model.

[0077] 3) The present invention aims at the interactive action recognition in the two-person interaction scenario. The MSA-CLSTM model proposed in the present invention integrates the individual's own action characteristics and the interaction relationship between the two, which solves the problem that the model pays too much attention to individual actions and ignores the overall interaction relationship, or pays too much attention to the overall interaction relationship and ignores the details of individual actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0079] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] like Figure 1 As shown, the embodiment of the present invention provides a method for identifying two-person interactive behavior based on MSA-CLSTM body local activity, and the specific steps are as follows:

[0081] S1. Continuous tracking of interactive characters through frame optimization method

[0082] S1.1 obtains the bounding box of the person in the video through the existing AlphaPose posture estimation technology, and sets the number of bounding boxes to 2. The offset threshold is set to 200 pixels, which is consistent with the natural movement speed of the human body between two frames.

[0083] S1.2 When an unrelated person appears in the picture, AlphaPose will capture his or her bounding box, so that the offset distance between the two frames exceeds the offset threshold. At this time, the optimization mechanism is triggered, and the bounding box position tracked in the previous frame covers the bounding box position of this frame, so that the two frames continue to track the same person.

[0084] S2. Add frames to action sequences using frame optimization methods

[0085] S2.1 uses the existing AlphaPose posture estimation technology to extract the human skeleton of the person in the human bounding box and obtain the human skeleton action sequence.

[0086] S2.2 records the key point positions of the human skeleton in each frame. When the human body overlaps in the video, the key point positions cannot be obtained and are automatically set to the position of the pixel origin (0,0) of the screen. This causes the key point position offset between the two frames to exceed the offset threshold, triggering the optimization mechanism.

[0087] When the S2.3 optimization mechanism is triggered, the current frame is overwritten according to the key point position of the previous frame, so that when the key point position is abnormal, the key points can still form a complete human skeleton and will not destroy the contextual relevance of the action sequence, thereby obtaining the skeleton action sequence of the two interactors. The key points form the skeleton P according to the natural connection method of the human body joints. sub =(p sub (x 0 ,y 0 ),…,p sub (x n-1 ,y n-1 )),sub∈{A,B}.

[0088] S3 establishes the MSA activity evaluation module, highlighting individual movements and local active limbs with interactive semantics.

[0089] S3.1 Obtaining the relative position change matrix

[0090] The minimum joint connection method is adopted, each key point and adjacent points form several joints, and adjacent joints are connected to form a skeleton. The Euclidean distance between the two key points i and j of individual A and individual B at time t is taken as the relative distance.

[0091]

[0092] The interactive activity evaluator calculates the mutual activity of the n key points of individual A and individual B. After determining the relative distance of each pair of key points, the relative position change between time t and time t-1 is calculated as follows:

[0093]

[0094] It reflects the motion state of each pair of key points, and the relative position changes of the n key points of individual A to the n key points of individual B constitute a relative position change matrix The calculation is as follows:

[0095]

[0096] S3.2 Calculation of Mutual Activity

[0097] right The L1 norm of the row vector and column vector is calculated to obtain the preliminary Mutual activity score of each key point. It is the sum of the absolute values ​​of the relative position changes of key point i and other key points of individual A, calculated as follows:

[0098]

[0099] Matrix column vector Represents the relative position change relationship between the i-th key point of individual A and all the key points of individual B. The row vector represents the relative position change relationship between the jth key point of individual B and all key points of individual A. The preliminary Mutual activity score of each person at the current moment is calculated by formula (4): The mutual activity is obtained by normalizing them respectively using formula (5).

[0100]

[0101] Where α is a minimum constant and δ is the equilibrium constant. To splice the row vectors in the vertical direction, splice the mutual activity of the key points at each moment to obtain the mutual activity matrix of the entire time period and t end For the last time point.

[0102] S3.3 Secondary weighting of Mutual activity through spatial attention mechanism

[0103] Will and Splice into a feature map t is the total time step, 2n is the total number of key points of individual A and individual B, and the key points of the two people are weighted twice through the SE Attention module. The specific calculation is as follows:

[0104] s=σ(W 2 δ(W 1 z))

[0105]

[0106] W 1 ,W 2 is the weight of the fully connected layer, σ is the Sigmoid activation function, δ is the ReLU activation function, and ⊙ represents element-by-element multiplication. A ,s B are the quadratic weights of the key points belonging to individual A and individual B respectively. The obtained quadratic weighted feature map Split into and They are the mutual activities of individual A and individual B respectively.

[0107] S3.4 Get Self activity

[0108] The calculation of the Self-activity evaluator for Self-activity is similar to that of the interactive activity evaluator. The difference is that the self-activity evaluator only focuses on the individual's own actions.

[0109] Get the relative position change matrix of A and B Let B = A in the interactive activity evaluator, and we can get the Self activity of individual A. Let A=B in the interactive activity evaluator, and we can get the Self activity of individual B.

[0110] S4. Based on the relative positions of the key points at each moment, the relative distance matrix of the key points of the individual is regarded as an image. The image is transferred from the spatial domain to the spatial-frequency domain by using a two-dimensional discrete wavelet transform, and the individual action details are extracted at multiple scales to construct the individual action feature stream.

[0111] S4.1 Obtaining the relative distance matrix of key points of the individual

[0112] Make A and B equal in formula (1), and calculate the relative distance between the key point i of each individual and its own n key points at time t Get the relative distance matrix of the individual's own key points as follows:

[0113]

[0114] S4.2 The relative distance matrix of the key points of the individual Extract high-frequency and low-frequency components as an image

[0115] The scale functions and wavelet functions of different scales and positions are defined as follows:

[0116]

[0117] In the formula is the scaling function, ψ l is a wavelet function, H, V, D represent the horizontal, vertical and diagonal directions respectively,

[0118] h, k correspond to the positions of the scale function and wavelet function in the horizontal and vertical directions respectively. The two-dimensional discrete wavelet transform is used to extract the low-frequency and high-frequency components of the image as follows:

[0119]

[0120] The approximate coefficient That is, low-frequency components. Detail coefficient That is, the high-frequency component is divided into high-frequency information in the vertical, horizontal, and diagonal directions of the image. Definition is the horizontal splicing matrix, and each individual is obtained based on the relative position feature flow of key points

[0121] S5. Select each pair of adjacent key points as a local bone, calculate the direction of the local bone unit vector, and calculate the angle between adjacent local bones according to the natural connection method of the human body to describe the human body's movement posture.

[0122] S5.1 Get the local bone unit vector direction

[0123] The unit vector direction of a bone at time t is calculated by formula (10):

[0124]

[0125] Where i and j are the numbers of two adjacent key points. Get the individual bone direction at all times t end is the last time step.

[0126] S5.2 Calculation of angles between adjacent bones

[0127] According to the bone combination method of posture estimation, calculate the angle between the two bones formed by the three key points i, j, and k:

[0128]

[0129] The bone angle of an individual at all times is expressed as Horizontally concatenate the bone direction and the bone angle as the local bone direction feature flow

[0130] S6. Obtaining two-person action interaction features

[0131] The distance between each key point of individual A and individual B at time t is calculated by formula (1): Get the mutual feature space at time t:

[0132]

[0133] The mutual feature space for all moments Generate spatial scale weights:

[0134]

[0135] Where T is the total number of time steps, σ is the sigmoid function, H,t is the spatial position index, W is the 1*1 convolution kernel, and · is the convolution operation. The generated spatial weight matrix S is combined with the mutual feature space Multiply element by element to get the mutual feature flow after spatial scale weighting

[0136] S7. A multi-feature stream fusion model MSA-CLSTM for two-person interactive behavior recognition is established. Each individual feature stream is given a learnable coefficient to dynamically adjust the weight of each feature stream. Multiple feature streams are fused through a neural network, and the fused features are used as the basis for classifying two-person interactive actions to achieve two-person interactive action recognition.

[0137] Assign a learnable coefficient Θ to each individual feature stream, set four CNNs as feature extractors, and each feature extractor is responsible for processing one feature stream. Use the same feature extractor to extract the individual features of two people. The spliced ​​output feature is the fused individual feature stream F sub =[f P,sub ·Θ P ,f T,sub ·Θ T ,f V,sub ·Θ V ]. sub Input each individual sub-LSTM and pass it through the fully connected layer to get the individual action prediction label. Model output prediction label Where c is the number of interactive action categories, n is the sample number, Predict the probability distribution of category k for the nth sample, and the true label of the nth sample is t (n) ∈{0,…,c-1}, the predicted label and the true label are calculated through the cross entropy loss function to obtain the individual action loss. sub The calculation is as follows:

[0138]

[0139] Mutual characteristic flow M A,B After inputting into CNN and F A and F B The two are input into LSTM together and the mutual action prediction labels are obtained through full connection. Where c is the number of interactive action categories, n is the sample number, The probability distribution of the nth sample predicting category k is obtained by formula (15): mutual .

[0140]

[0141] The total loss of the model MSA-CLSTM is Loss = Loss mutual +Loss A +Loss B According to the total loss, the parameter weights are back-propagated to update the MSA-CLSTM parameters, so that MSA-CLSTM can automatically learn the recognition of two-person interaction behaviors.

[0142] The above are preferred embodiments of the present invention. A person skilled in the art may make various changes or improvements on this basis. Without departing from the general concept of the present invention, these changes or improvements should fall within the scope of protection required by the present invention.

Claims

1. A two-person interactive behavior recognition method based on MSA-CLSTM body local activity, characterized in that: The specific steps are as follows: S1. Continuously track interactive characters through frame optimization method; S2. Fill in frames of the action sequence by frame optimization method; S3 establishes the MSA activity evaluation module, which highlights individual actions and local active limbs with interactive semantics; S4. Based on the relative position of the key points at each moment, the relative distance matrix of the key points of the individual is regarded as an image. The image is transferred from the spatial domain to the spatial-frequency domain by using a two-dimensional discrete wavelet transform, and the individual action details are extracted at multiple scales to construct the individual action feature stream; S5. Select each pair of adjacent key points as a local bone, calculate the direction of the local bone unit vector, and calculate the angle between adjacent local bones according to the natural connection mode of the human body to describe the movement posture of the human body; S6. Obtaining two-person action interaction features; S7. A multi-feature stream fusion model MSA-CLSTM for two-person interactive behavior recognition is established. Each individual feature stream is given a learnable coefficient to dynamically adjust the weight of each feature stream. Multiple feature streams are fused through a neural network, and the fused features are used as the basis for classifying two-person interactive actions to achieve two-person interactive action recognition.

2. A two-person interactive behavior recognition method based on MSA-CLSTM body local activity as claimed in claim 1, characterized in that: Step S1 is specifically as follows: S1.1 obtains the bounding box of the person in the video through the existing AlphaPose posture estimation technology, sets the number of bounding boxes to 2, and sets the offset threshold to 200 pixels, which is consistent with the natural movement speed of the human body between two frames; S1.2 When an unrelated person appears in the picture, AlphaPose will capture his or her bounding box, so that the offset distance between the bounding box of the two frames exceeds the offset threshold. At this time, the optimization mechanism is triggered, and the bounding box position of the previous frame is overwritten by the bounding box position of this frame, so that the two frames can continuously track the same person.

Images