A video motion segmentation and recognition method based on frame sequence order perception coding

Abstract

The application discloses a video motion segmentation and recognition method based on frame sequence order perception coding, wherein the segmentation method comprises the following steps: extracting a feature vector of a given video, and coding the feature vector to obtain the motion feature of the given video; learning a coding matrix based on the motion feature of the given video to obtain the coding matrix of the given video; constructing an affinity graph based on the coding matrix of the given video; and completing the motion segmentation of the given video based on the affinity graph to obtain the motion segment of the given video. The recognition method comprises the following steps: segmenting a video to be recognized into several motion segments based on the above segmentation method; inputting the feature of each motion segment into a pre-trained classifier; and assigning a motion label according to the probability of the category to which each motion segment belongs. The application can not only fully code the motion feature and frame sequence order information of the video, but also can overcome the influence of noise in the video to accurately segment the motion segment and recognize the category.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a video action segmentation and recognition method based on frame sequence order-aware coding. Background Technology

[0002] Video action recognition is a crucial and widely studied problem in the field of computer vision, with practical applications in areas such as video surveillance, video indexing, and autonomous navigation. Video action recognition refers to the task of labeling video clips involving human movement into different action categories.

[0003] Early work on video action recognition primarily employed "video-level" methods. These methods typically involved three steps: feature extraction, feature encoding, and action classification. Overall, they treated each video as a whole across all stages of the task. However, real-world videos are mostly untrimmed, containing multiple actions and noise caused by subtle human and camera movements. Therefore, "video-level" methods are difficult to apply in real-world scenarios. Consequently, the focus of video action recognition research has gradually shifted from "video-level" to "segment-level" methods. In segment-level methods, to improve the accuracy of human action recognition in untrimmed videos, the video is first divided into action segments based on the frames containing the human actions—that is, video action segmentation is performed first, followed by action segment recognition. In recent years, numerous video action segmentation methods have been proposed, mainly categorized into supervised and unsupervised methods. Compared to supervised methods, unsupervised methods do not require time-consuming and labor-intensive video action labeling and significantly reduce network model training time, thus offering greater practical application value. Among existing unsupervised video action segmentation methods, subspace clustering is a typical and one of the most effective methods. However, current subspace clustering techniques face several technical challenges when applied to video action segmentation and recognition:

[0004] 1) Most existing subspace clustering methods ignore the sequence information of video frames. However, learning video codes that incorporate both motion features and sequence information can more fully represent the video. Therefore, most current subspace clustering methods struggle to learn and obtain an effective coding matrix, thus failing to achieve ideal action segmentation and classification accuracy.

[0005] 2) Most existing affinity graph segmentation methods use the traditional normalized segmentation (NCuts) method. However, this method is easily affected by various noises from the video, such as noise caused by subtle human movements and camera movements, resulting in inaccurate final video action segmentation results. Summary of the Invention

[0006] Purpose of the invention: To address the problems of existing technologies, such as difficulty in fully learning and obtaining an effective coding matrix and susceptibility to noise from the video, this invention provides a video action segmentation and recognition method based on frame sequence order-aware coding.

[0007] Technical solution:

[0008] This invention provides a video action segmentation method based on frame sequence order-aware coding, comprising the following steps:

[0009] Step 1: Extract the feature vector of the given video and encode the feature vector to obtain the motion features of the given video;

[0010] Step 2: Based on the motion features of the given video, learn the encoding matrix to obtain the encoding matrix of the given video;

[0011] Step 3: Construct an affinity map based on the encoding matrix of the given video;

[0012] Step 4: Based on the affinity map, complete the action segmentation of the given video to obtain the action segments of the given video.

[0013] Furthermore, step one specifically includes:

[0014] Step 1: Extract the iDT feature vector of a given video of length τ frames;

[0015] Step 2: Encode the iDT feature vectors using the Fisher vector coding algorithm to obtain the motion features V of the given video. Where the i-th element of V This represents the i-th element of the encoded iDT feature vector, where m is the total number of elements in the motion feature V.

[0016] Furthermore, step two specifically involves:

[0017] Step I: Initial encoding matrix calculation

[0018] First, define training vectors for m arbitrary samples. Where V k Let F represent the iDT feature vector of the k-th frame of a given video. k This represents the sequence order information of the k-th frame in a given video.

[0019] Then, initialize the iteration count j = 1, and randomly initialize the parameters of the first summation point of the coding layer of the multi-layer neural network. It is the weight of the first summary point. It is the bias of the first summary point;

[0020] Finally, using this summary point and given activation function g s (·), calculate the initial encoding matrix Where V represents the motion features of the given video;

[0021] Step II: Learning Layer Parameter Update

[0022] First, the encoding matrix of the j-th iteration... As input to the learning layers of a multi-layer neural network, This represents the first summation point of the coding layer in a multi-layer neural network;

[0023] Then, using the desired output video frame sequence order information F and the given activation function g... t (·) and normalization function u j (·), calculate the parameters of the learning layer in the j-th iteration. in These are the node weights of the learning layer during the j-th iteration. It is the node bias of the learning layer at the j-th iteration;

[0024] Step III: Error Feedback Data Update

[0025] The output error e of the j-th iteration is updated using the desired video frame sequence order information F and the output F′ of the j-th iteration learning layer. j Then use the output error e j Update the error feedback data P for the j-th iteration. j ;

[0026] Step IV: Encoding Matrix Update

[0027] First, let j = j + 1;

[0028] Then, using the error feedback data P from the j-th iteration in step III... j Add a new summary point at the coding layer.

[0029] Finally, the encoding matrix is ​​updated using the added summary points.

[0030] Step V: Iterate and optimize

[0031] First, repeat steps II to IV L-1 times until j = L, thus summing the set of point parameters. The optimal parameters for the coding layer are determined; where L is the total number of iterations.

[0032] Then, by removing the first summary point parameter The influence of randomness is considered, and optimization is used to obtain the final encoding matrix Z.

[0033] Furthermore, the parameters of the learning layer in the j-th iteration The specific calculation method is as follows:

[0034]

[0035]

[0036] in, yes The Moore-Penrose generalized inverse matrix, And C is a constant, I is the identity matrix, F is the frame sequence order information of the given video, and g t (·) and u j (·) are the activation function and the normalization function for the j-th summation point, respectively, and u j (F): This is the root mean square error.

[0037] Furthermore, the error feedback data P of the j-th iteration j The specific calculation method is as follows:

[0038]

[0039] Wherein, the output error of the j-th iteration u j -1 (·) is u j The inverse function of (·).

[0040] Furthermore, the specific method for calculating the final encoding matrix Z is as follows:

[0041]

[0042] in, Let represent a d×m dimension encoding matrix, where d is both the number of child nodes of each summary point in the encoding layer and the dimension of the encoding matrix.

[0043] Furthermore, the affinity map in step three is specifically as follows:

[0044]

[0045] in,, Represents an m×m dimension affinity graph, Z = {z1, z2, ..., z...} m}, α, β∈{1, 2, ..., m}, where m is the number of motion feature vectors V, z i and z jThese are the i-th and j-th elements of Z, respectively.

[0046] Furthermore, step four specifically involves:

[0047] Step i: Extraction of the diagonal of the affinity map

[0048] First, the affinity diagram is processed using the discrete Laplace operator K;

[0049] Then, the diagonal of the affinity map is extracted and a one-dimensional signal ρ0 is generated;

[0050] Step ii: Signal smoothing

[0051] A Gaussian blur filter is used to smooth the one-dimensional signal ρ0 to filter out noise, thereby obtaining a smooth one-dimensional signal vector ρ.

[0052] Step iii: Initial frame boundary detection

[0053] First, in the one-dimensional signal vector ρ, the initial frame boundary where different actions exist is determined based on two adjacent peak values;

[0054] Then, the given video is divided into B frame blocks using the initial frame boundaries, resulting in a frame block sequence T = {T1, T2, ..., T...}. B};

[0055] Step iv: Redundant frame boundary removal

[0056] First, in the frame block sequence T = {T1, T2, ..., T...} B In this process, by comparing the similarity between adjacent frame blocks, frame blocks with a similarity less than a threshold ε are repeatedly merged, thereby removing redundant frame boundaries and re-dividing the given video into N frame blocks, resulting in a new frame block sequence E = {E1, E2, ..., E...}. N};

[0057] Then, based on E, the given video is divided into N action segments C = {c1, ..., c2}. N}, where action fragment c i Corresponding to frame block E i .

[0058] Furthermore, if the similarity between adjacent frame blocks is less than a set threshold, the two frame blocks are merged, and the above steps are repeated until the similarity between adjacent frame blocks is not less than the set threshold.

[0059] This invention also provides a video action recognition method based on frame sequence order-aware coding, which specifically includes the following steps:

[0060] Step A: Based on the segmentation method described above, the video to be identified is segmented into N action segments;

[0061] Step B: Input the features of each action segment into a pre-trained classifier;

[0062] Step C: Assign action labels based on the probability of each action fragment belonging to its category.

[0063] Beneficial effects:

[0064] (1) This invention designs a video action segmentation and recognition method based on frame sequence order perception coding. This method takes into account the sequence order information of a given video frame, can fully encode the motion features and order information of the video, and improves the action clustering performance and classification performance.

[0065] (2) Considering the presence of noise, the present invention designs a robust cutting algorithm to cut the constructed affinity graph. The algorithm first extracts the one-dimensional signal of the diagonal of the affinity graph, then uses a filter to filter the noise in the video for the extracted one-dimensional signal, and finally removes noise in the process of cutting the one-dimensional signal and removing redundant frame boundaries to accurately obtain the action segment. Attached Figure Description

[0066] Figure 1 This is a flowchart illustrating the feature extraction and encoding process of the present invention.

[0067] Figure 2 Diagram of the multi-layer neural network structure used;

[0068] Figure 3 This is a diagram illustrating the encoding matrix learning process of the present invention.

[0069] Figure 4 This is a process diagram of the affinity map construction and segmentation of the present invention;

[0070] Figure 5 This is a flowchart illustrating the motion recognition process of the present invention. Detailed Implementation

[0071] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0072] This embodiment provides a video action segmentation and recognition method based on frame sequence order-aware coding. The method includes four processes: feature extraction and coding, coding matrix learning, affinity map construction and segmentation, and action recognition.

[0073] Feature extraction and encoding: In this embodiment, the feature extraction and precoding process extracts feature vectors from a given video and encodes them into motion features, such as... Figure 1 As shown, the specific steps are as follows:

[0074] Step 1: Extract the iDT feature vector of a given video of length f frames;

[0075] Step 2: Encode the feature vector using the Fisher vector coding algorithm to obtain the motion features V of the given video. Where the f-th element of V This represents the f-th element of the encoded iDT feature vector, where m is the total number of features and m is the total number of elements in the motion feature V.

[0076] Encoding matrix learning: In this embodiment, the encoding matrix learning process utilizes... Figure 2 The multi-layer neural network shown learns and obtains the encoding matrix of a given video using a sequence-aware encoding method, such as... Figure 3 As shown, the specific steps are as follows:

[0077] Step 1: Initial encoding matrix calculation. First, define training vectors for m arbitrary samples. Where V k Let F represent the feature vector of the k-th frame. k This represents the sequence order information of the k-th frame; then, the iteration count j = 1 is initialized, and the first summation point of the coding layer of the multi-layer neural network is randomly initialized. Finally, use this summary point and given activation function g s (·) Calculate the initial encoding matrix Z, where Let represent a d×m dimension encoding matrix, where d is both the number of child nodes of each summary point in the encoding layer and the dimension of the encoding matrix. in, This represents the first summation point of the coding layer in a multi-layer neural network.

[0078] Step II: Learning layer parameter update. First, update the encoding matrix of the j-th iteration. The parameters of the learning layer are then calculated using the desired output video frame sequence order information F, along with the given activation function gt(·) and normalization function un(·).

[0079] Step III: Error Feedback Data Update. Using the desired output sequence order information F and the predicted sequence order information F′, update the output error e of the j-th iteration. j Then use the output error e j Update error feedback data Pj ;

[0080] Step IV: Encoding Matrix Update. First, increment the iteration count by 1, i.e., j = j + 1; then, use the error feedback data P from Step III. j Add a new summary point at the coding layer. Finally, the encoding matrix is ​​updated using the added summary points.

[0081] Step V: Iterate and optimize. First, repeat steps II through IV L-1 times until j = L, thereby adjusting the parameters. To become the optimal parameters for the coding layer; then, by removing the parameters of the first summary point. The effect of randomness is optimized to obtain the final encoding matrix Z.

[0082] Affinity Graph Construction and Segmentation: In this embodiment, the affinity graph construction and segmentation process utilizes the encoding matrix learned from the given video to construct an affinity graph and then segments the affinity graph using a robust affinity graph segmentation algorithm to obtain action segments from the given video, such as... Figure 4 As shown, the specific steps are as follows:

[0083] Step i: Affinity map construction. Construct an affinity map G using the learned encoding matrix Z of the given video;

[0084] Step ii: Diagonal extraction of affinity map. First, the affinity map G is processed using the discrete Laplacian operator K; then, the diagonal of the affinity map G is extracted and a one-dimensional signal ρ0 is generated.

[0085] Step iii: Signal smoothing. A Gaussian blur filter is used to smooth the obtained one-dimensional signal ρ0 to filter out noise, thereby obtaining a smoothed one-dimensional signal vector ρ.

[0086] Step iv: Initial frame boundary detection. First, in the one-dimensional signal vector ρ, the initial frame boundaries where different actions exist are determined based on two adjacent peaks; then, the given video is divided into B frame blocks T = {T1, T2, ..., T...} using the determined initial frame boundaries. B};

[0087] Step v: Redundant frame boundary removal. In the frame block sequence T = {T1, T2, ..., T...} B In this process, by comparing the similarity between adjacent frame blocks, frame blocks with a similarity less than a threshold ε are repeatedly merged, thereby removing redundant frame boundaries and obtaining new frame blocks E = {E1, E2, ..., E}. N}. And based on E, the given video is clustered into N related action segments C = {c1, ..., c2}. N}, where action fragment cf Corresponding to frame block E f .

[0088] Action recognition: In this embodiment, the action recognition process uses the features of action segments as input to a classifier to obtain the action category, such as... Figure 5 As shown, the specific steps are as follows:

[0089] Step A: For each action segment c i The features are input into a pre-trained classifier;

[0090] Step B: Based on the input action fragment c f The probability of belonging to a category is used to assign an action label L = {l1, ..., l2}. N}

[0091] Preferably, in this embodiment, the given video length is 200 frames, i.e., f = 200.

[0092] Preferably, in this embodiment, the trajectory length of the iDT feature vector used is set to 15 frames.

[0093] Preferably, in this embodiment, the dimension of the encoding matrix Z is 600, i.e., d = 600.

[0094] Preferably, the specific calculation method for calculating the initial encoding matrix Z in step I is as follows:

[0095]

[0096] g s (·)=sin(·),

[0097]

[0098] in, I represents the first summation point of the encoding layer in a multi-layer neural network, where I is the identity matrix.

[0099] Preferably, in this embodiment, parameter C = 8.

[0100] Preferably, in step II, the parameters of the learning layer are updated. The specific method is as follows:

[0101]

[0102]

[0103] g t (·) = arctan(·).

[0104] in, yes The Moore-Penrose generalized inverse matrix, And C is a constant, I is the identity matrix, F is the frame sequence order information of the given video, and g t (·) and u j (·) are the activation function and the normalization function for the j-th summation point, respectively, and u j (F): This is the root mean square error.

[0105] Preferably, in step III, the error feedback data P is updated. j The specific calculation method is as follows:

[0106]

[0107]

[0108] Where u j -1 (·) is u j The inverse function of . For the Lth output error e L It satisfies the following inequalities:

[0109]

[0110] in And the sequence ||e1||, ||e2||, ..., ||e L || Gradually decreases and is bounded to 0.

[0111] Preferably, the specific calculation method for adding the summary point of the coding layer and updating the coding matrix Z in step IV is as follows:

[0112]

[0113]

[0114] g s -1 (·)=arcsin(·),

[0115]

[0116] Where t represents the number of iterations.

[0117] Preferably, the specific calculation method for optimizing the final encoding matrix Z in step V is as follows:

[0118]

[0119] Preferably, the specific calculation method for constructing the affinity map G in step i is as follows:

[0120]

[0121] Where Z = {z1, z2, ..., z} m}, i, j∈{1, 2, ..., m}, where m is the number of motion feature vectors. This represents an affinity diagram with an m×m dimension.

[0122] Preferably, the specific method for extracting the diagonal of the affinity map G and generating a one-dimensional signal in step ii is as follows:

[0123] ρ0=x(diag(G)),

[0124] Here, diag(·) is the function that extracts the diagonal from the affinity map G using the discrete Laplace operator K, and x(·) is the function that converts the two-dimensional signal into the one-dimensional signal ρ0. The discrete Laplace operator K is expressed as follows:

[0125]

[0126] diag(G) means that the affinity map G is processed by operator K to obtain G′, and then the diagonal of G′ is extracted to obtain the two-dimensional signal δ0.

[0127] Preferably, the specific method for smoothing the one-dimensional signal ρ0 using a Gaussian blur filter in step iii is as follows:

[0128] ρ = Gauss(ρ0),

[0129] Among them, Gauss(·) is a Gaussian blur filter used to filter noise in video.

[0130] Preferably, in this embodiment, the threshold is set to 24, i.e., ε = 24.

[0131] Preferably, the specific method for removing redundant frame boundaries of frame block T in step v is as follows:

[0132] First, in frame block T = {T1, T2, ..., T...} B In the context of comparison, the similarity S(T) between adjacent frame blocks is evaluated. t T t+1 ), t = 1, 2, 3, ..., B, if the similarity S(T) t T t+1 If the value is less than the threshold ε, then frame block T will be... t+1 Merge into T t Then, repeat the above steps until all frame blocks are traversed, thereby removing redundant frame boundaries; finally, obtain a new frame block E = {E1, E2, ..., E...} based on the merged frame blocks. N The above process can be represented by the following formula:

[0133] E = R(T)

[0134] Where R(.) is the operation function for removing redundant frame boundaries described above. And based on E, the given video is clustered into N related action segments C = {c1, ..., c...}. N}, where action fragment c f Corresponding to frame block E i .

[0135] Preferably, in this embodiment, the classifier used is a linear support vector machine (fSVM).

[0136] In this embodiment, motion features and the order information of video frame sequences are considered as features for learning the coding matrix in steps I to V. During the coding matrix learning process, motion features are input into a multi-layer neural network and the coding matrix is ​​learned in a sequence-order-aware coding manner, thereby effectively improving the performance of action clustering and classification.

[0137] In this embodiment, the algorithm for cutting affinity maps does not use the traditional normalized cutting (fNCuts) method, but instead uses a robust affinity map cutting algorithm. This algorithm removes noise from the video during the affinity map cutting process, thereby accurately obtaining the action segments.

Claims

1. A video action segmentation method based on frame sequence order-aware coding, characterized in that, Includes the following steps: Step 1: Extract the feature vector of the given video and encode the feature vector to obtain the motion features of the given video; Step 2: Based on the motion features of the given video, learn the encoding matrix to obtain the encoding matrix of the given video; Step 3: Construct an affinity map based on the encoding matrix of the given video; Step 4: Based on the affinity map, complete the action segmentation of the given video to obtain the action segments of the given video; The affinity map in step three is specifically as follows: in, Represents an m×m dimension affinity diagram. m is the number of motion feature vectors V. and These are the i-th and j-th elements of Z, respectively; Step four specifically involves: Step i: Extraction of the diagonal of the affinity map First, the affinity diagram is processed using the discrete Laplace operator K; Then, the diagonal of the affinity map is extracted and a one-dimensional signal ρ0 is generated; Step ii: Signal smoothing A Gaussian blur filter is used to smooth the one-dimensional signal ρ0 to filter out noise, thereby obtaining a smooth one-dimensional signal vector ρ. Step iii: Initial frame boundary detection First, in the one-dimensional signal vector ρ, the initial frame boundary where different actions exist is determined based on two adjacent peak values; Then, the given video is divided into B frame blocks using the initial frame boundaries, resulting in a frame block sequence T = {T1, T2, ..., T...} B }; Step iv: Redundant frame boundary removal First, in the frame block sequence T = {T1, T2, ..., T... B In this process, by comparing the similarity between adjacent frame blocks, frame blocks with a similarity less than a threshold ε are repeatedly merged, thereby removing redundant frame boundaries and re-dividing the given video into N frame blocks, resulting in a new frame block sequence E = {E1, E2, ..., E...}. N }; Then, based on E, the given video is divided into N action segments C = {c1,…,c...} N }, where action fragment c i Corresponding to frame block E i ; If the similarity between adjacent frame blocks is less than a set threshold, then the two frame blocks are merged, and the above steps are repeated until the similarity between adjacent frame blocks is not less than the set threshold.

2. The video action segmentation method based on frame sequence order-aware coding according to claim 1, characterized in that, Step one specifically involves: Step 1: Extract the iDT feature vector of a given video of length τ frames; Step 2: Encode the iDT feature vectors using the Fisher vector coding algorithm to obtain the motion features V of the given video. Where the i-th element of V This represents the i-th element of the encoded iDT feature vector, where m is the total number of elements in the motion feature V.

3. The video action segmentation method based on frame sequence order-aware coding according to claim 1, characterized in that, Step two specifically involves: Step I: Initial encoding matrix calculation First, define training vectors for m arbitrary samples. Where V k Let F represent the iDT feature vector of the k-th frame of a given video. k This represents the sequence order information of the k-th frame in a given video. Then, initialize the iteration count j = 1, and randomly initialize the parameters of the first summation point of the coding layer of the multi-layer neural network. It is the weight of the first summary point. It is the bias of the first summary point; Finally, using this summary point and given activation function g s (·), calculate the initial encoding matrix Where V represents the motion features of the given video; Step II: Learning Layer Parameter Update First, the encoding matrix of the j-th iteration... As input to the learning layers of a multi-layer neural network, Then, using the desired output video frame sequence order information F and the given activation function g... t (·) and normalization function u j (·), calculate the parameters of the learning layer in the j-th iteration. in These are the node weights of the learning layer during the j-th iteration. It is the node bias of the learning layer at the j-th iteration; Step III: Error Feedback Data Update Using the desired output video frame sequence order information F and the output F of the j-th iteration learning layer ' Update the output error e of the j-th iteration. j Then use the output error e j Update the error feedback data P for the j-th iteration. j ; Step IV: Encoding Matrix Update First, let the iteration number j = j + 1; Then, using the error feedback data P from the j-th iteration in step III... j Add a new summary point at the coding layer. Finally, the encoding matrix is ​​updated using the added summary points. Step V: Iterate and optimize First, repeat steps II through IVL-1 times until j = L, thus summing the set of parameters for the point. The optimal parameters for the coding layer are determined; where L is the total number of iterations. Then, by removing the first summary point parameter The influence of randomness is considered, and optimization is used to obtain the final encoding matrix Z.

4. The video action segmentation method based on frame sequence order-aware coding according to claim 3, characterized in that, Parameters of the learning layer in the j-th iteration The specific calculation method is as follows: in, yes The Moore-Penrose generalized inverse matrix, And C is a constant, I is the identity matrix, F is the frame sequence order information of the given video, and g t (·) and u j (·) are the activation function and the normalization function for the j-th summation point, respectively, and u j (F): This is the root mean square error.

5. A video action segmentation method based on frame sequence order-aware coding according to claim 3, characterized in that, The error feedback data P of the j-th iteration j The specific calculation method is as follows: Wherein, the output error of the j-th iteration u j -1 (·) is u j The inverse function of (·).

6. The video action segmentation method based on frame sequence order-aware coding according to claim 3, characterized in that, The specific calculation method for the final encoding matrix Z is as follows: in, 7. A video action recognition method based on frame sequence order-aware coding, characterized in that, The method specifically includes the following steps: Step A: Based on the segmentation method as described in any one of claims 1 to 6, segment the video to be identified into N action segments; Step B: Input the features of each action segment into a pre-trained classifier; Step C: Assign action labels based on the probability of each action fragment belonging to its category.

Images