Method and apparatus for action time-series data segmentation based on multi-view features

Abstract

This invention relates to the field of data processing technology, providing a method and apparatus for segmenting action time-series data based on multi-view features. This method fully utilizes the complementarity between multi-view data and introduces a latent representation learning mechanism, enabling features from different views to share a unified latent representation, thereby more comprehensively describing the subspace structure of the action time-series data. Simultaneously, introducing two temporal regularization terms into the objective function ensures the continuity of the latent representation and subspace representation of adjacent data frames in the action time-series data, enhancing temporal consistency between adjacent data frames and making subspace partitioning more stable, thus achieving effective segmentation of the action time-series data. Compared to traditional time-series data segmentation methods, this method can more effectively combine the multi-view features and time-series characteristics of action time-series data, improving the segmentation effect of action time-series data through collaborative optimization of latent representation and subspace representation.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and apparatus for segmenting action time-series data based on multi-view features. Background Technology

[0002] Currently, clustering is used for data segmentation by dividing a set of original unlabeled datasets into several different clusters according to certain features or rules. Due to the high dimensionality and nonlinearity of data, subspace clustering methods are usually used to handle high-dimensional data clustering problems.

[0003] Traditional subspace clustering generally consists of two steps: (1) Connecting all data points based on the self-representation of the data to obtain a coefficient matrix that can describe the relationship between all data points. (2) Using the coefficient matrix to construct an affinity matrix and placing it in the spectral clustering algorithm to obtain the data segmentation result. Traditional subspace clustering has achieved some success in the segmentation of general data, but these methods have not explored the temporal characteristics of time series data in depth, and the accuracy is significantly reduced when segmenting time series data. In order to solve the above problems, the sequence subspace clustering method, which adds temporal constraints to the coefficient matrix on the traditional subspace clustering method, has emerged.

[0004] However, these methods are all based on single-view design, modeling only from a specific perspective. Their temporal adjacency graph construction depends entirely on the characteristics of a single view, making them not only susceptible to interference from perspective limitations but also difficult to adapt to diverse dynamic changes. Summary of the Invention

[0005] This invention provides a method and apparatus for segmenting action time sequence data based on multi-view features, in order to overcome the deficiencies existing in the prior art.

[0006] This invention provides a method for segmenting action time-series data based on multi-view features, including:

[0007] Acquire action timing data;

[0008] Multiple features are extracted from the action time series data to construct a multi-view feature matrix. The multi-view feature matrix is ​​represented as a product of a projection matrix and a latent representation matrix. The latent representation matrix is ​​self-expressive modeled based on the sparse subspace representation matrix obtained by subspace reconstruction. The projection matrix satisfies orthogonal constraints.

[0009] An objective function is established; the objective function includes an error term, a first temporal regularization term of the latent representation matrix, and a second temporal regularization term of the sparse subspace representation matrix; the constraints of the objective function include the product form, the self-expressive modeling result of the latent representation matrix, and the orthogonality constraint.

[0010] The objective function is solved to obtain the objective value of the sparse subspace representation matrix. Based on the objective value, a symmetric similarity matrix is ​​constructed, and the action time series data is segmented based on the symmetric similarity matrix.

[0011] According to the action time-series data segmentation method based on multi-view features provided by the present invention, the step of solving the objective function to obtain the objective value of the sparse subspace representation matrix includes:

[0012] Construct the augmented Lagrangian function of the objective function;

[0013] Within the alternating direction multiplier framework, the augmented Lagrangian function is solved to obtain the target value of the sparse subspace representation matrix.

[0014] According to the present invention, a method for segmenting action-time data based on multi-view features, wherein solving the augmented Lagrangian function within an alternating direction multiplier framework to obtain the target value of the sparse subspace representation matrix includes:

[0015] In the alternating direction multiplier framework, the problem of solving the augmented Lagrangian function is decomposed into subproblems concerning the projection matrix, the latent representation matrix, the sparse subspace representation matrix, and the comprehensive error in the error term;

[0016] The target value is obtained by simultaneously iterating through each of the sub-problems.

[0017] According to the present invention, a method for segmenting action time-series data based on multi-view features, wherein the step of synchronously iteratively solving each of the sub-problems to obtain the target value includes:

[0018] For the first subproblem concerning the projection matrix, which is an orthogonal Prouk problem, the optimal solution to the projection matrix is ​​obtained by singular value decomposition.

[0019] For the second subproblem concerning the latent representation matrix, the subgradient of the second subproblem is calculated, and the optimal solution of the latent representation matrix is ​​obtained based on the subgradient update algorithm;

[0020] For the third subproblem concerning the sparse subspace representation matrix, the third subproblem is solved based on the near-end gradient method to obtain the optimal solution of the sparse subspace representation matrix as the objective value;

[0021] For the fourth subproblem concerning the overall error, each column of the overall error is solved individually to obtain the optimal solution for the overall error.

[0022] According to the present invention, an action time-series data segmentation method based on multi-view features is provided, wherein the objective function is expressed based on the following formula:

[0023] ;

[0024] Where P is the projection matrix, H is the latent representation matrix, and Z is the sparse subspace representation matrix. For potential representation error, Let E represent the error in the subspace. and The overall error is denoted by R, which is the difference matrix between adjacent data frames in the action timing data. and All of these are hyperparameters. For L 21 Norm, It is an L1 norm. It is the Frobenious norm. The weight matrix is ​​a diagonal matrix, determined by the similarity between the latent representations of adjacent data frames in the action time series data. For the error term, Let be the sparsity constraint term of the sparse subspace representation matrix. This is the second time-series regularization term. This is the first time-series regularization term. Let X represent the constraint conditions, X be the multi-view feature matrix, and I be the identity matrix.

[0025] According to the present invention, a method for segmenting action time-series data based on multi-view features is provided, wherein the diagonal elements of the diagonal weight matrix are... And there are:

[0026] ;

[0027] in, The similarity weight represents the relationship between the (t+1)th data frame and the potential representation of the tth data frame in the action time sequence data. This is the potential representation of the (t+1)th data frame in the action timing data. This is the potential representation of the t-th data frame in the action timing data. For hyperparameters, It is an L2 norm.

[0028] According to the present invention, a method for segmenting action time-series data based on multi-view features is provided, wherein segmenting the action time-series data based on the symmetric similarity matrix includes:

[0029] Based on the symmetric similarity matrix, the action time series data is segmented using a spectral clustering algorithm.

[0030] The present invention also provides an action time-series data segmentation device based on multi-view features, comprising:

[0031] The data acquisition module is used to acquire action timing data;

[0032] The representation module is used to extract multiple features from the action time series data, construct a multi-view feature matrix, and represent the multi-view feature matrix as a product of a projection matrix and a latent representation matrix. The latent representation matrix is ​​self-expressive modeled based on the sparse subspace representation matrix obtained by subspace reconstruction.

[0033] A function establishment module is used to establish an objective function; the objective function includes an error term, a first temporal regularization term of the latent representation matrix, and a second temporal regularization term of the sparse subspace representation matrix; the constraints of the objective function include the product form, the self-expressive modeling result of the latent representation matrix, and the orthogonality constraint of the projection matrix;

[0034] The function solving module is used to solve the objective function to obtain the objective value of the sparse subspace representation matrix, and construct a symmetric similarity matrix based on the objective value, and segment the action time series data based on the symmetric similarity matrix.

[0035] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the action timing data segmentation method based on multi-view features as described above.

[0036] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the action-time data segmentation method based on multi-view features as described above.

[0037] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the action-time data segmentation method based on multi-view features as described above.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] This invention provides a method and apparatus for segmenting action time-series data based on multi-view features. This method fully utilizes the complementarity between multi-view data and introduces a latent representation learning mechanism, enabling features from different views to share a unified latent representation, thereby more comprehensively describing the subspace structure of the action time-series data. Simultaneously, introducing two temporal regularization terms into the objective function ensures the continuity of the latent representation and subspace representation between adjacent data frames in the action time-series data, enhancing temporal consistency between adjacent data frames and making subspace partitioning more stable, thus achieving effective segmentation of the action time-series data. Compared to traditional time-series data segmentation methods, this method can more effectively combine the multi-view features and time-series characteristics of action time-series data, improving the segmentation effect by collaboratively optimizing the latent representation and subspace representation. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on the drawings described below without creative effort.

[0041] Figure 1 This is one of the flowcharts illustrating the action time-series data segmentation method based on multi-view features provided by the present invention;

[0042] Figure 2 This is the second flowchart of the action time sequence data segmentation method based on multi-view features provided by the present invention;

[0043] Figure 3 This is a schematic diagram of the action time sequence data segmentation device based on multi-view features provided by the present invention;

[0044] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0046] Because existing subspace clustering algorithms are all based on single-view design and model only from a specific perspective, their temporal adjacency graph construction relies entirely on the features of a single view. This makes them not only susceptible to interference from viewpoint limitations but also difficult to adapt to diverse dynamic changes. Therefore, this invention provides a method for action temporal data segmentation based on multi-view features.

[0047] Figure 1 This is a flowchart illustrating a method for segmenting action time-series data of multi-view features provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0048] S1, acquire action timing data;

[0049] S2, extract multiple features from the action time series data, construct a multi-view feature matrix, and represent the multi-view feature matrix as a product of a projection matrix and a latent representation matrix. The latent representation matrix performs self-expression modeling based on the sparse subspace representation matrix obtained by subspace reconstruction. The projection matrix satisfies orthogonal constraints.

[0050] S3, Establish the objective function; the objective function includes an error term, a first temporal regularization term of the latent representation matrix, and a second temporal regularization term of the sparse subspace representation matrix; the constraints of the objective function include the product form, the self-expressive modeling result of the latent representation matrix, and the orthogonality constraint;

[0051] S4. Solve the objective function to obtain the objective value of the sparse subspace representation matrix, and construct a symmetric similarity matrix based on the objective value. Then, segment the action time series data based on the symmetric similarity matrix.

[0052] Specifically, the action time sequence data segmentation method based on multi-view features provided in this embodiment of the invention is executed by an action time sequence data segmentation device based on multi-view features. This device can be configured in a computer, which can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0053] First, step S1 is executed to acquire motion timing data. This motion timing data can be images or videos continuously captured by an image acquisition device during the action of the subject. The subject can be a real person, a robot, etc. The type of action performed by the subject depends on the scene in which the subject is located, and can include at least one of several types of actions such as grasping, walking, running, and jumping.

[0054] Then, step S2 is executed to extract multiple types of features from the action time-series data and construct a multi-view feature matrix. The action time-series data can include multiple data frames, and multiple types of features can be extracted from each data frame. Each type of feature is a view feature, and the multi-view feature matrix can be constructed using the multiple types of features from each data frame. In other words, the multi-view feature matrix includes multiple types of features from each data frame of the action time-series data.

[0055] For each view That is, for each type of feature, extract the feature vector of the t-th data frame of the action time sequence data. The feature matrix of view v is formed. Where n is the total number of data frames for the action timing data. For view v, the feature dimension is...

[0056] The feature matrices of all views extracted from each data frame of the action timing data are vertically stacked and integrated to obtain a unified multi-view feature matrix X, i.e.:

[0057] ;

[0058] Where D is the total feature dimension of all views, and V is the number of all views, i.e. the number of classes of all features.

[0059] Understandably, multiple features can include local texture features, keypoint features, and edge orientation features. Local texture features can be captured from action time-series data using local binary patterns (LBP), keypoint features can be extracted using the scale-invariant feature transform (SIFT) algorithm, and edge orientation features can be captured using image feature description methods based on histogram of oriented gradients (HOG).

[0060] Since each data frame of the action timing data has V views, it is assumed that the V views of each data frame in the action timing data all originate from the same latent representation matrix. This approach aims to uncover the inherent consistency among multiple views. The latent representation matrix H can be composed of the latent representations of each data frame, with each data frame's latent representation serving as a column in the latent representation matrix, and d representing the dimension of the latent representation in each data frame.

[0061] Multi-view feature matrices can be projected through the matrix. The latent representation matrix is ​​processed to obtain the multi-view feature matrix, which can be expressed as the product of the projection matrix and the latent representation matrix, as shown in the formula:

[0062] ;

[0063] Wherein, the projection matrix P satisfies the orthogonality constraint. To avoid potential spatial dimensional redundancy, The latent representation error is used to capture noise and outliers in action time-series data. After obtaining the latent representation matrix, further subspace reconstruction is performed on the latent representation matrix to obtain a more robust sparse subspace representation matrix. The latent representation matrix H is modeled through the sparse subspace representation matrix Z, as shown in the following formula:

[0064] ;

[0065] in, The subspace represents the error, indicating the deviation that exists in the self-expressive modeling process.

[0066] Next, step S3 is executed to establish the objective function. This objective function includes an error term, a first temporal regularization term of the latent representation matrix, and a second temporal regularization term of the sparse subspace representation matrix. The constraints of this objective function include a product form, the self-expressive modeling result of the latent representation matrix, and orthogonal constraints on the projection matrix. Here, the self-expressive modeling result of the latent representation matrix is ​​the formula for self-expressive modeling.

[0067] In the objective function, since the temporal characteristics of action-series data are key prior knowledge for data segmentation, adjacent data frames in action-series data usually have high similarity. Therefore, their corresponding latent representations should also be consistent, thereby increasing the probability of them belonging to the same subspace. Based on this, a first temporal regularization term is introduced into the latent representation matrix to constrain the latent representations of adjacent data frames, making their changes smooth. The formula is as follows:

[0068] ;

[0069] in, Let be the latent representation matrix. It is the difference matrix between adjacent data frames:

[0070] ;

[0071] ;

[0072] in, Let H be the t-th column of the latent representation matrix, which represents the latent representation of the t-th data frame. use The purpose of norms is to encourage similarity between consecutive columns of the latent representation matrix H, ensuring temporal consistency.

[0073] Since the segmentation result depends on the sparse subspace representation matrix Z, in order to further maintain the temporal consistency between the latent representation and the subspace representation, this embodiment of the invention provides a second temporal regularization term for the sparse subspace representation matrix. This second temporal regularization term is a structured sparse temporal regularization term. In this embodiment of the invention, the objective function can be expressed based on the following formula:

[0074] ;

[0075] Where P is the projection matrix, H is the latent representation matrix, and Z is the sparse subspace representation matrix. For potential representation error, Let E represent the error in the subspace. and The overall error is denoted by R, which is the difference matrix between adjacent frames in the action timing data. and All of these are hyperparameters. For L 21 Norm, It is an L1 norm. It is the Frobenious norm. The weight matrix is ​​a diagonal matrix, determined by the similarity between the latent representations of adjacent data frames in the action time series data. For the error term, Let be the sparsity constraint term of the sparse subspace representation matrix. This is the second time-series regularization term. This is the first time-series regularization term. For structured sparse time regularization, Let X represent the constraint conditions, X be the multi-view feature matrix, and I be the identity matrix.

[0076] Among them, the comprehensive error can be obtained through and Obtained by vertical splicing, the overall error is assessed using... Norms can enhance the sparsity of columns in the comprehensive error and suppress the influence of outliers on the clustering results. A norm is used for the sparse subspace representation matrix. Norms can improve the sparsity of sparse subspace representation matrices, ensuring that each data frame is associated with only a few data frames, thus learning a tighter subspace structure. Orthogonality constraints ensure that the projection matrix is ​​an orthogonal transformation matrix. Hyperparameters All are greater than 0, used to balance the importance of each term in the objective function.

[0077] Finally, step S4 is executed, which solves the objective function within the Alternating Direction Method of Multipliers (ADMM) framework, yielding the objective value of the sparse subspace representation matrix. This objective value can have a block-corner symmetric structure, with each diagonal block corresponding to a different subspace, providing high-quality data input for subsequent data segmentation.

[0078] Furthermore, by representing the target value of the matrix using a sparse subspace, a symmetric similarity matrix is ​​constructed, which can be expressed as:

[0079] ;

[0080] in, The objective value of the matrix is ​​represented in a sparse subspace.

[0081] Using symmetric similarity matrices, spectral clustering algorithms can be used to segment action time series data and obtain clustering results.

[0082] The action time-series data segmentation method based on multi-view features provided in this invention extracts multiple types of features from action time-series data, constructs a multi-view feature matrix, and represents the multi-view feature matrix as the product of a projection matrix and a latent representation matrix. The latent representation matrix performs self-expression modeling based on the sparse subspace representation matrix obtained from subspace reconstruction. Finally, an objective function is constructed and solved. This method fully utilizes the complementarity between multi-view data and introduces a latent representation learning mechanism, enabling features from different views to share a unified latent representation, thereby more comprehensively describing the subspace structure of the action time-series data. Simultaneously, the introduction of two temporal regularization terms into the objective function ensures the continuity of the latent representation and subspace representation of adjacent data frames in the action time-series data, enhancing temporal consistency between adjacent data frames, making subspace partitioning more stable, and thus achieving effective segmentation of the action time-series data. Compared to traditional time-series data segmentation methods, this method can more effectively combine the multi-view features and time-series characteristics of action time-series data, improving the segmentation effect of action time-series data through collaborative optimization of latent representation and subspace representation.

[0083] Based on the above embodiments, solving the objective function to obtain the objective value of the sparse subspace representation matrix includes:

[0084] Construct the augmented Lagrangian function of the objective function;

[0085] Within the alternating direction multiplier framework, the augmented Lagrangian function is solved to obtain the target value of the sparse subspace representation matrix.

[0086] Specifically, in solving the objective function, we can first establish the augmented Lagrangian function of the objective function. The augmented Lagrangian function can be expressed by the following formula:

[0087] ;

[0088] in, and All are Lagrange multipliers. For penalty parameters, This represents the inner product of matrices, which is the product of corresponding elements of two matrices and the sum of the results.

[0089] Subsequently, within the alternating direction multiplier framework, the augmented Lagrangian function is solved to obtain the objective value of the sparse subspace representation matrix.

[0090] The solution for the augmented Lagrange function includes not only the objective value of the sparse subspace representation matrix, but also the objective values ​​of the projection matrix, the latent representation matrix, the synthesis error, and the values ​​of the Lagrange multipliers. Specifically, the solution can be expressed as:

[0091] ;

[0092] in, The target value of the projection matrix, To represent the target value of the matrix, The target value for the overall error is... and Lagrange multipliers and The value of .

[0093] Based on the above embodiments, the step of solving the augmented Lagrangian function within the alternating direction multiplier framework to obtain the target value of the sparse subspace representation matrix includes:

[0094] In the alternating direction multiplier framework, the problem of solving the augmented Lagrangian function is decomposed into subproblems concerning the projection matrix, the latent representation matrix, the sparse subspace representation matrix, and the comprehensive error in the error term;

[0095] The target value is obtained by simultaneously iterating through each of the sub-problems.

[0096] Specifically, when solving the augmented Lagrangian function, the problem can be decomposed into a first subproblem concerning the projection matrix, a second subproblem concerning the latent representation matrix, a third subproblem concerning the sparse subspace representation matrix, and a fourth subproblem concerning the comprehensive error in the error term within the framework of alternating direction multipliers.

[0097] The optimization function for the first subproblem can be expressed as:

[0098] ;

[0099] The optimization function for the second subproblem can be expressed as:

[0100] ;

[0101] The optimization function for the third subproblem can be expressed as:

[0102] ;

[0103] The optimization function for the fourth subproblem can be expressed as:

[0104] ;

[0105] in, .

[0106] Subsequently, by simultaneously iterating through each subproblem, the target value of the desired sparse subspace representation matrix can be obtained.

[0107] Based on the above embodiments, the step of synchronously iteratively solving each of the sub-problems to obtain the target value includes:

[0108] For the first subproblem concerning the projection matrix, which is an orthogonal Prouk problem, the optimal solution to the projection matrix is ​​obtained by singular value decomposition.

[0109] For the second subproblem concerning the latent representation matrix, the subgradient of the second subproblem is calculated, and the optimal solution of the latent representation matrix is ​​obtained based on the subgradient update algorithm;

[0110] For the third subproblem concerning the sparse subspace representation matrix, the third subproblem is solved based on the near-end gradient method to obtain the optimal solution of the sparse subspace representation matrix as the objective value;

[0111] For the fourth subproblem concerning the overall error, each column of the overall error is solved individually to obtain the optimal solution for the overall error.

[0112] Specifically, in solving the first subproblem, the first subproblem is an orthogonal Pluke problem. For an orthogonal Pluke problem, given a function... If the constraints are satisfied Its optimal solution is .in, and They are The left and right singular matrices after SVD decomposition.

[0113] Based on this, the optimal solution of the projection matrix can be obtained through Singular Value Decomposition (SVD), that is:

[0114] ;

[0115] in, They are The left and right singular matrices after singular value decomposition.

[0116] When solving the second subproblem, because Since it is not differentiable at 0, we need to first find the subgradient of the second subproblem, that is:

[0117] ;

[0118] in, This is the subgradient.

[0119] Subsequently, the optimal solution for the latent representation matrix is ​​obtained using the subgradient update algorithm. The update formula for the latent representation matrix in the subgradient update algorithm is as follows:

[0120] ;

[0121] Where k represents the update at the k-th step. The update step size parameter for the latent representation matrix. This represents the update result of the latent representation matrix at step k. This represents the update result of the latent representation matrix at step k+1.

[0122] When solving the third subproblem, the proximal gradient method can be used to solve the third subproblem and obtain the optimal solution of the sparse subspace representation matrix as the objective value. Here, auxiliary variables are defined. ,make Then the optimization function for the third subproblem can be rewritten as:

[0123] .

[0124] Because the optimization function of the third subproblem also contains non-smoothness. Norm terms and smooth quadratic terms Therefore, the proximal gradient method is adopted, using gradient descent combined with soft threshold update to optimize the third subproblem. Find the gradient:

[0125] ;

[0126] in, for The gradient.

[0127] Subsequently, the sparse subspace representation matrix is ​​iteratively updated using the soft thresholding operator:

[0128] ;

[0129] in, Let be the update step size parameter for the sparse subspace representation matrix. This is a soft threshold operator.

[0130] When solving the fourth subproblem, each column of the comprehensive error can be solved separately to obtain the optimal solution for the comprehensive error. When solving each column of the comprehensive error separately, the update formula for each column can be expressed as:

[0131] .

[0132] in, The i-th column of the comprehensive error, for The i-th column.

[0133] Based on the above embodiments, the diagonal elements of the diagonal weight matrix in the objective function are: And there are: ;

[0134] in, The similarity weight represents the relationship between the (t+1)th data frame and the potential representation of the tth data frame in the action time sequence data. This is the potential representation of the (t+1)th data frame in the action timing data. This is the potential representation of the t-th data frame in the action timing data. For hyperparameters, It is an L2 norm.

[0135] Specifically, the difference between the sparse subspace representation of the (t+1)th data frame and the tth data frame in the action timing data can be expressed as:

[0136] ;

[0137] in, Let be the sparse subspace representation of the (t+1)th data frame. Let be the sparse subspace representation of the t-th data frame.

[0138] Let be the similarity weight between the (t+1)th data frame and the latent representation of the tth data frame in the action time sequence data, and it can be expressed by the following formula:

[0139] ;

[0140] in, These are hyperparameters used for adjustment. , This is the potential representation of the (t+1)th data frame in the action timing data. Let be the potential representation of the t-th data frame in the action timing data. and When similar, Take a larger value, thereby strengthening the effect. and The similarity between them; if and If the difference is large, then Take a smaller value to allow for greater subspace variation.

[0141] Due to vectors The square of the norm is equal to the sum of the squares of all elements in a column; the square of the Frobenious norm of a matrix is ​​equal to the sum of the squares of all elements in the matrix. The formula can be equivalent to:

[0142] ;

[0143] in, It is a diagonal weight matrix, and its diagonal elements are .

[0144] like Figure 2 As shown, based on the above embodiments, this embodiment of the invention provides a method for segmenting action time-series data based on multi-view features, including:

[0145] Acquire action timing data; the action timing data includes images of different types of actions, such as images of walking, running, and jumping;

[0146] Extract multiple types of features from action time-series data and construct a multi-view feature matrix;

[0147] The multi-view feature matrix is ​​represented as a product of the projection matrix and the latent representation matrix;

[0148] Subspace reconstruction is performed on the latent representation matrix to obtain the sparse subspace representation matrix;

[0149] Self-expressive modeling of the sparse subspace representation matrix;

[0150] By combining the product form of the projection matrix and the latent representation matrix, the self-expressive modeling results of the latent representation matrix, and the orthogonality constraint of the projection matrix, an objective function is established.

[0151] Solving the objective function yields the objective value of the sparse subspace representation matrix;

[0152] Construct symmetric similar matrices by representing the target value of a matrix using a sparse subspace;

[0153] Using a symmetric similarity matrix and a spectral clustering algorithm, action time-series data is segmented to obtain segmentation results. In this segmentation result, images containing the same type of action are grouped into one category, resulting in images of walking, running, and jumping, respectively. Images containing the same type of action are then aligned.

[0154] like Figure 3 As shown, based on the above embodiments, this embodiment of the invention provides an action time sequence data segmentation device based on multi-view features, comprising:

[0155] Data acquisition module 31 is used to acquire action timing data;

[0156] The representation module 32 is used to extract multiple features from the action time series data, construct a multi-view feature matrix, and represent the multi-view feature matrix as a product of a projection matrix and a latent representation matrix. The latent representation matrix is ​​self-expressive modeled based on the sparse subspace representation matrix obtained by subspace reconstruction.

[0157] Function establishment module 33 is used to establish an objective function; the objective function includes an error term, a first temporal regularization term of the latent representation matrix, and a second temporal regularization term of the sparse subspace representation matrix; the constraints of the objective function include the product form, the self-expressive modeling result of the latent representation matrix, and the orthogonality constraint of the projection matrix;

[0158] The function solving module 34 is used to solve the objective function to obtain the objective value of the sparse subspace representation matrix, and construct a symmetric similarity matrix based on the objective value, and segment the action time series data based on the symmetric similarity matrix.

[0159] Based on the above embodiments, the action time-series data segmentation device based on multi-view features provided in this embodiment of the invention, wherein the function solving module is specifically used for:

[0160] Construct the augmented Lagrangian function of the objective function;

[0161] Within the alternating direction multiplier framework, the augmented Lagrangian function is solved to obtain the target value of the sparse subspace representation matrix.

[0162] Based on the above embodiments, the action time-series data segmentation device based on multi-view features provided in this embodiment of the invention, wherein the function solving module is specifically used for:

[0163] In the alternating direction multiplier framework, the problem of solving the augmented Lagrangian function is decomposed into subproblems concerning the projection matrix, the latent representation matrix, the sparse subspace representation matrix, and the comprehensive error in the error term;

[0164] The target value is obtained by simultaneously iterating through each of the sub-problems.

[0165] Based on the above embodiments, the action time-series data segmentation device based on multi-view features provided in this embodiment of the invention, wherein the function solving module is specifically used for:

[0166] For the first subproblem concerning the projection matrix, which is an orthogonal Prouk problem, the optimal solution to the projection matrix is ​​obtained by singular value decomposition.

[0167] For the second subproblem concerning the latent representation matrix, the subgradient of the second subproblem is calculated, and the optimal solution of the latent representation matrix is ​​obtained based on the subgradient update algorithm;

[0168] For the third subproblem concerning the sparse subspace representation matrix, the third subproblem is solved based on the near-end gradient method to obtain the optimal solution of the sparse subspace representation matrix as the objective value;

[0169] For the fourth subproblem concerning the overall error, each column of the overall error is solved individually to obtain the optimal solution for the overall error.

[0170] Based on the above embodiments, the action time-series data segmentation device based on multi-view features provided in this embodiment of the invention has an objective function expressed by the following formula:

[0171] ;

[0172] Where P is the projection matrix, H is the latent representation matrix, and Z is the sparse subspace representation matrix. For potential representation error, Let E represent the error in the subspace. and The overall error is denoted by R, which is the difference matrix between adjacent data frames in the action timing data. and All of these are hyperparameters. For L 21 Norm, It is an L1 norm. It is the Frobenious norm. The weight matrix is ​​a diagonal matrix, determined by the similarity between the latent representations of adjacent data frames in the action time series data. For the error term, Let be the sparsity constraint term of the sparse subspace representation matrix. This is the second time-series regularization term. This is the first time-series regularization term. Let X represent the constraint conditions, X be the multi-view feature matrix, and I be the identity matrix.

[0173] Based on the above embodiments, the action time-series data segmentation device based on multi-view features provided in this embodiment of the invention has the diagonal elements of the diagonal weight matrix as follows: And there are:

[0174] ;

[0175] in, The similarity weight represents the relationship between the (t+1)th data frame and the potential representation of the tth data frame in the action time sequence data. This is the potential representation of the (t+1)th data frame in the action timing data. This is the potential representation of the t-th data frame in the action timing data. For hyperparameters, It is an L2 norm.

[0176] Based on the above embodiments, the action time-series data segmentation device based on multi-view features provided in this embodiment of the invention, wherein the function solving module is specifically used for:

[0177] Based on the aforementioned symmetric similarity matrix, a spectral clustering algorithm is applied to segment the action time-series data. (Walk)

[0178] Specifically, the functions of each module in the action time sequence data segmentation device based on multi-view features provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0179] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the action timing data segmentation method based on multi-view features provided in the above embodiments.

[0180] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0181] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the action time-series data segmentation method based on multi-view features provided in the above embodiments.

[0182] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the action-time data segmentation method based on multi-view features provided in the above embodiments.

[0183] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0184] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0185] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for action temporal data segmentation based on multi-view features, characterized in that, The method comprises the following steps: acquiring action time series data; the action time series data comprises different types of action images, including walking images, running images and jumping images; extracting multi-class features from the action time series data, constructing a multi-view feature matrix, representing the multi-view feature matrix in the form of a product of a projection matrix and a latent representation matrix, and modeling the latent representation matrix based on a sparse subspace representation matrix obtained through subspace reconstruction for self-expression; the projection matrix satisfies an orthogonal constraint; establishing a target function; the target function comprises an error term, a first time series regularization term of the latent representation matrix and a second time series regularization term of the sparse subspace representation matrix; the constraint conditions of the target function comprise the product form, the self-expression modeling result of the latent representation matrix and the orthogonal constraint; solving the target function to obtain a target value of the sparse subspace representation matrix, and based on the target value, constructing a symmetric similarity matrix, and based on the symmetric similarity matrix, applying a spectral clustering algorithm to segment the action time series data; the target function is represented based on the following formula: ； where P is the projection matrix, H is the latent representation matrix, Z represents the sparse subspace representation matrix, is the latent representation error, is the subspace representation error, E represents is the combined error of , R is the difference matrix between adjacent data frames in the action time series data, and are hyperparameters, is the L 21 norm, is the L1 norm, is the Frobenious norm, is the diagonal weight matrix determined by the similarity between the latent representations of adjacent data frames in the action time series data, is the error term, is the sparsity constraint term of the sparse subspace representation matrix, is the second time series regularization term, is the first time series regularization term, represents the constraint condition, X is the multi-view feature matrix, and I is the identity matrix. The diagonal elements of the diagonal weight matrix are and have: ； wherein, denotes a similarity weight between the latent representation of the t+1th data frame and the tth data frame in the action time series data, is a latent representation of the t+1th data frame in the action time series data, is a latent representation of the tth data frame in the action time series data, is a hyperparameter, is an L2 norm.

2. The method of claim 1, wherein, the solving of the target function to obtain the target value of the sparse subspace representation matrix comprises: establishing an augmented Lagrangian function of the target function; solving the augmented Lagrangian function in an alternating direction multiplier framework to obtain the target value of the sparse subspace representation matrix.

3. The method of claim 2, wherein, the solving of the augmented Lagrangian function in the alternating direction multiplier framework to obtain the target value of the sparse subspace representation matrix comprises: in the alternating direction multiplier framework, decomposing the solving problem of the augmented Lagrangian function into sub-problems about the projection matrix, the latent representation matrix, the sparse subspace representation matrix and the comprehensive error in the error term; synchronously solving each of the sub-problems to obtain the target value.

4. The method of claim 3, wherein, the synchronous solving of each of the sub-problems to obtain the target value comprises: for a first sub-problem about the projection matrix, the first sub-problem is an orthogonal Procrustes problem, and the optimal solution of the projection matrix is obtained through singular value decomposition; for a second sub-problem about the latent representation matrix, solving a sub-gradient of the second sub-problem, and based on a sub-gradient update algorithm, solving the optimal solution of the latent representation matrix; for a third sub-problem about the sparse subspace representation matrix, based on a proximal gradient method, solving the third sub-problem to obtain the optimal solution of the sparse subspace representation matrix as the target value; for a fourth sub-problem about the comprehensive error, separately solving each column of the comprehensive error to obtain the optimal solution of the comprehensive error.

5. A motion timing data segmentation device based on multi-view features, characterized in that, application of the multi-view feature-based action time series data segmentation method according to any one of claims 1-4 comprises: a data acquisition module for acquiring action time series data; The representation module is configured to extract multi-class features from the action time series data, construct a multi-view feature matrix, and represent the multi-view feature matrix in a product form of a projection matrix and a latent representation matrix, wherein the latent representation matrix is modeled based on a sparse subspace representation matrix obtained through subspace reconstruction and self-expression. The function establishing module is configured to establish a target function, wherein the target function comprises an error term, a first time series regularization term of the latent representation matrix, and a second time series regularization term of the sparse subspace representation matrix, and the constraint condition of the target function comprises the product form, the self-expression modeling result of the latent representation matrix, and the orthogonal constraint of the projection matrix. The function solving module is configured to solve the target function to obtain a target value of the sparse subspace representation matrix, construct a symmetric similar matrix based on the target value, and segment the action time series data based on the symmetric similar matrix.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the action time series data segmentation method based on multi-view features according to any one of claims 1-4.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the action time series data segmentation method based on multi-view features according to any one of claims 1-4.

Images