A 3D human motion prediction method based on correlation multi-scale graph clustering network

Abstract

The present invention discloses a 3D human motion prediction method based on a correlation multi-scale graph clustering network, which belongs to the field of image processing technology and includes the following steps: S1, the correlation multi-scale graph clustering network CMGC includes an adaptive multi-scale computing unit AMCU branch and a motion decomposition modeling module DMMB branch; S2, the historical motion sequence enters the adaptive multi-scale computing unit AMCU branch after discrete cosine transformation and embedding; S3, the historical motion sequence enters the motion decomposition modeling module DMMB branch; S4, the correlation multi-scale graph clustering network CMGC outputs the predicted future motion sequence. The present invention adopts the above-mentioned 3D human motion prediction method based on a correlation multi-scale graph clustering network, significantly improves the prediction accuracy, enhances the motion feature extraction capability, compensates for the signal loss caused by discrete cosine transform domain modeling, aggregates comprehensive features to effectively learn joint dynamics, and constructs an adaptive multi-scale graph to solve the problem of insufficient flexibility of fixed multi-scale graphs in different movements.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a 3D human motion prediction method based on a correlation multi-scale graph clustering network. Background Art

[0002] Existing human motion prediction technologies define the human body graph structure based on the skeleton structure. This approach limits the modeling of dynamic correlations between joints. In space, the distribution of human joint motion is non-Euclidean, without consistent distance constraints or clear semantic associations between joints, that is, the joint relationships are dynamically changing. In the time dimension, the cascaded rotational relationships between joints and the influence of external forces make the prediction of joint trajectories complicated. In order to better learn the spatiotemporal correlation representation of joint motion, the existing technology proposes a neural network structure for multi-scale graph modeling, which explicitly utilizes the relationships or constraints between different body parts, thereby improving the ability to model the spatiotemporal and temporal human joint motion. These multi-scale graphs are trainable and decomposable, enabling the network to effectively model the dynamic correlations between skeletal joints.

[0003] However, these techniques rely on fixed multi-scale graphs to create coarse scales by artificially aggregating adjacent human joints. This approach imposes strict kinematic constraints on multi-scale joint modeling, which weakens the flexibility of the network to capture implicit relationships. These implicit relationships vary in different movements, but fixed multi-scale graphs cannot adaptively adjust feature extraction methods to capture these implicit relationships when modeling different movements.

[0004] In addition, existing technologies use DCT to convert joint motion into the frequency domain to achieve smoother prediction. However, these methods only retain low-frequency signals in the frequency domain space, resulting in a certain degree of signal loss, and noise information is not effectively processed. Summary of the invention

[0005] The purpose of the present invention is to provide a 3D human motion prediction method based on a correlation multi-scale graph clustering network, which significantly improves the prediction accuracy, enhances the motion feature extraction capability, compensates for the signal loss caused by discrete cosine transform domain modeling, aggregates comprehensive features to effectively learn joint dynamics, and constructs adaptive multi-scale graphs, which solves the problem of insufficient flexibility of fixed multi-scale graphs in different motions, and has low system cost.

[0006] To achieve the above object, the present invention provides a 3D human motion prediction method based on a correlation multi-scale graph clustering network, comprising the following steps:

[0007] S1, the historical motion sequence enters the adaptive multi-scale computing unit AMCU branch after discrete cosine transformation and embedding;

[0008] S2, the historical motion sequence enters the motion decomposition modeling module DMMB branch;

[0009] S3. Future motion sequence predicted by the correlation multi-scale graph clustering network CMGC output.

[0010] Preferably, for AMCU and DMMB, human joints are regarded as a set of graph nodes to construct a human body structure graph, and the complex relationship between joints is modeled through the graph convolution layer GCL. In GCL, the potential joint correlation is captured by a trainable adjacency matrix while maintaining structural invariance, and an initial residual connection and identity mapping are added to GCL.

[0011] Preferably, the GCL is expressed as:

[0012] ;

[0013] in, It is The hidden state of the layer, yes The trainable parameters of the layer, is a batch normalization layer, is a trainable adjacency matrix, It is The hidden state of the layer, is the hidden state of the first layer of the network It is The scaling factor of the layer, is the activation function, is a hyperparameter, ensuring that The final representation of the layer contains information about the initial features. is the identity matrix. As the GCL gets deeper, Weight The bigger.

[0014] Preferably, in S1, the last frame of the historical motion sequence is repeated S times and concatenated with T historical frames. The repeated T historical frames are the initial guess of the future motion. The motion sequence is converted from the time domain to the frequency domain using the discrete cosine transform DCT matrix. It is expressed as:

[0015] ;

[0016] ;

[0017] in, are the elements in the DCT matrix, is the indicator function, is the row index, is the column index.

[0018] Preferably, the AMCU branch in S1 includes the following steps:

[0019] S1.1, using the correlation measurement method to calculate the correlation between the joints in the human body diagram, and then adaptively generating multiple new scale diagrams based on the calculated correlation;

[0020] S1.2. Use GCL to model the human body graph of each scale in parallel, and then integrate the human body graphs of all scales back to the original scale.

[0021] Preferably, in S1.1, the human joint set is , construct two subsets by striding sampling and , and use cosine similarity to calculate the correlation between the joints in the two subsets:

[0022] ;

[0023] in, It's a joint and joints The cosine similarity between yes The one-norm value of yes The one-norm value of ;

[0024] Select the top k joint pairs with the highest similarity scores, each of which contains a of joints and one from joints, and Each joint in The corresponding joints in the matching method allow A node in The multiple nodes in the human body are downsampled by aggregating the matched joints, which can be expressed as:

[0025] ;

[0026] in, Represents the average function, the joint With joint set polymerization, are the top k highest relevance scores, It is the new joint point after aggregation;

[0027] Retain joints with lower correlation and allow one joint to match multiple joint pairs during the aggregation process. By concatenating the aggregated joint set with the retained joints, a human body representation of a new scale is obtained:

[0028] .

[0029] Preferably, in the multi-scale generation process in S1.2, the aggregation relationship of the joints at the finer scale is recorded, and in the restoration process, the aggregated joints are copied from the coarser scale and accurately positioned at the finer scale according to the recorded aggregation relationship. Since the coarser scale is the result obtained after multiple aggregation steps, multiple restoration steps are required to return to the original scale. The output of AMCU is obtained by adding the joints at the original scale to the joints restored from the coarse scale to the original scale; let Represents r different scales, and the recovery process is expressed as follows:

[0030] ;

[0031] in, Indicates The scale is restored to scale, is the scaling factor, is the scaling factor of the first scale, is the scaling factor after reconstructing from the second scale back to the first scale, and so on.

[0032] Preferably, the DMMB in S2 decomposes the motion sequence into approximate coefficients by DWT and detail factor , and model them separately, using biorthogonal wavelet bior1.3 which can maintain signal characteristics and is robust to noise as the wavelet operator. Given the The historical observation sequence of each joint , using a low pass filter and high pass filter Convolve and downsample the sequence:

[0033] ;

[0034] ;

[0035] in, , , is the approximate coefficient after downsampling, are the approximate coefficients after convolution without downsampling, is the detail coefficient after downsampling, is the detail coefficient after convolution without downsampling;

[0036] After modeling the approximate signal and detail coefficients respectively, the complete motion sequence is reconstructed using the inverse discrete wavelet transform IDWT. The values ​​of the low-pass filter and high-pass filter in IDWT are and .

[0037] Preferably, the joint loss function and the skeleton loss function are defined in S3, and the predicted action is assumed to be , the corresponding true value is , the joint loss function is expressed as:

[0038] ;

[0039] in The predicted t The first m Joint points, Represents the true t The first m Joint points, T Represents the number of historical frames, M Indicates the number of joint points, F Represents the calculation of the second norm.

[0040] The present invention proposes a new paradigm for defining multi-scale graphs of human motion. An adaptive multi-scale computational unit (AMCU) is developed, which uses the correlation of joints at fine scales to adaptively generate new joints at coarse scales. Based on this design, the present invention can construct a multi-scale graph that is not limited by physical constraints and the number of layers, and adaptively learn the semantic relationship between joints at different levels. The present invention introduces discrete wavelet transform (DWT) in motion prediction technology, decomposes the motion sequence into approximate signals and detail coefficients, and learns a comprehensive motion spectrum representation while compensating for these information losses.

[0041] Therefore, the present invention adopts the above-mentioned 3D human motion prediction method based on correlation multi-scale graph clustering network, which has the following beneficial effects:

[0042] (1) Significantly improve prediction accuracy: CMGC effectively captures the implicit relationship between motion joints through adaptive multi-scale graph representation learning.

[0043] (2) Enhanced motion feature extraction capabilities: The discrete wavelet transform (DWT) is introduced to compensate for the signal loss caused by discrete cosine transform (DCT) domain modeling, learn rich spectral representations, and aggregate comprehensive features to effectively learn joint dynamics.

[0044] (3) Adaptive multi-scale graph construction: Through the adaptive multi-scale computing unit (AMCU), a new scale graph is constructed according to the correlation of joints, and the semantic relationships at different levels are adaptively learned, solving the problem of insufficient flexibility of fixed multi-scale graphs in different movements.

[0045] (4) Low system cost: CMGC can run on a single RTX-2080 graphics card and can meet the predicted real-time requirements.

[0046] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of a CMGC embodiment of a 3D human motion prediction method based on a correlation multi-scale graph clustering network of the present invention;

[0048] Figure 2 It is a schematic diagram of a multi-scale generation process of an embodiment of a 3D human motion prediction method based on a correlation multi-scale graph clustering network of the present invention;

[0049] Figure 3 It is a schematic diagram of the process of fine-scale restoration of an embodiment of a 3D human motion prediction method based on a correlation multi-scale graph clustering network of the present invention. DETAILED DESCRIPTION

[0050] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0051] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0052] Embodiment 1

[0053] like Figure 1 As shown, the present invention provides a 3D human motion prediction method based on a correlation multi-scale graph clustering network, comprising the following steps:

[0054] The correlation multi-scale graph clustering network CMGC includes an adaptive multi-scale computing unit AMCU branch and a motion decomposition modeling module DMMB branch. CMGC can run on a single RTX-2080 graphics card and can meet the real-time requirements of prediction. For AMCU and DMMB, human joints are regarded as a set of graph nodes to construct a human structure diagram. The complex relationship between joints is modeled through the graph convolution layer GCL. In GCL, a trainable adjacency matrix is ​​used to capture potential joint correlations while maintaining structural invariance. This adjacency matrix establishes interactions between non-adjacent but strongly correlated joints. In order to solve the problem of over-smoothing caused by deep GCL, initial residual connections and identity mappings are added to GCL to establish a channel for forward and backward information transmission. GCL is expressed as:

[0055] ;

[0056] in, It is The hidden state of the layer, yes The trainable parameters of the layer, is a batch normalization layer, is a trainable adjacency matrix, It is The hidden state of the layer, is the hidden state of the first layer of the network, It is The scaling factor of the layer, is the activation function, is a hyperparameter, ensuring that The final representation of the layer contains information about the initial features. is the identity matrix. As the GCL gets deeper, Weight The larger the bit matrix, the more information can be passed directly between multiple layers of the network without being affected by the intermediate layers.

[0057] S1, the historical motion sequence enters the adaptive multi-scale computing unit AMCU branch after discrete cosine transformation and embedding;

[0058] The last frame of the historical motion sequence is repeated S times and concatenated with T historical frames. The repeated T historical frames are the initial guess of future motion. The motion sequence is converted from the time domain to the frequency domain using the discrete cosine transform (DCT) matrix. It is expressed as:

[0059] ;

[0060] ;

[0061] in, are the elements in the DCT matrix, is the indicator function, is the row index, is the column index.

[0062] Adaptively generate r scales and use GCL to embed the motion at each scale to obtain the initial state. In each AMCU, adaptively create r scales. Combined with the initial state of each scale, two layers of GCL are used to model the features of different scales. Finally, all scales are restored and added back to the original scale to obtain the output of the AMCU, which will be fed into the subsequent AMCU for further calculation. After all AMCU calculations are completed, the IDCT transformation matrix (i.e. The inverse matrix of ( ) restores the data from the frequency domain to the time domain to obtain a motion sequence represented by 3D coordinates.

[0063] The AMCU branch includes the following steps:

[0064] S1.1, using the correlation measurement method to calculate the correlation between the joints in the human body diagram, and then adaptively generating multiple new scale diagrams based on the calculated correlation;

[0065] like Figure 2 As shown, suppose the set of human joints is , construct two subsets by striding sampling and , and use cosine similarity to calculate the correlation between the joints in the two subsets:

[0066] ;

[0067] in, It's a joint and joints The cosine similarity between yes The one-norm value of yes The one-norm value of ;

[0068] Select the top k joint pairs with the highest similarity scores, each of which contains a of joints and one from joints, and Each joint in The corresponding joints in the matching method allow A node in The multiple nodes in the human body are downsampled by aggregating the matched joints, which can be expressed as:

[0069] ;

[0070] in, Represents the average function, the joint With joint set polymerization, are the top k highest relevance scores, It is the new joint point after aggregation;

[0071] Retain joints with low correlation and allow one joint to match multiple joint pairs during the aggregation process. By concatenating the aggregated joint set with the retained joints, a human body representation of a new scale is obtained:

[0072] .

[0073] S1.2, use GCL to model the human body graph of each scale in parallel, and then integrate the human body graphs of all scales back to the original scale;

[0074] like Figure 3 As shown in , independent modeling at different scales may lead to significant semantic differences between data. To solve this problem, the joints of the original scale are reconstructed at each scale, and then they are reintegrated back to the original scale at the end of AMCU. In the multi-scale generation process, the aggregation relationship of joints at finer scales is recorded. In the recovery process, the aggregated joints are copied from the coarser scale and accurately positioned to the finer scale according to their recorded aggregation relationship. Since the coarser scale is the result of multiple aggregation steps, multiple recovery steps are required to return to the original scale. The output of AMCU is obtained by adding the joints of the original scale to the joints restored from the coarse scale to the original scale; let Represents r different scales, and the recovery process is expressed as follows:

[0075] ;

[0076] in, Indicates The scale is restored to scale, is the scaling factor, is the scaling factor of the first scale, is the scaling factor after reconstructing from the second scale back to the first scale, and so on.

[0077] S2, the historical motion sequence enters the motion decomposition modeling module DMMB branch;

[0078] DMMB uses DWT to decompose the motion sequence into approximate coefficients and detail factor , and model them separately, using biorthogonal wavelet bior1.3 which can maintain signal characteristics and is robust to noise as the wavelet operator. Given the The historical observation sequence of each joint , using a low pass filter and high pass filter Convolve and downsample the sequence:

[0079] ;

[0080] ;

[0081] in, , , is the approximate coefficient after downsampling, are the approximate coefficients after convolution without downsampling, is the detail coefficient after downsampling, is the detail coefficient after convolution without downsampling;

[0082] After modeling the approximate signal and detail coefficients respectively, the complete motion sequence is reconstructed using the inverse discrete wavelet transform IDWT. The values ​​of the low-pass filter and high-pass filter in IDWT are and Modeling the approximate signal significantly reduces the complexity of the fitting process and eliminates noise interference. Modeling the detail coefficients extracts significant changes and motion noise in the motion sequence.

[0083] Use DWT to decompose the motion sequence and get the approximate signal and detail factor , and use graph convolution to embed both into a high-dimensional space. Then, DMMB uses N GCLs to model and Finally, the feature dimensions of both are reduced through graph convolution, and the complete sequence is reconstructed using IDWT.

[0084] S3, the future motion sequence predicted by the correlation multi-scale graph clustering network CMGC output;

[0085] The joint loss function and skeleton loss function are defined, and the predicted action is assumed to be , the corresponding true value is , the joint loss function is expressed as:

[0086] ;

[0087] in The predicted t The first m Joint points, Represents the true t The first m Joint points, T Represents the number of historical frames, M Indicates the number of joint points, F Represents the calculation of the second norm.

[0088] Experimental results on three large-scale datasets, Human3.6M, CMU Mocap and 3DPW, show that the 3D mean per-joint position error (MPJPE) of the motion prediction technology proposed in this invention is reduced by 11.2%, 10.1% and 11.2% respectively.

[0089] Therefore, the present invention adopts the above-mentioned 3D human motion prediction method based on correlation multi-scale graph clustering network, which significantly improves the prediction accuracy, enhances the motion feature extraction capability, compensates for the signal loss caused by discrete cosine transform domain modeling, aggregates comprehensive features to effectively learn joint dynamics, and constructs adaptive multi-scale graphs, which solves the problem of insufficient flexibility of fixed multi-scale graphs in different movements, and has low system cost.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A 3D human motion prediction method based on a correlation multi-scale graph clustering network, characterized by: The following steps are involved: S1. The correlation multi-scale graph clustering network CMGC includes an adaptive multi-scale computing unit AMCU branch and a motion decomposition modeling module DMMB branch. The historical motion sequence enters the adaptive multi-scale computing unit AMCU branch after discrete cosine transformation and embedding; S1.1, using the correlation measurement method to calculate the correlation between the joints in the human body diagram, and then adaptively generating multiple new scale diagrams based on the calculated correlation; S1.

2. Use GCL to model the human body graph of each scale in parallel, and then integrate the human body graphs of all scales back to the original scale S2, the historical motion sequence enters the motion decomposition modeling module DMMB branch; DMMB decomposes the motion sequence into approximate coefficients c through DWT A and detail coefficient c D , and model them separately, using biorthogonal wavelet bior1.3 which can maintain signal characteristics and is robust to noise as the wavelet operator. Given the historical observation sequence of the vth joint Use a low pass filter and high pass filter Convolve and downsample the sequence: c A (k)=c A (2k) c D (k)=c D (2k) in, c A (k) is the approximate coefficient after downsampling, c A (2k) is the approximate coefficient after convolution without downsampling, c D (k) is the detail coefficient after downsampling, c D (2k) is the detail coefficient after convolution without downsampling; After modeling the approximate signal and detail coefficients respectively, the complete motion sequence is reconstructed using the inverse discrete wavelet transform IDWT. The values ​​of the low-pass filter and high-pass filter in IDWT are and S3. The correlation multi-scale graph clustering network CMGC outputs the predicted future motion sequence based on the motion sequence obtained by AMCU and DMMB.

2. The 3D human motion prediction method based on correlation multi-scale graph clustering network according to claim 1, characterized in that: For AMCU and DMMB, human joints are regarded as a set of graph nodes to construct a human body structure graph. The complex relationship between joints is modeled through the graph convolutional layer GCL. In GCL, the potential joint correlation is captured by a trainable adjacency matrix while maintaining structural invariance. Initial residual connections and identity mapping are added to GCL.

3. The 3D human motion prediction method based on correlation multi-scale graph clustering network according to claim 2, characterized in that: The GCL is expressed as: Among them, H l+1 is the hidden state of the l+1th layer, W l is the trainable parameter of layer l, is a batch normalization layer, A is a trainable adjacency matrix, and H l is the hidden state of layer l, H 0 is the hidden state of the first layer of the network, β l is the scaling factor of the lth layer, σ is the activation function, α is a hyperparameter that ensures that the final representation of the lth layer contains the information of the initial features, I is the identity matrix, and the weight of I (1-β) is larger when the GCL is deeper. l ) is larger.

4. The 3D human motion prediction method based on correlation multi-scale graph clustering network according to claim 3, characterized in that: In S1, the last frame of the historical motion sequence is repeated S times and concatenated with T historical frames. The repeated T historical frames are the initial guess of future motion. The motion sequence is converted from the time domain to the frequency domain using the discrete cosine transform DCT matrix. The DCT matrix D is expressed as: Among them, D i,j is an element in the DCT matrix, δ i,j is the indicator function, i is the row index, and j is the column index.

5. The 3D human motion prediction method based on correlation multi-scale graph clustering network according to claim 4, characterized in that: S1.1S a is a subset of the human joint set, S b is another subset of the human joint set, and uses cosine similarity to calculate the correlation between the joints in the two subsets: Among them, g(J i ,J j ) is the joint J i and joints J j The cosine similarity between ||J i ||It is J i The one-norm value of ||J j ||It is J j The one-norm value of ; Select the top k joint pairs with the highest similarity scores, each of which contains a a The joint and one from S b joints, and S b Each joint in S a The corresponding joints in the matching relationship formed in this way allow S a A node in corresponds to S b The multiple nodes in the human body are downsampled by aggregating the matched joints, which is expressed as: in, Represents the average function, and the joint J i With joint set {J j } aggregation, Corr[:k] is the highest top k correlation scores, J r It is the new joint point after aggregation; Retain joints with lower correlation and allow one joint to match multiple joint pairs during the aggregation process. By concatenating the aggregated joint set with the retained joints, a human body representation of a new scale is obtained:

6. The 3D human motion prediction method based on correlation multi-scale graph clustering network according to claim 5, characterized in that: In the multi-scale generation process in S1.2, the aggregation relationship of the joints at the finer scale is recorded. In the restoration process, the aggregated joints are copied from the coarser scale and accurately positioned at the finer scale according to the recorded aggregation relationship. Since the coarser scale is the result obtained after multiple aggregation steps, multiple restoration steps are required to return to the original scale. The output of AMCU is obtained by adding the joints at the original scale and the joints restored from the coarse scale to the original scale. Let J1, J2, …, J r Represents r different scales, and the recovery process is expressed as follows: in, Indicates that the i-th scale is restored to the j-th scale, α e is the scaling factor, where e = 1, 2, 3, ..., α1 is the scaling factor of the first scale, α2 is the scaling factor after reconstruction from the second scale back to the first scale, and so on.

7. The 3D human motion prediction method based on correlation multi-scale graph clustering network according to claim 6, characterized in that: The joint loss function and the skeleton loss function are defined in S3. Let the predicted action be The corresponding true value is X, and the joint loss function is expressed as: in represents the mth joint point on the predicted tth frame, represents the mth joint point on the real tth frame, T represents the number of historical frames, M represents the number of joint points, and F represents the second norm calculation.

Images