Three-dimensional human skeleton posture estimation method and model training method

Abstract

The invention provides a three-dimensional human skeleton attitude estimation method and a model training method. Four types of modal data are input into a feature extraction module, four types of features are extracted, fused by an inter-modal feature fusion module and then input into a three-dimensional coordinate estimation module, and a three-dimensional human skeleton attitude prediction value is output; during each round of iterative training of the three-dimensional human skeleton attitude estimation model, a single-modal contribution value is calculated based on a Sharpley value method, and modal categories are divided; carrying out simulation training on the current batch of samples, respectively fitting different modal types of noise modal data, establishing two Gaussian distributions by a root-mean-square error of the simulation training, training a three-dimensional human skeleton posture estimation model by the current batch of samples after the noise data corresponding to the sample with a higher mean value is removed, and estimating the posture of the three-dimensional human skeleton. And on the basis of the root mean square error of the predicted value and the true value and the plastic weight consolidation loss, constructing a combined loss update three-dimensional human skeleton attitude estimation model parameter, and fusing the feature extraction module parameters corresponding to different modalities with the initial feature extraction module parameters according to a preset weight.

Description

Technical Field

[0001] The present invention relates to the technical field of image and video information processing, and in particular to a three-dimensional human skeleton posture estimation method and a model training method. Background Art

[0002] 3D human skeleton pose estimation is the process of recovering the 3D coordinates of human joints from various input sources, and has attracted extensive research attention in applications such as human-computer interaction and computer animation. Camera-based input images and videos are easy for users to operate and the captured data contains rich human information, but they have limitations under complex lighting conditions and need to rely on accurate camera parameters for complex 2D to 3D space conversion; multimodal human skeleton pose estimation methods show broad application prospects through multiple sensor modes. There is a problem of modal imbalance in multimodal learning. The optimization process of the feature extraction module of the main modal data will inhibit the optimization process of other modal data, resulting in the final 3D human skeleton pose estimation model performing worse than the theoretical optimal performance.

[0003] In the prior art, the single-modal human skeleton posture estimation method extracts two-dimensional human skeleton posture features from videos or images through a deep learning model and maps them to three-dimensional coordinates. When the image is blurred due to lighting or extreme climate conditions, the human skeleton posture estimation result will be greatly affected; the multimodal method only uses two modal data, and still has disadvantages when facing complex scenes such as weak lighting, no lighting or extreme climate conditions, and three or more modal data have not been fully explored and studied. In addition, the existing solution to the modal imbalance problem in multimodal learning is mainly to adaptively adjust the training process of the main modality by quantifying the suppression ratio between modalities to speed up the training and optimization of other modalities, or to add additional training modules to each modal branch to balance the learning process between modalities. This method has not been applied to regression tasks such as posture estimation, and ignores the inherent limitations of modal data. When some modalities have no task-related information or there is more noise, there is a gap in the task-related information content between different modal data and the problem of modal imbalance cannot be solved. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a three-dimensional human skeleton posture estimation method and a model training method to eliminate or improve one or more defects existing in the prior art, and solves the problem of poor three-dimensional human skeleton posture estimation effect in the prior art.

[0005] One aspect of the present invention provides a three-dimensional human skeleton posture estimation model training method, the method comprising the following steps:

[0006] Acquire a training sample set, wherein each sample in the training sample set contains four types of modal data generated for the sample object, including image data, laser radar data, millimeter wave radar data, and wireless channel data, and the sample contains a label of a true value of a three-dimensional human skeleton posture;

[0007] The three-dimensional human skeleton posture estimation model is trained by using the training sample set, and the three-dimensional human skeleton posture estimation model includes an image feature extraction module, a first point cloud feature extraction module, a second point cloud feature extraction module, a wireless channel feature extraction module, an inter-modal feature fusion module and a three-dimensional coordinate estimation module; the image feature extraction module is used to extract the image features of the image data, the first point cloud feature extraction module is used to extract the laser radar features of the laser radar data, the second point cloud feature extraction module is used to extract the millimeter wave radar features of the millimeter wave radar data, and the wireless channel feature extraction module is used to extract the wireless channel features of the wireless channel data; the inter-modal feature fusion module is used to fuse the image features, the laser radar features, the millimeter wave radar features and the wireless channel features, and then input them into the three-dimensional coordinate estimation module to output a three-dimensional human skeleton posture prediction value, based on the root mean square error of the three-dimensional human skeleton posture prediction value and the true value, and introduce a plastic weight consolidation loss to construct a joint loss, minimize the joint loss to update the parameters of the three-dimensional human skeleton posture estimation model, and obtain a target three-dimensional human skeleton posture estimation model;

[0008] Among them, in each round of iteration, the current three-dimensional human skeleton posture estimation model is used for fast fitting and the single modal contribution values ​​of the four types of modal data are calculated based on the Shapley value method, and the four types of modalities are divided into superior modal classes and inferior modal classes according to the single modal contribution values ​​through a preset clustering algorithm; the single modal data whose single modal contribution values ​​of the current batch of samples decrease by a set condition compared with the previous batch of samples are defined as noise modal data; the current batch of samples are used for simulation training, wherein the fitting of the corresponding feature extraction module is suspended for the noise modal data belonging to the superior modal class, and the noise modal data belonging to the inferior class is suspended. The fitting of the corresponding feature extraction module of the noise modal data of the isomodal class is enhanced based on resampling, and two Gaussian distributions with different means are established for the root mean square error generated by the simulation training based on the Gaussian mixture model, and the noise data corresponding to the higher mean is discarded; the three-dimensional human skeleton posture estimation model is trained using the denoised current batch samples; the feature extraction module parameters corresponding to the superior modal class are fused with the initial feature extraction module parameters according to the first weight, and the feature extraction module parameters corresponding to the inferior modal class are fused with the initial feature extraction module parameters according to the second weight.

[0009] In some embodiments, the image feature extraction module adopts four convolutional networks and three dilated convolutional networks; the first point cloud feature extraction module adopts a first five-layer self-attention network, each layer of the first five-layer self-attention network is connected to a downsampling layer and a first average pooling layer is connected to the final output of the first five-layer self-attention network; the second point cloud feature extraction module adopts a second five-layer self-attention network, and the output of the second five-layer self-attention network is connected to a second average pooling layer; the wireless channel feature extraction module adopts three residual neural networks, a self-attention network and a two-dimensional average pooling layer fusion network.

[0010] In some embodiments, the step of the inter-modality feature fusion module fusing the image feature, the laser radar feature, the millimeter wave radar feature and the wireless channel feature comprises:

[0011] The image features, the laser radar features, the millimeter wave radar features, and the wireless channel features are uniformly mapped to the same feature space through the fully connected layer of the inter-modal feature fusion module and unified to a preset dimension through dimensional transformation;

[0012] The image features, the lidar features, the millimeter-wave radar features and the wireless channel features that are in the same feature space and have the same dimension are spliced ​​to obtain spliced ​​features, and the spliced ​​features are input into the four self-attention layers of the inter-modal feature fusion module for feature fusion and average pooling is performed to output multi-modal fusion features.

[0013] In some embodiments, the expression for calculating the single-modal m-contribution value of the four types of modal data based on the Shapley value method is:

[0014]

[0015] V(S,m)=s(p 3D , MM(S∪{m}))-s(p 3D , MM(S));

[0016]

[0017] in, represents the set of modes m, and S is represented by subset of S, V(S, m) represents the additional benefit obtained by merging mode m into subset S, p 3D Represents the true value of the 3D human skeleton posture, represents the three-dimensional human skeleton posture prediction value, MM(·) represents the three-dimensional human skeleton posture estimation model, Indicates the Pearson correlation coefficient between the true value of the 3D human skeleton posture and the predicted value of the 3D human skeleton posture when the 3D human skeleton point is k. j represents the number of 3D human body posture skeleton points.

[0018] In some embodiments, the expression of the root mean square error generated by the simulation training is:

[0019]

[0020] Among them, j represents the number of 3D human body posture skeleton points, It represents the true value of the posture of the 3D human skeleton point k when the training batch sample is i. It represents the posture prediction value of the 3D human skeleton point k when the training batch sample is i.

[0021] In some embodiments, the root mean square error generated by the simulation training is used to establish two Gaussian distributions with different means based on the Gaussian mixture model and the noise data corresponding to the one with the higher mean is discarded. The noise sample composed of the noise data corresponding to the higher mean A normal sample consisting of normal data corresponding to the one with a lower mean The expression is:

[0022]

[0023]

[0024]

[0025] in, represents the complete sample, represents the sample points in the complete sample, represents the posterior probability distribution when the training batch sample is i, g represents the distribution component with the lower mean in the Gaussian mixture model, represents the root mean square error when the training batch sample is i, represents samples without noise data, Represents a sample containing noisy data.

[0026] In some embodiments, based on the root mean square error of the predicted value and the true value of the three-dimensional human skeleton posture, the expression of the joint loss constructed by introducing the plastic weight consolidation loss is:

[0027]

[0028]

[0029]

[0030]

[0031]

[0032] in, represents the root mean square error generated by the simulated training, λ represents the preset hyperparameter, represents the plastic weight consolidation loss, j represents the number of 3D human body posture skeleton points, i represents the i-th sample in the training batch, It represents the true value of the posture of the 3D human skeleton point k when the training batch sample is i. It indicates the posture prediction value when the training batch sample is i and the 3D human skeleton point is k. Indicated by Sample of time The calculated adaptive Fisher information matrix, θ t，m express The mth feature extraction module parameters at time, express The feature extraction module parameters at the moment, represents the information matrix calculated when the training batch sample is i, Represents the feature extraction module parameter θ t The gradient of , β represents the preset hyperparameter.

[0033] In some embodiments, the feature extraction module parameters corresponding to the superior modality class are fused with the initial feature extraction module parameters according to the first weight, and the feature extraction module parameters corresponding to the inferior modality class are fused with the initial feature extraction module parameters according to the second weight. The expression is:

[0034]

[0035] in, represents the first weight, α I represents the second weight, represents the initial feature extraction module parameters of mode m, represents the feature extraction module parameters of mode m when the preset iteration round is l, represents the superior modal class, Represents the secondary modal class.

[0036] On the other hand, the present invention provides a method for estimating a 3D human skeleton posture, the method comprising the following steps:

[0037] Obtain four types of modal data to be predicted;

[0038] The four types of modal data to be predicted are input into the target three-dimensional human skeleton posture estimation model obtained by any of the three-dimensional human skeleton posture estimation model training methods described above to obtain the three-dimensional human skeleton posture.

[0039] On the other hand, the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of any one of the above-mentioned methods when executed by a processor.

[0040] The beneficial effects of the present invention are at least:

[0041] In the three-dimensional human skeleton posture estimation method and model training method described in the present invention, four types of modal data containing labels of true values ​​of three-dimensional human skeleton postures are input into four types of feature extraction modules in the three-dimensional human skeleton posture estimation model to respectively extract features of the four types of data, deeply explore human skeleton posture information and improve the expressiveness and robustness of the three-dimensional human skeleton posture estimation model; the four types of features are input into the inter-modal feature fusion module to obtain multi-modal fusion features and input into the three-dimensional coordinate estimation module to obtain the three-dimensional human skeleton posture prediction value; in the training process of the three-dimensional human skeleton posture estimation model, the single modal contribution values ​​of the four types of modal data are calculated based on the Shapley value method to judge the training state of the three-dimensional human skeleton posture estimation model, and the corresponding feature extraction modules are fitted according to the noise modal data and the modal category, and the mean square error of the fitting process is calculated according to the mean square error of the fitting process. The root error establishes two Gaussian distributions with different means based on a Gaussian mixture model and discards the noise data corresponding to the one with the higher mean, thereby improving the training effect by removing the noise data that affects the training in the sample; a joint loss is constructed based on the root mean square error of the three-dimensional human skeleton posture prediction value and the true value and the plastic weight consolidation loss, which is beneficial for the three-dimensional human skeleton posture estimation model to overcome forgetting and avoid the influence of noise on the model; when multiple samples in each batch are trained for multiple rounds of iterations and reach a preset iteration round, a first weight and a second weight are set to help the feature extraction module reshape the learning state and control the intensity of re-learning, thereby improving multimodal learning, and fusing the feature extraction module parameters with the initial feature extraction module parameters to retain the feature extraction module while enhancing the generalization ability of the three-dimensional human skeleton estimation model and improving the three-dimensional human skeleton posture estimation effect.

[0042] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0043] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0045] Figure 1 The figure is a flow chart of a method for training a 3D human skeleton posture estimation model according to an embodiment of the present invention.

[0046] Figure 2 Schematic diagram of the structure of a three-dimensional human skeleton posture estimation model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0048] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0049] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0050] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0051] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0052] In the prior art, the single-modal human skeleton posture estimation method extracts two-dimensional human skeleton posture features from videos or images through a deep learning model and maps them to three-dimensional coordinates. When the image is blurred due to lighting or extreme climate conditions, the human skeleton posture estimation result will be greatly affected; the multimodal method only uses two modal data, and still has disadvantages when facing complex scenes such as weak lighting, no lighting or extreme climate conditions; the present invention proposes a three-dimensional human skeleton posture estimation method and a model training method, inputs four types of modal data containing labels of true values ​​of three-dimensional human skeleton posture into a feature extraction module in a three-dimensional human skeleton posture estimation model, inputs the extracted four types of features into an inter-modal feature fusion module for fusion, and then inputs into the three-dimensional coordinate estimation module to output a three-dimensional human skeleton posture prediction value; in each round of iterative training of the three-dimensional human skeleton posture estimation model, the current three-dimensional human skeleton posture estimation model is used for fast fitting, the single modal contribution values ​​of the four types of modal data are calculated based on the Shapley value method, and the four types of modal data are clustered by a preset clustering algorithm. The modalities are divided into superior modal classes and inferior modal classes; the unimodal data whose unimodal contribution value of the current batch of samples decreases is the noise modal data and is subjected to simulation training, and the noise modal data belonging to the superior modal class and the noise modal data belonging to the inferior modal class are fitted respectively, and the root mean square error generated by the simulation training is used to establish two Gaussian distributions with different means based on a Gaussian mixture model and discard the noise data corresponding to the one with a higher mean; the denoised current batch of samples is used to train the three-dimensional human skeleton posture estimation model, and the root mean square error between the predicted value and the true value of the three-dimensional human skeleton posture is based on the introduction of a plastic weight consolidation loss to construct a joint loss, and the three-dimensional human skeleton posture estimation model parameters are updated by minimizing the joint loss, and when the preset iteration round is reached during the training of each batch of samples, the feature extraction module parameters corresponding to the superior modal class are fused with the initial feature extraction module parameters according to the first weight, and the feature extraction module parameters corresponding to the inferior modal class are fused with the initial feature extraction module parameters according to the second weight.

[0053] Figure 1 The present invention provides a method for training a 3D human skeleton posture estimation model, which includes the following steps S101 to S102:

[0054] Step S101: Obtain a training sample set, where each sample in the training sample set contains four types of modal data generated for the sample object, including image data, lidar data, millimeter wave radar data, and wireless channel data, and the sample contains a label of the true value of the three-dimensional human skeleton posture.

[0055] Step S102: A training sample set is used to train a three-dimensional human skeleton posture estimation model, wherein the three-dimensional human skeleton posture estimation model includes an image feature extraction module, a first point cloud feature extraction module, a second point cloud feature extraction module, a wireless channel feature extraction module, an inter-modal feature fusion module and a three-dimensional coordinate estimation module; the image feature extraction module is used to extract image features of image data, the first point cloud feature extraction module is used to extract laser radar features of laser radar data, the second point cloud feature extraction module is used to extract millimeter wave radar features of millimeter wave radar data, and the wireless channel feature extraction module is used to extract wireless channel features of wireless channel data; the inter-modal feature fusion module is used to fuse image features, laser radar features, millimeter wave radar features and wireless channel features, and then input them into the three-dimensional coordinate estimation module to output a three-dimensional human skeleton posture prediction value, based on the root mean square error between the three-dimensional human skeleton posture prediction value and the true value, and introducing a plastic weight consolidation loss to construct a joint loss, minimizing the joint loss to update the parameters of the three-dimensional human skeleton posture estimation model to obtain a target three-dimensional human skeleton posture estimation model.

[0056] Among them, in each round of iteration, the current three-dimensional human skeleton posture estimation model is used for fast fitting and the single modal contribution values ​​of the four types of modal data are calculated based on the Shapley value method. The four types of modes are divided into superior modal classes and inferior modal classes according to the single modal contribution values ​​through a preset clustering algorithm; the single modal data whose single modal contribution values ​​of the current batch of samples decrease to the set conditions compared with the previous batch of samples are defined as noise modal data; the current batch of samples are used for simulation training, wherein the fitting of the corresponding feature extraction module is suspended for the noise modal data belonging to the superior modal class, and the fitting of the corresponding feature extraction module is strengthened based on resampling for the noise modal data belonging to the inferior modal class, and two Gaussian distributions with different means are established for the root mean square error generated by the simulation training based on the Gaussian mixture model, and the noise data corresponding to the higher mean is discarded; the denoised current batch of samples is used to train the three-dimensional human skeleton posture estimation model; the feature extraction module parameters corresponding to the superior modal class are fused with its initial feature extraction module parameters according to the first weight, and the feature extraction module parameters corresponding to the inferior modal class are fused with its initial feature extraction module parameters according to the second weight.

[0057] In step S101, the four types of modal data in the training sample set are preprocessed, and the high-resolution network (HRNet) is used to process the image data and extract the two-dimensional human skeleton posture data; the farthest point sampling method is used for the lidar data to obtain the point cloud of a preset number of input points. The farthest point sampling method is an algorithm for achieving uniform sampling. It selects the farthest point from the existing sampling point set through continuous iteration, and finally obtains a set of sampling point sets with relatively uniform distribution of point clouds; the farthest point sampling method first randomly selects a point from the input point cloud as the initial point and adds it to the sampling point set. In each iteration, the distance from each point in the input point cloud to all points in the sampling point set is calculated, and the point farthest from the sampling point set is selected as the next sampling point and added to the sampling point set. Until the points in the sampling point set reach a preset 1024 points; the millimeter-wave radar data is aggregated into a point cloud of a preset number of input points using an adjacent frame method, and the preset number of the present invention is 128; the adjacent frame method samples the data of two or more frames adjacent to a certain frame of data while sampling the data, and performs feature extraction and data fusion on the sampled data; the extracted features include but are not limited to color features, texture features and edge features, and the data fusion method includes splicing the features of each frame and weighted summing the features of each frame according to preset weights; wavelet denoising is used to remove noise from the wireless channel data, and the wavelet denoising first uses a wavelet transform to decompose the signal into components of different scales and frequencies, removes the noise components in the signal by a threshold processing method, and reconstructs the denoised signal by an inverse wavelet transform.

[0058] In step S102, the four types of modal data use corresponding feature extraction modules to extract image features, lidar features, millimeter wave radar features and wireless channel features.

[0059] In some embodiments, the image feature extraction module uses four convolutional networks and three dilated convolutional networks; the first point cloud feature extraction module uses a first five-layer self-attention network, each layer of the first five-layer self-attention network is connected to a downsampling layer and a first average pooling layer is connected to the final output of the first five-layer self-attention network; the second point cloud feature extraction module uses a second five-layer self-attention network, and the output of the second five-layer self-attention network is connected to a second average pooling layer; the wireless channel feature extraction module uses three residual neural networks, a self-attention network and a two-dimensional average pooling layer fusion network.

[0060] Specifically, the image feature extraction module adopts a four-layer convolutional network including four convolutional networks and three dilated convolutional networks. The image data is input into the four-layer convolutional network to obtain the image features. The dilated convolutional network expands the coverage of the convolution kernel without increasing the amount of calculation. After being combined with the convolutional network, it extracts more global features while maintaining high resolution; in the first point cloud feature extraction module, the first five-layer attention network includes a preset number of attention heads and an attention output layer. The attention head extracts features from the lidar data through an attention calculation algorithm, a softmax function for normalizing the attention weights, and matrix multiplication, reduces the dimension through downsampling, and finally obtains the lidar features through the first average pooling layer. In the second point cloud feature extraction module, the second The five-layer self-attention network includes a preset number of attention heads and an attention output layer. The attention heads extract features from the millimeter-wave radar data through an attention calculation algorithm, a softmax function for normalizing the attention weights, and matrix multiplication. The extracted features are fused through the second average pooling layer. In the wireless channel feature extraction module, three residual neural networks are used to extract features from the three antenna data of the wireless channel data. The antenna is used to send and receive signals in wireless communication devices. The three antenna data are data related to the signals sent and received by the three antennas. The three features are concatenated and flattened to obtain flat features. After the flat features are interacted and fused through the self-attention network and the original feature form is restored, they are input into the two-dimensional average pooling layer to obtain the wireless channel features.

[0061] In some embodiments, the step of fusing the image feature, the laser radar feature, the millimeter wave radar feature and the wireless channel feature by the inter-modality feature fusion module includes steps S1021 to S1022:

[0062] Step S1021: The image features, lidar features, millimeter-wave radar features, and wireless channel features are uniformly mapped to the same feature space through the fully connected layer of the inter-modal feature fusion module and unified to a preset dimension through dimensional transformation.

[0063] Step S1022: The image features, lidar features, millimeter-wave radar features, and wireless channel features in the same feature space and with the same dimension are spliced ​​to obtain spliced ​​features, and the spliced ​​features are input into the four-layer self-attention layer of the inter-modal feature fusion module, and the spliced ​​features are fused and average pooled to output multi-modal fusion features.

[0064] Specifically, four types of modal features are mapped to the same feature space through four fully connected layers of the inter-modal feature fusion module, and each fully connected layer includes two linear layers and one normalization layer. In order to further enhance the feature expression capability and adapt to the task, the dimensions of the image features, lidar features, millimeter-wave radar features and wireless channel features extracted by the four types of feature extraction modules are adjusted to the same preset dimension size through dimensional transformation to facilitate subsequent feature fusion. The preset dimension size of the present invention is 256; the dimensional transformation method includes but is not limited to principal component analysis, linear discriminant analysis and polynomial feature expansion; four layers of self-attention layers and average pooling layers perform deep feature fusion to obtain more comprehensive human skeleton posture features; the multimodal fusion features are input into the three-dimensional coordinate estimation module, and the three-dimensional coordinate estimation module uses multiple layers of linear layers and batch normalization layers to perform multiple feature transformations and normalization operations to obtain three-dimensional coordinate values. There are 51 three-dimensional coordinate values ​​in the present invention, and the three-dimensional coordinate values ​​are matrix transformed to obtain three-dimensional human skeleton posture prediction values.

[0065] Furthermore, the contribution value of each modal data calculated by the Shapley value method is used to judge the relative importance of different modal features to the final output result and the training status of the three-dimensional human skeleton posture estimation model; the preset clustering algorithm of the present invention adopts k-means clustering (k-means clustering), firstly selects two initial clustering centers for representing the superior modal class and the inferior modal class, wherein the modal included in the clustering center with a larger contribution value is the superior modal class, and the modal included in the clustering center with a smaller contribution value is the inferior modal class, calculates the distance of each modality to the two initial clustering centers and assigns the modality to the category with a closer distance, and after all modalities are assigned, the modal redistribution is performed by calculating the mean of all modalities in the two categories as the clustering center of the two categories, until the clustering result is stable and the modality no longer shifts categories to complete the division of the superior modal class and the inferior modal class.

[0066] In some embodiments, the expression for calculating the single-modal m-contribution value of four types of modal data based on the Shapley value method is:

[0067]

[0068] V(S,m)=s(p 3D ,MM(S∪{m}))-s(p 3D ,MM(S));

[0069]

[0070] in, represents the set of modes m, and S is represented by subset of S, V(S,m) represents the additional benefit obtained by merging mode m into subset S, p 3D Represents the true value of the 3D human skeleton posture, represents the three-dimensional human skeleton posture prediction value, MM(·) represents the three-dimensional human skeleton posture estimation model, Indicates the Pearson correlation coefficient between the true value of the 3D human skeleton posture and the predicted value of the 3D human skeleton posture when the 3D human skeleton point is k. j represents the number of 3D human body posture skeleton points.

[0071] Furthermore, noise data affects the training process of the three-dimensional human skeleton posture estimation model, and is removed by the noise recognition and separation module. The noise recognition and classification module compares the unimodal contribution value of the current batch with that of the previous batch. When the unimodal contribution value in the current batch samples decreases to the set condition, the unimodal data is judged to be noise modal data. During simulation training, when the noise modal data is a superior modal class, the corresponding feature extraction module stops fitting the superior modal class. When the noise modal data is a sub-optimal modal class, resampling is performed and fitting is continued through the corresponding feature extraction module, so that the distribution difference between the noise data and the normal data becomes larger. The root mean square error generated during the simulation training is modeled by the Gaussian mixture model. Two Gaussian distributions are obtained by modeling and the modal data with a higher mean is used as noise data.

[0072] In some embodiments, the expression of the root mean square error generated by the simulation training is:

[0073]

[0074] Among them, j represents the number of 3D human body posture skeleton points, It represents the true value of the posture of the 3D human skeleton point k when the training batch sample is i. It represents the posture prediction value of the 3D human skeleton point k when the training batch sample is i.

[0075] In some embodiments, the root mean square error generated by the simulation training is used to establish two Gaussian distributions with different means based on the Gaussian mixture model and the noise data corresponding to the one with the higher mean is discarded. The noise sample composed of the noise data corresponding to the higher mean A normal sample consisting of normal data corresponding to the one with a lower mean The expression is:

[0076]

[0077]

[0078]

[0079] in, represents the complete sample, represents the sample points in the complete sample, represents the posterior probability distribution when the training batch sample is i, g represents the distribution component with the lower mean in the Gaussian mixture model, represents the root mean square error when the training batch sample is i, represents samples without noise data, Represents a sample containing noisy data.

[0080] In some embodiments, the root mean square error between the predicted value and the true value of the 3D human skeleton posture and the introduction of plastic weight consolidation loss to construct the expression of the joint loss is:

[0081]

[0082]

[0083]

[0084]

[0085]

[0086] in, represents the root mean square error generated by the simulated training, λ represents the preset hyperparameter, represents the plastic weight consolidation loss, j represents the number of 3D human body posture skeleton points, i represents the i-th sample in the training batch, It represents the true value of the posture of the 3D human skeleton point k when the training batch sample is i. It indicates the posture prediction value when the training batch sample is i and the 3D human skeleton point is k. Indicated by Sample of time The calculated adaptive Fisher information matrix, θ t,m express The mth feature extraction module parameters at time, express The feature extraction module parameters at the moment, represents the information matrix calculated when the training batch sample is i, Represents the feature extraction module parameter θ t The gradient of , β represents the preset hyperparameter.

[0087] Specifically, the posterior probability distribution takes into account the root mean square error information, and based on the probability of the distribution component with a lower mean obtained in the Gaussian mixture model according to the error information, the posterior probability can better meet the data requirements of model training. According to the comparison between the posterior probability distribution and 0.5, the modal data less than 0.5 is divided into samples containing noise data to obtain noise samples, and vice versa, it is divided into samples not containing noise data to obtain normal samples. Normal samples can improve the accuracy of the training of the 3D human skeleton posture estimation model and avoid the impact of noise data on model performance; the expectation maximization algorithm is used to iteratively update the initialization mean and variance of the Gaussian mixture model and establish two Gaussian distributions with different means and variances based on the two means and variances obtained.

[0088] Furthermore, an adaptive plastic weight consolidation loss is used to improve the robustness of knowledge memory in the 3D human skeleton pose estimation model and avoid the influence of noisy data on knowledge memory. The strength of the plastic weight consolidation loss is adjusted by presetting the hyperparameter λ and then added to the root mean square error to obtain the joint loss function.

[0089] In some embodiments, the feature extraction module parameters corresponding to the superior modality class are fused with the initial feature extraction module parameters according to the first weight, and the feature extraction module parameters corresponding to the inferior modality class are fused with the initial feature extraction module parameters according to the second weight. The expression is:

[0090]

[0091] in, represents the first weight, α I represents the second weight, represents the initial feature extraction module parameters of mode m, represents the feature extraction module parameters of mode m when the preset iteration round is l, represents the superior modal class, Represents the secondary modal class.

[0092] Specifically, during multiple iterative training of multiple samples in each batch, when the iterative training reaches a preset round, the feature extraction module parameters corresponding to the superior modal class and the feature extraction parameters corresponding to the inferior modal class are respectively fused with the corresponding initial feature extraction module parameters; the adaptive re-learning method sets different feature extraction module weights according to the superior modal class and the inferior modal class to control the intensity of re-learning and thus reshape the learning state of the feature extraction module, and performs weighted fusion of the initial feature extraction module parameters and the trained feature extraction parameters to achieve better performance.

[0093] On the other hand, the present invention provides a method for estimating a 3D human skeleton posture, the method comprising the following steps S201-S202:

[0094] Step S201: Obtain four types of modal data to be predicted.

[0095] Step S202: input the four types of modal data to be predicted into the target three-dimensional human skeleton posture estimation model obtained by any one of the three-dimensional human skeleton posture estimation model training methods mentioned above to obtain the three-dimensional human skeleton posture.

[0096] On the other hand, the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of any one of the above methods when the computer program / instruction is executed by a processor.

[0097] The present invention is described below in conjunction with a specific embodiment:

[0098] Figure 2 The present invention is a schematic diagram of the structure of a three-dimensional human skeleton posture estimation model according to an embodiment of the present invention. The present invention uses a multimodal three-dimensional human skeleton posture estimation model to perform human skeleton posture estimation; uses a balanced multimodal learning method to optimize the training process of the model to obtain better performance; and uses a denoising continuous learning method to help a multimodal network achieve multi-task learning.

[0099] 1. Obtain training sample sets and labels.

[0100] A high-resolution network is used to extract and save the two-dimensional human skeleton posture data from the image data in the training sample set. The farthest point sampling method is used to obtain the point cloud of 1024 input points for the lidar data in the training sample set. The adjacent frame method is used to aggregate the millimeter-wave radar data in the training sample set to a point cloud of 128 input points. Wavelet denoising is used to obtain denoised data for the wireless channel data in the training sample set. The data of these four modes are used as the input of the subsequent network to obtain the true value p of the three-dimensional human skeleton posture. 3D ∈R 17×3 Used as labels to calculate the loss.

[0101] 2. Construct an image feature extraction module, a first point cloud feature extraction module, a second point cloud feature extraction module, a wireless channel feature extraction module, an inter-modal feature fusion module and a three-dimensional coordinate estimation module.

[0102] 2.1 Construct the image feature extraction module, the first point cloud feature extraction module, the second point cloud feature extraction module and the wireless channel feature extraction module. Different modal data use different feature extraction modules. The image feature extraction module is a four-layer convolutional network, which contains four convolutional networks and three dilated convolutional networks; the first point cloud feature extraction module is a five-layer self-attention network, which includes n attention heads. The attention head consists of an attention calculation algorithm and a softmax for normalizing the attention weights and matrix multiplication. Each layer of the self-attention network is followed by a downsampling layer and an average pooling layer is connected at the end of the five-layer self-attention network to fuse data features; the second point cloud feature extraction module is also a five-layer self-attention network, but there is no downsampling layer, and only an average pooling layer is connected at the end of the five-layer self-attention network; the wireless channel feature extraction module includes three residual neural networks. For the features extracted by these three residual networks, a self-attention network is used for feature interaction and fusion, and a two-dimensional average pooling layer is used to fuse features at the end.

[0103] 2.2 Construct an inter-modal feature fusion module. The network includes four fully connected layers, a four-layer self-attention layer and an average pooling layer. Each fully connected layer contains two linear layers and a layer normalization operation. The fully connected layer performs feature mapping on the features extracted by each modality feature extraction network to achieve better fusion effect. The four-layer self-attention layer and the average pooling layer perform deep feature fusion.

[0104] 2.3 Construct a 3D coordinate estimation module, which includes multiple layers of linear layers and batch normalization layers.

[0105] 3. Input each modal data into each modal feature extraction network and output each modal feature.

[0106] The image data is input into the image feature extraction module, and the image features (1×1024) are extracted through the multi-layer convolutional network. The lidar data is input into the first point cloud feature extraction module. The initial input data is the three-dimensional coordinates of 1024 point cloud points (1024×3). After the first self-attention layer, the point cloud data (1024×32) is obtained, and then four times downsampling is performed to obtain (256×64). After four layers of self-attention layers, the point cloud features (4×512) are obtained, and the lidar features (1×512) are obtained by average pooling. For millimeter wave radar data, the same as the lidar data, after four layers, The millimeter-wave features (128×512) are averaged and pooled to obtain the final millimeter-wave radar features (1×512). For the wireless channel data, three residual neural networks are first used to extract features (17×4×512) from the three antenna data of the wireless channel data respectively. Then, these three features are concatenated to obtain the combined features (17×12×512) and flattened to obtain the flat features (20×512). After the features are fused through the self-attention network, they are restored to the original feature form (17×12×512). Finally, two-dimensional average pooling is used to obtain the wireless channel features (1×512).

[0107] 4. Input the image features, lidar features, millimeter-wave radar features and wireless channel features into the inter-modal feature fusion module and output multi-modal fusion features.

[0108] The image features, lidar features, millimeter-wave radar features and wireless channel features are input into the fully connected layer of the inter-modal feature fusion module. The features of each modality are uniformly mapped into the same feature space by using the fully connected layer, and the dimension of each feature is unified to 256. The four features are concatenated to obtain the concatenated features (4×256) and input into the inter-modal feature fusion module. In the inter-modal feature fusion module, four self-attention layers are used to perform deep feature fusion on the concatenated features, and finally average pooling is used to obtain the multi-modal fusion features (1×256).

[0109] 5. Input the multimodal fusion features into the three-dimensional coordinate estimation module, estimate the three-dimensional coordinates of the human skeleton points and output the three-dimensional human skeleton posture.

[0110] The multimodal fusion features are input into the 3D coordinate estimation module, and 51 coordinate values ​​are obtained by multiple feature transformations and normalization operations. The 3D human skeleton posture can be obtained by performing matrix transformation on these 51 coordinate values.

[0111] 6. Use the balanced multimodal learning method to train the model and obtain a trained model.

[0112] The balanced multimodal learning method consists of three parts: a single-modal contribution value calculation algorithm based on the Shapley value method, an adaptive relearning method, and a root mean square error.

[0113] The contribution module based on the Shapley value method uses the single-modal contribution calculation algorithm based on the Shapley value method to obtain the contribution value of each mode. The calculated contribution value can help the model determine the training status of the overall multimodal model. Among them, the expression of the single-modal m contribution value is:

[0114]

[0115] V(S,m)=s(p 3D ,MM(S∪{m}))-s(p 3D ,MM(S));

[0116]

[0117] in, represents the set of modes m, and S is represented by subset of S, V(S,m) represents the additional benefit obtained by merging mode m into subset S, p 3D Represents the true value of the 3D human skeleton posture, represents the three-dimensional human skeleton posture prediction value, MM(·) represents the three-dimensional human skeleton posture estimation model, Indicates the Pearson correlation coefficient between the true value of the 3D human skeleton posture and the predicted value of the 3D human skeleton posture when the 3D human skeleton point is k. j represents the number of 3D human body posture skeleton points.

[0118] The adaptive relearning module performs an adaptive relearning method based on the calculated contribution scores of each modality to help the feature extraction module relearn to achieve better performance. Specifically, it consists of a score clustering module and a relearning module. The score clustering module takes the contribution values ​​of each modality as input and combines the modalities through k-means clustering. Divided into two categories: superior modal class and submodal classes Since these two modality categories will produce inconsistent training states during the training process, adaptive relearning can help the feature extraction module reshape the learning state, which can be expressed by the following formula:

[0119]

[0120] in, represents the feature extraction module parameters of mode m when the preset iteration round is l, and α is an artificially set hyperparameter. and Different α values ​​will be set to control the intensity of relearning to achieve better performance. By weightedly fusing the parameters of the feature extraction modules of each modality with the parameters of the initial feature extraction module at the training round r, the parameters of the relearned feature extraction module can be obtained, which can retain some feature extraction capabilities and greatly enhance the generalization ability of the model, thereby achieving better estimation performance.

[0121] In balanced multimodal learning, the root mean square error of three-dimensional key points is used as the loss function, which is calculated as follows:

[0122]

[0123] Among them, j represents the number of 3D human body posture skeleton points, It represents the true value of the posture of the 3D human skeleton point k when the training batch sample is i, It represents the posture prediction value of the 3D human skeleton point k when the training batch sample is i.

[0124] 7. Based on the balanced multimodal learning method, the denoising continuous learning method is used to train the model to obtain a trained model. The denoising continuous learning method consists of two parts: a noise identification and separation module and an adaptive plastic weight consolidation module.

[0125] After the new training batch samples arrive, the data of the new training batch samples are first quickly fitted on the existing model, and then the contribution values ​​of the four modal data in the new training batch samples are calculated. The noise identification and separation module will monitor the contribution values, compare the newly added single modal contribution values ​​with the existing single modal contribution values, and obtain the mode with a significantly reduced contribution value, thereby determining it as a noise mode.

[0126] The training data is input into the model and quickly fitted for a small number of rounds, and the root mean square error of each sample is calculated. The training sample is modeled based on the root mean square error using a Gaussian mixture model to obtain two Gaussian distributions with different means and variances. Among them, when the noise mode is in the superior mode class, the fitting of the feature extraction module of the superior class is suspended, so that the distribution difference between the noise data and the normal data becomes larger; when the noise mode is in the inferior mode class, the fitting of the inferior mode class is strengthened by resampling, and the distribution difference between the inferior noise data and the normal data can also be increased. Finally, for the two Gaussian distributions obtained by modeling, the data with high mean distribution will be discarded when training the 3D human skeleton posture estimation model, and noise separation will be performed, because this part of the data is identified as noise data, which is not beneficial to the training process. The noise separation calculation based on the high and mixture model is as follows:

[0127]

[0128]

[0129]

[0130] in, represents the complete sample, represents the sample points in the complete sample, represents the posterior probability distribution when the training batch sample is i, g represents the distribution component with the lower mean in the Gaussian mixture model, represents the root mean square error when the training batch sample is i, represents samples without noise data, Represents a sample containing noisy data.

[0131] In addition, in order to help the model remember the knowledge of old tasks and avoid the influence of noise on model memory, adaptive plastic weight consolidation is used As the loss function, its calculation formula is as follows:

[0132]

[0133]

[0134]

[0135] Finally, based on the root mean square error of the predicted value and the true value of the three-dimensional human skeleton posture, the plastic weight consolidation loss is introduced to construct the expression of the joint loss:

[0136]

[0137] in, represents the root mean square error generated by the simulated training, λ represents the preset hyperparameter, represents the plastic weight consolidation loss, j represents the number of 3D human body posture skeleton points, i represents the i-th sample in the training batch, It represents the true value of the posture of the 3D human skeleton point k when the training batch sample is i, It indicates the posture prediction value when the training batch sample is i and the 3D human skeleton point is k. Indicated by Sample of time The calculated adaptive Fisher information matrix, θ t,m express The mth feature extraction module parameters at time, express The feature extraction module parameters at the moment, represents the information matrix calculated when the training batch sample is i, Represents the feature extraction module parameter θ t The gradient of , β represents the preset hyperparameter.

[0138] 8. Input the test set into the trained target 3D human skeleton pose estimation model to obtain the 3D human skeleton pose.

[0139] In summary, the present invention proposes a three-dimensional human skeleton posture estimation method and a model training method, wherein four types of modal data containing labels of true values ​​of three-dimensional human skeleton postures are input into a feature extraction module in a three-dimensional human skeleton posture estimation model, and the extracted four types of features are input into an inter-modal feature fusion module for fusion and then input into the three-dimensional coordinate estimation module to output a three-dimensional human skeleton posture prediction value; during the training process of the three-dimensional human skeleton posture estimation model, in each round of iteration, the current three-dimensional human skeleton posture estimation model is used for fast fitting and the single modal contribution values ​​of the four types of modal data are calculated based on the Shapley value method, and the four types of modes are divided into superior modal classes and inferior modal classes through a preset clustering algorithm; the current batch of samples is compared with the previous batch of samples. The unimodal data whose unimodal contribution value drops to the set conditions is defined as noise modal data; the current batch of samples is used for simulation training, and the noise modal data belonging to the superior modal class and the noise modal data belonging to the inferior modal class are fitted respectively, and two Gaussian distributions with different means are established for the root mean square error generated by the simulation training based on the Gaussian mixture model, and the noise data corresponding to the higher mean is discarded; the denoised current batch of samples is used to train the three-dimensional human skeleton posture estimation model, based on the root mean square error of the predicted value and the true value of the three-dimensional human skeleton posture, and the plastic weight consolidation loss is introduced to construct a joint loss, and the parameters of the three-dimensional human skeleton posture estimation model are updated by minimizing the joint loss.

[0140] Furthermore, through an adaptive re-learning method, the feature extraction module parameters corresponding to the superior modal class at a preset iteration round are fused with its initial feature extraction module parameters according to a first weight, and the feature extraction module parameters corresponding to the inferior modal class are fused with its initial feature extraction module parameters according to a second weight.

[0141] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0142] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0143] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0144] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A three-dimensional human skeleton posture estimation model training method, characterized in that: The method comprises the following steps: Acquire a training sample set, wherein each sample in the training sample set contains four types of modal data generated for the sample object, including image data, laser radar data, millimeter wave radar data, and wireless channel data, and the sample contains a label of a true value of a three-dimensional human skeleton posture; The three-dimensional human skeleton posture estimation model is trained by using the training sample set, and the three-dimensional human skeleton posture estimation model includes an image feature extraction module, a first point cloud feature extraction module, a second point cloud feature extraction module, a wireless channel feature extraction module, an inter-modal feature fusion module and a three-dimensional coordinate estimation module; the image feature extraction module is used to extract the image features of the image data, the first point cloud feature extraction module is used to extract the laser radar features of the laser radar data, the second point cloud feature extraction module is used to extract the millimeter wave radar features of the millimeter wave radar data, and the wireless channel feature extraction module is used to extract the wireless channel features of the wireless channel data; the inter-modal feature fusion module is used to fuse the image features, the laser radar features, the millimeter wave radar features and the wireless channel features, and then input them into the three-dimensional coordinate estimation module to output a three-dimensional human skeleton posture prediction value, based on the root mean square error of the three-dimensional human skeleton posture prediction value and the true value, and introduce a plastic weight consolidation loss to construct a joint loss, minimize the joint loss to update the parameters of the three-dimensional human skeleton posture estimation model, and obtain a target three-dimensional human skeleton posture estimation model; Among them, in each round of iteration, the current three-dimensional human skeleton posture estimation model is used for fast fitting and the single modal contribution values ​​of the four types of modal data are calculated based on the Shapley value method, and the four types of modalities are divided into superior modal classes and inferior modal classes according to the single modal contribution values ​​through a preset clustering algorithm; the single modal data whose single modal contribution values ​​of the current batch of samples decrease by a set condition compared with the previous batch of samples are defined as noise modal data; the current batch of samples are used for simulation training, wherein the fitting of the corresponding feature extraction module is suspended for the noise modal data belonging to the superior modal class, and the noise modal data belonging to the inferior class is suspended. The fitting of the corresponding feature extraction module of the noise modal data of the isomodal class is enhanced based on resampling, and two Gaussian distributions with different means are established for the root mean square error generated by the simulation training based on the Gaussian mixture model, and the noise data corresponding to the higher mean is discarded; the three-dimensional human skeleton posture estimation model is trained using the denoised current batch samples; the feature extraction module parameters corresponding to the superior modal class are fused with the initial feature extraction module parameters according to the first weight, and the feature extraction module parameters corresponding to the inferior modal class are fused with the initial feature extraction module parameters according to the second weight.

2. The three-dimensional human skeleton posture estimation model training method according to claim 1, characterized in that: The image feature extraction module adopts four convolutional networks and three dilated convolutional networks; the first point cloud feature extraction module adopts a first five-layer self-attention network, each layer of the first five-layer self-attention network is connected to a downsampling layer and a first average pooling layer is connected to the final output of the first five-layer self-attention network; the second point cloud feature extraction module adopts a second five-layer self-attention network, and the output of the second five-layer self-attention network is connected to a second average pooling layer; the wireless channel feature extraction module adopts three residual neural networks, a self-attention network and a two-dimensional average pooling layer fusion network.

3. The three-dimensional human skeleton posture estimation model training method according to claim 1, characterized in that: The step of the inter-modality feature fusion module fusing the image feature, the laser radar feature, the millimeter wave radar feature and the wireless channel feature comprises: The image features, the laser radar features, the millimeter wave radar features, and the wireless channel features are uniformly mapped to the same feature space through the fully connected layer of the inter-modal feature fusion module and unified to a preset dimension through dimensional transformation; The image features, the lidar features, the millimeter-wave radar features and the wireless channel features that are in the same feature space and have the same dimension are spliced ​​to obtain spliced ​​features, and the spliced ​​features are input into the four self-attention layers of the inter-modal feature fusion module for feature fusion and average pooling is performed to output multi-modal fusion features.

4. The three-dimensional human skeleton posture estimation model training method according to claim 1, characterized in that: The expression for calculating the single-mode m contribution value of the four types of modal data based on the Shapley value method is: V(S,m)=s(p 3D ,MM(S∪{m}))-s(p 3D ,MM(S)); in, represents the set of modes m, and S is represented by subset of S, V(S,m) represents the additional benefit obtained by merging mode m into subset S, p 3D Represents the true value of the 3D human skeleton posture, represents the three-dimensional human skeleton posture prediction value, MM(·) represents the three-dimensional human skeleton posture estimation model, Indicates the Pearson correlation coefficient between the true value of the 3D human skeleton posture and the predicted value of the 3D human skeleton posture when the 3D human skeleton point is k. j represents the number of 3D human body posture skeleton points.

5. The three-dimensional human skeleton posture estimation model training method according to claim 1, characterized in that: The expression of the root mean square error generated by the simulated training is: Among them, j represents the number of 3D human body posture skeleton points, It represents the true value of the posture of the 3D human skeleton point k when the training batch sample is i, It represents the posture prediction value of the 3D human skeleton point k when the training batch sample is i.

6. The three-dimensional human skeleton posture estimation model training method according to claim 1, characterized in that: For the root mean square error generated by the simulation training, two Gaussian distributions with different means are established based on the Gaussian mixture model, and the noise data corresponding to the one with the higher mean is discarded. The noise sample composed of the noise data corresponding to the higher mean is A normal sample consisting of normal data corresponding to the one with a lower mean The expression is: in, represents the complete sample, represents the sample points in the complete sample, represents the posterior probability distribution when the training batch sample is i, g represents the distribution component with the lower mean in the Gaussian mixture model, represents the root mean square error when the training batch sample is i, represents samples without noise data, Represents a sample containing noisy data.

7. The three-dimensional human skeleton posture estimation model training method according to claim 1, characterized in that: Based on the root mean square error of the predicted value and the true value of the three-dimensional human skeleton posture, the expression of the joint loss constructed by introducing the plastic weight consolidation loss is: in, represents the root mean square error generated by the simulated training, λ represents the preset hyperparameter, represents the plastic weight consolidation loss, j represents the number of 3D human body posture skeleton points, i represents the i-th sample in the training batch, It represents the true value of the posture of the 3D human skeleton point k when the training batch sample is i, It indicates the posture prediction value when the training batch sample is i and the 3D human skeleton point is k. Indicated by Sample of time The calculated adaptive Fisher information matrix, θ t,m express The mth feature extraction module parameters at time, express The feature extraction module parameters at the moment, represents the information matrix calculated when the training batch sample is i, Represents the feature extraction module parameter θ t The gradient of , β represents the preset hyperparameter.

8. According to the three-dimensional human skeleton posture estimation model training method of claim 1, the feature extraction module parameters corresponding to the superior modality class are fused with the initial feature extraction module parameters according to the first weight, and the feature extraction module parameters corresponding to the inferior modality class are fused with the initial feature extraction module parameters according to the second weight. The expression is: in, represents the first weight, α I represents the second weight, represents the initial feature extraction module parameters of mode m, represents the feature extraction module parameters of mode m when the preset iteration round is l, represents the superior modal class, Represents the secondary modal class.

9. A three-dimensional human skeleton posture estimation method, characterized in that: The method comprises the following steps: Obtain four types of modal data to be predicted; The four types of modal data to be predicted are input into the target three-dimensional human skeleton posture estimation model obtained by the three-dimensional human skeleton posture estimation model training method as described in any one of claims 1 to 8 to obtain the three-dimensional human skeleton posture.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method as claimed in any one of claims 1 to 9 are implemented.

Images