A three-dimensional human skeleton pose estimation method and a model training method

Abstract

The application provides a three-dimensional human skeleton posture estimation method and a model training method. Four types of modal data are input into a feature extraction module, four types of features are extracted, fused through an inter-modal feature fusion module, and then input into a three-dimensional coordinate estimation module to output a three-dimensional human skeleton posture prediction value. In each round of iterative training of the three-dimensional human skeleton posture estimation model, a single-modal contribution value is calculated based on a Shannon value method and modal categories are divided. The current batch of samples are simulated and trained, different modal category noise modal data are fitted respectively, two Gaussian distributions are established according to the root mean square error of the simulation training, the current batch of samples after the noise data corresponding to the higher mean value are removed are used to train the three-dimensional human skeleton posture estimation model, a joint loss is constructed based on the root mean square error and the plastic weight consolidation loss of the prediction value and the true value, the three-dimensional human skeleton posture estimation model parameters are updated, and the initial feature extraction module parameters are respectively fused according to preset weights according to the feature extraction module parameters corresponding to different modes.

Description

Technical Field

[0001] The present invention relates to the technical field of image and video information processing, and in particular to a three-dimensional human skeleton posture estimation method and a model training method. Background Art

[0002] Three-dimensional human skeletal pose estimation is the process of recovering the three-dimensional coordinates of human joints from various input sources. It has attracted extensive research attention in applications such as human-computer interaction and computer animation. Camera-based input images and videos are easy for users to operate and the captured data contains rich human body information. However, they have limitations under complex lighting conditions and require accurate camera parameters for complex two-dimensional to three-dimensional space conversion. Multimodal human skeletal pose estimation methods, which use multiple sensor modes, have shown broad application prospects. However, multimodal learning suffers from modal imbalance. The optimization process of the feature extraction module of the main modality data will inhibit the optimization process of the other modal data, resulting in the final three-dimensional human skeletal pose estimation model performing worse than the theoretical optimal performance.

[0003] In the existing technology, single-modal human skeleton pose estimation methods use deep learning models to extract two-dimensional human skeleton pose features from videos or images and map them to three-dimensional coordinates. When the image is blurred due to lighting or extreme weather conditions, the human skeleton pose estimation results will be greatly affected. Multimodal methods only use two modal data and still have disadvantages when facing complex scenes such as low lighting, no lighting, or extreme weather conditions. Three or more modal data have not been fully explored and studied. In addition, existing solutions to the modal imbalance problem in multimodal learning mainly quantify the suppression ratio between modalities to adaptively adjust the training process of the main modality to accelerate the training and optimization of other modalities, or add additional training modules to each modal branch to balance the learning process between modalities. This method has not been applied to regression tasks such as pose estimation and ignores the inherent limitations of modal data. When some modalities lack task-related information or contain more noise, there will be a gap in the task-related information content between different modal data, resulting in the inability to solve the modal imbalance problem. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides a three-dimensional human skeleton posture estimation method and a model training method to eliminate or improve one or more defects in the existing technology, and solves the problem of poor three-dimensional human skeleton posture estimation effect in the existing technology.

[0005] One aspect of the present invention provides a method for training a 3D human skeleton posture estimation model, the method comprising the following steps:

[0006] obtain a training sample set, each sample in the training sample set containing four types of modal data generated for a sample object, including image data, lidar data, millimeter wave radar data and wireless channel data, and the sample containing a label of a three-dimensional human body skeleton pose true value;

[0007] train the three-dimensional human body skeleton pose estimation model using the training sample set, the three-dimensional human body skeleton pose estimation model including an image feature extraction module, a first point cloud feature extraction module, a second point cloud feature extraction module, a wireless channel feature extraction module, an inter-modal feature fusion module, and a three-dimensional coordinate estimation module; the image feature extraction module is used to extract image features of the image data, the first point cloud feature extraction module is used to extract lidar features of the lidar data, the second point cloud feature extraction module is used to extract millimeter wave radar features of the millimeter wave radar data, and the wireless channel feature extraction module is used to extract wireless channel features of the wireless channel data; the inter-modal feature fusion module is used to input the image features, the lidar features, the millimeter wave radar features and the wireless channel features after fusion into the three-dimensional coordinate estimation module to output a three-dimensional human body skeleton pose prediction value, based on a root mean square error of the three-dimensional human body skeleton pose prediction value and the true value, and introducing a plastic weight to consolidate the loss to construct a joint loss, minimizing the joint loss to update the parameters of the three-dimensional human body skeleton pose estimation model, and obtaining a target three-dimensional human body skeleton pose estimation model;

[0008] In each iteration, the current three-dimensional human body skeleton pose estimation model is used for fast fitting, and the single-modal contribution value of the four types of modal data is calculated based on the Sharpe value method, and the four types of modal data are divided into a superior modal class and an inferior modal class according to the single-modal contribution value by a preset clustering algorithm; the single-modal data whose single-modal contribution value decreases by a set condition from the last batch of samples is defined as noise modal data; the current batch of samples are used for simulation training, wherein the fitting of the corresponding feature extraction module for the noise modal data belonging to the superior modal class is suspended, the fitting of the corresponding feature extraction module for the noise modal data belonging to the inferior modal class is strengthened based on resampling, and the root mean square error generated by the simulation training is based on a Gaussian mixture model to establish two Gaussian distributions with different means and discard the noise data corresponding to the Gaussian distribution with the higher mean; the three-dimensional human body skeleton pose estimation model is trained using the current batch of samples after denoising; the parameters of the feature extraction module corresponding to the superior modal class are fused with the initial feature extraction module parameters according to a first weight, and the parameters of the feature extraction module corresponding to the inferior modal class are fused with the initial feature extraction module parameters according to a second weight.

[0009] In some embodiments, the image feature extraction module adopts four convolutional networks and three dilated convolutional networks; the first point cloud feature extraction module adopts a first five-layer self-attention network, each layer of the first five-layer self-attention network is followed by a downsampling layer and a first average pooling layer is connected to the final output of the first five-layer self-attention network; the second point cloud feature extraction module adopts a second five-layer self-attention network, and the output of the second five-layer self-attention network is connected to a second average pooling layer; the wireless channel feature extraction module adopts three residual neural networks, a self-attention network and a two-dimensional average pooling layer fusion network.

[0010] In some embodiments, the step of the inter-modality feature fusion module fusing the image feature, the lidar feature, the millimeter-wave radar feature, and the wireless channel feature includes:

[0011] The image features, the lidar features, the millimeter-wave radar features, and the wireless channel features are uniformly mapped to the same feature space through the fully connected layer of the inter-modal feature fusion module and unified to a preset dimension through dimensional transformation;

[0012] The image features, the lidar features, the millimeter-wave radar features, and the wireless channel features that are in the same feature space and have the same dimension are spliced ​​to obtain spliced ​​features, and the spliced ​​features are input into the four self-attention layers of the inter-modal feature fusion module for feature fusion and average pooling is performed to output multimodal fusion features.

[0013] In some embodiments, the expression for calculating the single-modal m-contribution value of the four types of modal data based on the Shapley value method is:

[0014]

[0015] V(S,m)=s(p 3D , MM(S∪{m}))-s(p 3D , MM(S));

[0016]

[0017] in, represents the set of modes m, and S is represented by subset of S, V(S, m) represents the additional benefit obtained by merging mode m into subset S, p 3D Represents the true value of the 3D human skeleton posture, represents the three-dimensional human skeleton posture prediction value, MM(·) represents the three-dimensional human skeleton posture estimation model, Indicates the Pearson correlation coefficient between the true value of the 3D human skeleton posture and the predicted value of the 3D human skeleton posture when the 3D human skeleton point is k. j represents the number of 3D human body posture skeleton points.

[0018] In some embodiments, the expression of the root mean square error generated by the simulation training is:

[0019]

[0020] Among them, j represents the number of 3D human body posture skeleton points, It represents the true value of the posture of the 3D human skeleton point k when the training batch sample is i. It represents the posture prediction value of the 3D human skeleton point k when the training batch sample is i.

[0021] In some embodiments, the root mean square error generated by the simulation training is used to establish two Gaussian distributions with different means based on the Gaussian mixture model and the noise data corresponding to the one with the higher mean is discarded. The noise sample composed of the noise data corresponding to the one with the higher mean is A normal sample consisting of normal data corresponding to the one with a lower mean The expression is:

[0022]

[0023]

[0024]

[0025] in, represents the complete sample, represents the sample points in the complete sample, represents the posterior probability distribution when the training batch sample is i, g represents the distribution component with the lower mean in the Gaussian mixture model, represents the root mean square error when the training batch sample is i, represents a sample without noise data, Represents a sample containing noisy data.

[0026] In some embodiments, the root mean square error between the predicted value and the true value of the three-dimensional human skeleton posture is calculated by introducing the plastic weight consolidation loss to construct the expression of the joint loss:

[0027]

[0028]

[0029]

[0030]

[0031]

[0032] in, represents the root mean square error generated by the simulation training, λ represents the preset hyperparameter, represents the plastic weight consolidation loss, j represents the number of 3D human posture skeleton points, i represents the i-th sample in the training batch, It represents the true value of the posture of the 3D human skeleton point k when the training batch sample is i. It represents the posture prediction value when the training batch sample is i and the 3D human skeleton point is k. Indicated by Sample of the moment The calculated adaptive Fisher information matrix, θ t，m express The parameters of the mth feature extraction module at time, express The feature extraction module parameters at the moment, represents the information matrix calculated when the training batch sample is i, Represents the feature extraction module parameter θ t The gradient of , β represents the preset hyperparameter.

[0033] In some embodiments, the feature extraction module parameters corresponding to the superior modality class are fused with the initial feature extraction module parameters according to a first weight, and the feature extraction module parameters corresponding to the inferior modality class are fused with the initial feature extraction module parameters according to a second weight. The expression is:

[0034]

[0035] in, represents the first weight, α I represents the second weight, represents the initial feature extraction module parameters of mode m, represents the feature extraction module parameters of mode m when the preset iteration round is l, represents the superior modal class, Represents the secondary modal class.

[0036] In another aspect, the present invention provides a method for estimating a 3D human skeleton posture, the method comprising the following steps:

[0037] Obtain four types of modal data to be predicted;

[0038] The four types of modal data to be predicted are input into the target three-dimensional human skeleton posture estimation model obtained by any one of the three-dimensional human skeleton posture estimation model training methods described above to obtain the three-dimensional human skeleton posture.

[0039] On the other hand, the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of any one of the above methods when executed by a processor.

[0040] The beneficial effects of the present invention are at least:

[0041] In the three-dimensional human skeleton posture estimation method and model training method described in the present invention, four types of modal data containing labels of true values ​​of three-dimensional human skeleton posture are input into four types of feature extraction modules in the three-dimensional human skeleton posture estimation model to respectively extract features of the four types of data, deeply explore human skeleton posture information and improve the expressiveness and robustness of the three-dimensional human skeleton posture estimation model; the four types of features are input into the inter-modal feature fusion module to obtain multimodal fusion features and input into the three-dimensional coordinate estimation module to obtain three-dimensional human skeleton posture prediction values; during the training process of the three-dimensional human skeleton posture estimation model, the single modal contribution values ​​of the four types of modal data are calculated based on the Shapley value method to judge the training status of the three-dimensional human skeleton posture estimation model, and the corresponding feature extraction modules are fitted according to the noise modal data and the modal category, and the mean square error of the fitting process is calculated. The root error establishes two Gaussian distributions with different means based on the Gaussian mixture model and discards the noise data corresponding to the one with the higher mean, thereby improving the training effect by removing the noise data that affects the training in the sample; a joint loss is constructed based on the root mean square error of the three-dimensional human skeleton posture prediction value and the true value and the plastic weight consolidation loss, which is beneficial for the three-dimensional human skeleton posture estimation model to overcome forgetting and avoid the influence of noise on the model; when multiple samples in each batch undergo multiple rounds of iterative training and reach the preset iteration round, the first weight and the second weight are set to help the feature extraction module reshape the learning state and control the intensity of re-learning, thereby improving multimodal learning. The fusion of the feature extraction module parameters with the initial feature extraction module parameters can retain the feature extraction module while enhancing the generalization ability of the three-dimensional human skeleton estimation model and improving the three-dimensional human skeleton posture estimation effect.

[0042] Additional advantages, objects, and features of the present invention will be set forth in part in the following description and will become apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained by the structures particularly pointed out in the description and drawings.

[0043] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present invention are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. In the drawings:

[0045] Figure 1 The figure is a flow chart of a method for training a 3D human skeleton pose estimation model according to an embodiment of the present invention.

[0046] Figure 2 Schematic diagram of the structure of a 3D human skeleton posture estimation model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0048] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0049] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0050] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0051] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0052] In the prior art, the single-modal human skeleton posture estimation method extracts two-dimensional human skeleton posture features from videos or images through a deep learning model and maps them to three-dimensional coordinates. When the image is blurred due to lighting or extreme climate conditions, the human skeleton posture estimation result will be greatly affected; the multimodal method only uses two modal data, and still has disadvantages when facing complex scenes such as weak lighting, no lighting or extreme climate conditions; the present invention proposes a three-dimensional human skeleton posture estimation method and a model training method, which inputs four types of modal data containing labels of true values ​​of three-dimensional human skeleton posture into the feature extraction module in the three-dimensional human skeleton posture estimation model, and inputs the extracted four types of features into the inter-modal feature fusion module for fusion and then inputs into the three-dimensional coordinate estimation module to output the three-dimensional human skeleton posture prediction value; in each round of iterative training of the three-dimensional human skeleton posture estimation model, the current three-dimensional human skeleton posture estimation model is used for fast fitting, and the single modal contribution values ​​of the four types of modal data are calculated based on the Shapley value method, and the four types of modal data are clustered by a preset clustering algorithm. The modalities are divided into superior modal classes and inferior modal classes; the unimodal data with decreased unimodal contribution values ​​of the current batch of samples are regarded as noise modal data and subjected to simulation training, and the noise modal data belonging to the superior modal class and the noise modal data belonging to the inferior modal class are fitted respectively. The root mean square error generated by the simulation training is used to establish two Gaussian distributions with different means based on the Gaussian mixture model and the noise data corresponding to the one with the higher mean is discarded; the denoised current batch of samples is used to train the three-dimensional human skeleton posture estimation model, and the root mean square error between the predicted value and the true value of the three-dimensional human skeleton posture is based on the introduction of plastic weight consolidation loss to construct a joint loss, and the joint loss is minimized to update the parameters of the three-dimensional human skeleton posture estimation model. When the preset iteration round is reached during the training of each batch of samples, the feature extraction module parameters corresponding to the superior modal class are fused with the initial feature extraction module parameters according to the first weight, and the feature extraction module parameters corresponding to the inferior modal class are fused with the initial feature extraction module parameters according to the second weight.

[0053] Figure 1 The figure is a flow chart of a method for training a 3D human skeleton pose estimation model according to an embodiment of the present invention. Specifically, the present application provides a method for training a 3D human skeleton pose estimation model, which includes the following steps S101 to S102:

[0054] Step S101: Obtain a training sample set, where each sample in the training sample set contains four types of modal data generated for the sample object, including image data, lidar data, millimeter-wave radar data, and wireless channel data. The sample contains a label of the true value of the three-dimensional human skeleton posture.

[0055] Step S102: A training sample set is used to train a three-dimensional human skeleton posture estimation model, where the three-dimensional human skeleton posture estimation model includes an image feature extraction module, a first point cloud feature extraction module, a second point cloud feature extraction module, a wireless channel feature extraction module, an inter-modal feature fusion module, and a three-dimensional coordinate estimation module; the image feature extraction module is used to extract image features of image data, the first point cloud feature extraction module is used to extract laser radar features of laser radar data, the second point cloud feature extraction module is used to extract millimeter wave radar features of millimeter wave radar data, and the wireless channel feature extraction module is used to extract wireless channel features of wireless channel data; the inter-modal feature fusion module is used to fuse image features, laser radar features, millimeter wave radar features, and wireless channel features, and then input them into the three-dimensional coordinate estimation module to output a three-dimensional human skeleton posture prediction value, based on the root mean square error between the three-dimensional human skeleton posture prediction value and the true value, and introducing a plastic weight consolidation loss to construct a joint loss, and minimizing the joint loss to update the parameters of the three-dimensional human skeleton posture estimation model to obtain a target three-dimensional human skeleton posture estimation model.

[0056] In each round of iteration, the current three-dimensional human skeleton posture estimation model is used for rapid fitting and the single-modal contribution values ​​of the four types of modal data are calculated based on the Shapley value method. The four types of modalities are divided into superior modal classes and inferior modal classes according to the single-modal contribution values ​​through a preset clustering algorithm. The single-modal data whose single-modal contribution values ​​of the current batch samples decrease by a set condition compared with the previous batch samples are defined as noise modal data. The current batch samples are used for simulation training, wherein the fitting of the corresponding feature extraction module is suspended for the noise modal data belonging to the superior modal class, and the fitting of the corresponding feature extraction module is strengthened based on resampling for the noise modal data belonging to the inferior modal class. The root mean square error generated by the simulation training is used to establish two Gaussian distributions with different means based on the Gaussian mixture model, and the noise data corresponding to the higher mean is discarded. The denoised current batch samples are used to train the three-dimensional human skeleton posture estimation model. The feature extraction module parameters corresponding to the superior modal class are fused with the initial feature extraction module parameters according to the first weight, and the feature extraction module parameters corresponding to the inferior modal class are fused with the initial feature extraction module parameters according to the second weight.

[0057] In step S101, the four types of modal data in the training sample set are preprocessed, and the high-resolution network (HRNet) is used to process the image data and extract the two-dimensional human skeleton posture data; the farthest point sampling method is used for the lidar data to obtain a point cloud of a preset number of input points. The farthest point sampling method is an algorithm for achieving uniform sampling. It selects the farthest point from the existing sampling point set through continuous iteration, and finally obtains a set of sampling points with relatively uniform distribution in the point cloud; the farthest point sampling method first randomly selects a point from the input point cloud as the initial point and adds it to the sampling point set. In each iteration, the distance from each point in the input point cloud to all points in the sampling point set is calculated and the point farthest from the sampling point set is selected as the next sampling point and added to the sampling point set. Until the points in the sampling point set reach a preset 1024 points; the millimeter-wave radar data is aggregated into a point cloud of a preset number of input points using the adjacent frame method, and the preset number of the present invention is 128; the adjacent frame method samples the data of two or more adjacent frames while sampling a certain frame of data, and performs feature extraction and data fusion on the sampled data; the extracted features include but are not limited to color features, texture features and edge features, and the data fusion method includes splicing the features of each frame and weighted summing the features of each frame according to preset weights; wavelet denoising is used to remove noise from the wireless channel data. Wavelet denoising first uses wavelet transform to decompose the signal into components of different scales and frequencies, removes the noise components in the signal through threshold processing, and reconstructs the denoised signal through inverse wavelet transform.

[0058] In step S102, the four types of modal data use corresponding feature extraction modules to extract image features, lidar features, millimeter wave radar features and wireless channel features.

[0059] In some embodiments, the image feature extraction module adopts four convolutional networks and three dilated convolutional networks; the first point cloud feature extraction module adopts a first five-layer self-attention network, each layer of the first five-layer self-attention network is followed by a downsampling layer and a first average pooling layer is connected to the final output of the first five-layer self-attention network; the second point cloud feature extraction module adopts a second five-layer self-attention network, and the output of the second five-layer self-attention network is connected to a second average pooling layer; the wireless channel feature extraction module adopts three residual neural networks, a self-attention network and a two-dimensional average pooling layer fusion network.

[0060] Specifically, the image feature extraction module adopts a four-layer convolutional network including four convolutional networks and three dilated convolutional networks. The image data is input into the four-layer convolutional network to obtain image features. The dilated convolutional network expands the coverage of the convolution kernel without increasing the amount of calculation. After combining with the convolutional network, it extracts more global features while maintaining high resolution; in the first point cloud feature extraction module, the first five-layer attention network includes a preset number of attention heads and an attention output layer. The attention head extracts features from the lidar data through the attention calculation algorithm, the softmax function for normalizing the attention weights, and matrix multiplication. The dimension is reduced by downsampling, and finally the lidar features are obtained by fusing features through the first average pooling layer; in the second point cloud feature extraction module, the second The five-layer self-attention network includes a preset number of attention heads and an attention output layer. The attention heads extract features from millimeter-wave radar data through an attention calculation algorithm, a softmax function for normalizing the attention weights, and matrix multiplication. The extracted features are fused through the second average pooling layer. In the wireless channel feature extraction module, three residual neural networks are used to extract features from the three antenna data of the wireless channel data. Antennas are used to send and receive signals in wireless communication devices. The three antenna data are data related to the signals sent and received by the three antennas. The three features are spliced ​​and flattened to obtain flat features. After the flat features are interacted and fused through the self-attention network and the original feature form is restored, they are input into the two-dimensional average pooling layer to obtain the wireless channel features.

[0061] In some embodiments, the inter-modality feature fusion module fuses the image features, the lidar features, the millimeter-wave radar features, and the wireless channel features, including steps S1021 to S1022:

[0062] Step S1021: The image features, lidar features, millimeter-wave radar features, and wireless channel features are uniformly mapped to the same feature space through the fully connected layer of the inter-modal feature fusion module and unified to the preset dimension through dimensional transformation.

[0063] Step S1022: The image features, lidar features, millimeter-wave radar features, and wireless channel features that are in the same feature space and have the same dimension are spliced ​​to obtain spliced ​​features, and the spliced ​​features are input into the four self-attention layers of the inter-modal feature fusion module. The spliced ​​features are fused and average pooled to output multimodal fusion features.

[0064] Specifically, the four types of modal features are mapped to the same feature space through the four fully connected layers of the inter-modal feature fusion module. Each fully connected layer contains two linear layers and one normalization layer. In order to further enhance the feature expression capability and adapt to the task, the dimensions of the image features, lidar features, millimeter-wave radar features and wireless channel features extracted by the four types of feature extraction modules are adjusted to the same preset dimension size through dimensional transformation to facilitate subsequent feature fusion. The preset dimension size of the present invention is 256; the dimensional transformation method includes but is not limited to principal component analysis, linear discriminant analysis and polynomial feature expansion; four layers of self-attention layers and average pooling layers perform deep feature fusion to obtain more comprehensive human skeleton posture features; the multimodal fusion features are input into the three-dimensional coordinate estimation module. The three-dimensional coordinate estimation module uses multi-layer linear layers and batch normalization layers to perform multiple feature transformations and normalization operations to obtain three-dimensional coordinate values. There are 51 three-dimensional coordinate values ​​in the present invention. The three-dimensional coordinate values ​​are matrix transformed to obtain three-dimensional human skeleton posture prediction values.

[0065] Furthermore, the contribution value of each modal data calculated by the Shapley value method is used to judge the relative importance of different modal features to the final output result and the training status of the three-dimensional human skeleton posture estimation model; the preset clustering algorithm of the present invention adopts k-means clustering (k-means clustering), firstly selects two initial cluster centers for representing the superior modal class and the inferior modal class, wherein the cluster center with a larger contribution value contains the modality as the superior modal class, and the cluster center with a smaller contribution value contains the modality as the inferior modal class, calculates the distance of each modality to the two initial cluster centers and assigns the modality to the category with a closer distance, and after all modalities are assigned, the modalities are redistributed by calculating the mean of all modalities in the two categories as the cluster centers of the two categories, until the clustering result is stable and the modality no longer shifts categories to complete the division of the superior modal class and the inferior modal class.

[0066] In some embodiments, the expression for calculating the single-modal m-contribution value of four types of modal data based on the Shapley value method is:

[0067]

[0068] V(S,m)=s(p 3D ,MM(S∪{m}))-s(p 3D ,MM(S));

[0069]

[0070] in, represents the set of modes m, and S is represented by subset of S, V(S,m) represents the additional benefit obtained by merging mode m into subset S, p 3D Represents the true value of the 3D human skeleton posture, represents the three-dimensional human skeleton posture prediction value, MM(·) represents the three-dimensional human skeleton posture estimation model, Indicates the Pearson correlation coefficient between the true value of the 3D human skeleton posture and the predicted value of the 3D human skeleton posture when the 3D human skeleton point is k. j represents the number of 3D human body posture skeleton points.

[0071] Furthermore, noise data affects the training process of the three-dimensional human skeleton posture estimation model, and is removed by the noise recognition and separation module. The noise recognition and classification module compares the single modal contribution value of the current batch with that of the previous batch. When the single modal contribution value in the current batch samples decreases to the set condition, the single modal data is judged to be noise modal data; during simulation training, when the noise modal data is a superior modal class, the corresponding feature extraction module stops fitting the superior modal class. When the noise modal data is a sub-optimal modal class, resampling is performed and fitting is continued through the corresponding feature extraction module, so that the distribution difference between the noise data and the normal data becomes larger. The root mean square error generated during the simulation training is modeled by the Gaussian mixture model. Two Gaussian distributions are obtained by modeling and the modal data with a higher mean is used as noise data.

[0072] In some embodiments, the expression of the root mean square error generated by the simulation training is:

[0073]

[0074] Among them, j represents the number of 3D human body posture skeleton points, It represents the true value of the posture of the 3D human skeleton point k when the training batch sample is i. It represents the posture prediction value of the 3D human skeleton point k when the training batch sample is i.

[0075] In some embodiments, the root mean square error generated by the simulation training is used to establish two Gaussian distributions with different means based on the Gaussian mixture model and the noise data corresponding to the one with the higher mean is discarded. The noise sample composed of the noise data corresponding to the one with the higher mean is A normal sample consisting of normal data corresponding to the one with a lower mean The expression is:

[0076]

[0077]

[0078]

[0079] in, represents the complete sample, represents the sample points in the complete sample, represents the posterior probability distribution when the training batch sample is i, g represents the distribution component with the lower mean in the Gaussian mixture model, represents the root mean square error when the training batch sample is i, represents a sample without noise data, Represents a sample containing noisy data.

[0080] In some embodiments, the root mean square error between the predicted value and the true value of the 3D human skeleton posture and the introduction of the plastic weight consolidation loss to construct the expression of the joint loss is:

[0081]

[0082]

[0083]

[0084]

[0085]

[0086] in, represents the root mean square error generated by the simulation training, λ represents the preset hyperparameter, represents the plastic weight consolidation loss, j represents the number of 3D human posture skeleton points, i represents the i-th sample in the training batch, It represents the true value of the posture of the 3D human skeleton point k when the training batch sample is i. It represents the posture prediction value when the training batch sample is i and the 3D human skeleton point is k. Indicated by Sample of the moment The calculated adaptive Fisher information matrix, θ t,m express The parameters of the mth feature extraction module at time, express The feature extraction module parameters at the moment, represents the information matrix calculated when the training batch sample is i, Represents the feature extraction module parameter θ t The gradient of , β represents the preset hyperparameter.

[0087] Specifically, the posterior probability distribution considers the root mean square error information, and the posterior probability can better adapt to the data requirements of model training based on the probability of the distribution component with a lower mean value obtained according to the error information. According to the comparison between the posterior probability distribution and 0.5, the modal data less than 0.5 is divided into the sample containing noise data to obtain noise samples, and otherwise, it is divided into the sample not containing noise data to obtain normal samples. The normal samples can improve the accuracy of the three-dimensional human skeleton pose estimation model training and avoid the influence of noise data on the model performance; the expectation maximization algorithm is used to iteratively update the initial mean and variance of the Gaussian mixture model, and two Gaussian distributions with different mean and variance are established according to the obtained two mean and variance.

[0088] Further, the adaptive plastic weight consolidates the loss to improve the robustness of the knowledge memory of the three-dimensional human skeleton pose estimation model and avoid the influence of noise data on the knowledge memory; the strength of the plastic weight consolidation loss is adjusted by a preset hyperparameter λ, and the joint loss function is obtained by adding the root mean square error.

[0089] In some embodiments, the feature extraction module parameters corresponding to the superior modal class are fused with the initial feature extraction module parameters thereof according to a first weight, and the feature extraction module parameters corresponding to the inferior modal class are fused with the initial feature extraction module parameters thereof according to a second weight, and the expression is as follows:

[0090]

[0091] wherein, the first weight is denoted as α I the second weight is denoted as β, the initial feature extraction module parameter of the modal m is denoted as θ the feature extraction module parameter of the modal m at a preset iteration round l is denoted as θ the superior modal class is denoted as C the inferior modal class is denoted as C.

[0092] Specifically, during the multiple iteration training processes of each batch of samples, the feature extraction module parameters corresponding to the superior modal class and the inferior modal class are respectively fused with the corresponding initial feature extraction module parameters when the iteration training reaches a preset round. The adaptive relearning method sets different feature extraction module weights according to the superior modal class and the inferior modal class to control the strength of relearning, thereby reshaping the learning state of the feature extraction module. The initial feature extraction module parameters and the trained feature extraction parameters are weighted and fused to achieve better performance.

[0093] On the other hand, the application provides a three-dimensional human skeleton pose estimation method, which comprises the following steps S201-S202:

[0094] Step S201: Obtain four types of modal data to be predicted.

[0095] Step S202: input the four types of modal data to be predicted into the target three-dimensional human skeleton pose estimation model obtained by any of the above three-dimensional human skeleton pose estimation model training methods to obtain a three-dimensional human skeleton pose.

[0096] On the other hand, the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of any one of the above methods when executed by a processor.

[0097] The present invention will be described below in conjunction with a specific embodiment:

[0098] Figure 2 This figure is a schematic diagram of the structure of a 3D human skeletal pose estimation model according to one embodiment of the present invention. This invention utilizes a multimodal 3D human skeletal pose estimation model for human skeletal pose estimation; optimizes the model's training process for better performance using a balanced multimodal learning method; and utilizes a denoising continuous learning method to help a multimodal network achieve multi-task learning.

[0099] 1. Obtain training sample sets and labels.

[0100] A high-resolution network is used to extract and save the two-dimensional human skeleton posture data from the image data in the training sample set. The farthest point sampling method is used for the lidar data in the training sample set to obtain a point cloud of 1024 input points. The adjacent frame method is used to aggregate the millimeter-wave radar data in the training sample set to a point cloud of 128 input points. Wavelet denoising is used for the wireless channel data in the training sample set to obtain denoised data. The data of these four modalities are used as the input of the subsequent network to obtain the true value p of the three-dimensional human skeleton posture. 3D ∈R 17×3 Used as labels to calculate the loss.

[0101] 2. Construct the image feature extraction module, the first point cloud feature extraction module, the second point cloud feature extraction module, the wireless channel feature extraction module, the inter-modal feature fusion module and the three-dimensional coordinate estimation module.

[0102] 2.1 Construct the image feature extraction module, the first point cloud feature extraction module, the second point cloud feature extraction module, and the wireless channel feature extraction module. Different feature extraction modules are used for different modal data. The image feature extraction module is a four-layer convolutional network, which contains four convolutional networks and three dilated convolutional networks. The first point cloud feature extraction module is a five-layer self-attention network, which includes n attention heads. The attention head consists of an attention calculation algorithm, a softmax for normalizing the attention weights, and matrix multiplication. Each layer of the self-attention network is followed by a downsampling layer, and the five-layer self-attention network is followed by an average pooling layer to fuse data features. The second point cloud feature extraction module is also a five-layer self-attention network, but without a downsampling layer. It is only followed by an average pooling layer at the end of the five-layer self-attention network. The wireless channel feature extraction module includes three residual neural networks. For the features extracted by these three residual networks, a self-attention network is used for feature interaction and fusion, and a two-dimensional average pooling layer is used to fuse the features at the end.

[0103] 2.2 Construct an inter-modal feature fusion module. The network consists of four fully connected layers, a four-layer self-attention layer and an average pooling layer. Each fully connected layer contains two linear layers and a layer normalization operation. The fully connected layer performs feature mapping on the features extracted by each modality feature extraction network to achieve better fusion effect. The four-layer self-attention layer and the average pooling layer perform deep feature fusion.

[0104] 2.3 Construct a 3D coordinate estimation module, which includes multiple linear layers and batch normalization layers.

[0105] 3. Input each modal data into each modal feature extraction network and output each modal feature.

[0106] The image data is input into the image feature extraction module, and the image features (1×1024) are extracted through the multi-layer convolutional network; the lidar data is input into the first point cloud feature extraction module. The initial input data is the three-dimensional coordinates of 1024 point cloud points (1024×3). After the first self-attention layer, the point cloud data (1024×32) is obtained, and then four times downsampling is performed to obtain (256×64). After four layers of self-attention layers, the point cloud features (4×512) are obtained, and average pooling is performed to obtain lidar features (1×512). For millimeter wave radar data, the same as the lidar data, can be obtained after four layers. The millimeter-wave features (128×512) are averaged and pooled to obtain the final millimeter-wave radar features (1×512). For the wireless channel data, three residual neural networks are first used to extract features (17×4×512) from the three antenna data of the wireless channel data respectively. Then, these three features are concatenated to obtain the combined features (17×12×512) and flattened to obtain the flat features (20×512). After the features are fused through the self-attention network, they are restored to the original feature form (17×12×512). Finally, two-dimensional average pooling is used to obtain the wireless channel features (1×512).

[0107] 4. Input image features, lidar features, millimeter-wave radar features, and wireless channel features into the inter-modal feature fusion module and output multimodal fusion features.

[0108] The image features, lidar features, millimeter-wave radar features, and wireless channel features are input into the fully connected layer of the inter-modal feature fusion module. The features of each modality are uniformly mapped into the same feature space using the fully connected layer, and the dimension of each feature is unified to 256. These four features are concatenated to obtain the spliced ​​features (4×256) and input into the inter-modal feature fusion module. In the inter-modal feature fusion module, four self-attention layers perform deep feature fusion on the spliced ​​features, and finally average pooling is used to obtain the multi-modal fusion features (1×256).

[0109] 5. Input the multimodal fusion features into the three-dimensional coordinate estimation module, estimate the three-dimensional coordinates of the human skeleton points and output the three-dimensional human skeleton posture.

[0110] The multimodal fusion features are input into the 3D coordinate estimation module, and 51 coordinate values ​​are obtained by multiple feature transformation and normalization operations. These 51 coordinate values ​​are transformed into a matrix to obtain the 3D human skeleton posture.

[0111] 6. Use the balanced multimodal learning method to train the model and obtain a trained model.

[0112] The balanced multimodal learning method consists of three parts: a single-modal contribution value calculation algorithm based on the Shapley value method, an adaptive relearning method, and a root mean square error.

[0113] The contribution module based on the Shapley value method uses a single-modal contribution calculation algorithm based on the Shapley value method to obtain the contribution value of each modality. The calculated contribution value can help the model determine the training status of the overall multimodal model. Among them, the expression of the single-modal m contribution value is:

[0114]

[0115] V(S,m)=s(p 3D ,MM(S∪{m}))-s(p 3D ,MM(S));

[0116]

[0117] in, represents the set of modes m, and S is represented by subset of S, V(S,m) represents the additional benefit obtained by merging mode m into subset S, p 3D Represents the true value of the 3D human skeleton posture, represents the three-dimensional human skeleton posture prediction value, MM(·) represents the three-dimensional human skeleton posture estimation model, Indicates the Pearson correlation coefficient between the true value of the 3D human skeleton posture and the predicted value of the 3D human skeleton posture when the 3D human skeleton point is k. j represents the number of 3D human body posture skeleton points.

[0118] The adaptive relearning module performs an adaptive relearning method based on the calculated contribution scores of each modality to help the feature extraction module relearn to achieve better performance. Specifically, it consists of a score clustering module and a relearning module. The score clustering module takes the contribution values ​​of each modality as input and combines the modalities through k-means clustering. Divided into two categories: superior modal class and submodal classes Since these two modal categories will produce inconsistent training states during the training process, adaptive relearning can help the feature extraction module reshape the learning state, which can be expressed by the following formula:

[0119]

[0120] in, represents the feature extraction module parameters of mode m when the preset iteration round is l, α is an artificially set hyperparameter. and Different α values ​​are set to control the intensity of relearning to achieve better performance. By weightedly fusing the parameters of the feature extraction modules of each modality with the parameters of the initial feature extraction module at training round r, the parameters of the relearned feature extraction module are obtained. This can preserve some feature extraction capabilities while greatly enhancing the model's generalization ability, thereby achieving better estimation performance.

[0121] In balanced multimodal learning, the root mean square error of three-dimensional key points is used as the loss function, which is calculated as follows:

[0122]

[0123] Among them, j represents the number of 3D human body posture skeleton points, It represents the true value of the posture of the 3D human skeleton point k when the training batch sample is i. It represents the posture prediction value of the 3D human skeleton point k when the training batch sample is i.

[0124] 7. Based on the balanced multimodal learning method, the model is trained using a denoising continuous learning method to obtain a trained model. The denoising continuous learning method consists of two parts: a noise identification and separation module and an adaptive plastic weight consolidation module.

[0125] After the new training batch samples arrive, the data of the new training batch samples are first quickly fitted on the existing model, and then the contribution values ​​of the four modal data in the new training batch samples are calculated. The noise identification and separation module will monitor the contribution values, compare the newly added single modal contribution values ​​with the existing single modal contribution values, and obtain the mode with a significantly reduced contribution value, thereby determining it as a noise mode.

[0126] The training data is input into the model and quickly fitted for a small number of rounds, and the root mean square error of each sample is calculated. The training sample is modeled based on the root mean square error using a Gaussian mixture model to obtain two Gaussian distributions with different means and variances. Among them, when the noise mode is in the superior mode class, the fitting of the feature extraction module of the superior class is suspended, thereby increasing the distribution difference between the noise data and the normal data; when the noise mode is in the inferior mode class, the fitting of the inferior mode class is strengthened by resampling, which can also increase the distribution difference between the inferior noise data and the normal data. Finally, for the two Gaussian distributions obtained by modeling, the data with high mean distribution will be discarded when training the 3D human skeleton posture estimation model for noise separation, because this part of the data is identified as noise data, which is not beneficial to the training process. The noise separation calculation based on the high and mixture model is as follows:

[0127]

[0128]

[0129]

[0130] in, represents the complete sample, represents the sample points in the complete sample, represents the posterior probability distribution when the training batch sample is i, g represents the distribution component with the lower mean in the Gaussian mixture model, represents the root mean square error when the training batch sample is i, represents a sample without noise data, Represents a sample containing noisy data.

[0131] In addition, in order to help the model remember the knowledge of old tasks and avoid the influence of noise on model memory, adaptive plastic weight consolidation is used As the loss function, its calculation formula is as follows:

[0132]

[0133]

[0134]

[0135] Finally, based on the root mean square error of the predicted value and the true value of the 3D human skeleton posture, the plastic weight consolidation loss is introduced to construct the expression of the joint loss:

[0136]

[0137] in, represents the root mean square error generated by the simulation training, λ represents the preset hyperparameter, represents the plastic weight consolidation loss, j represents the number of 3D human posture skeleton points, i represents the i-th sample in the training batch, It represents the true value of the posture of the 3D human skeleton point k when the training batch sample is i. It represents the posture prediction value when the training batch sample is i and the 3D human skeleton point is k. Indicated by Sample of the moment The calculated adaptive Fisher information matrix, θ t,m express The parameters of the mth feature extraction module at time, express The feature extraction module parameters at the moment, represents the information matrix calculated when the training batch sample is i, denotes the gradient of the feature extraction module parameter θ t denotes a preset hyperparameter.

[0138] 8. inputting the test set into the trained target three-dimensional human skeleton pose estimation model to obtain a three-dimensional human skeleton pose.

[0139] In summary, the present application proposes a three-dimensional human skeleton pose estimation method and a model training method. Four types of modal data containing labels of true values of three-dimensional human skeleton poses are input into a feature extraction module of a three-dimensional human skeleton pose estimation model, and the extracted four types of features are input into an inter-modal feature fusion module for fusion and then input into the three-dimensional coordinate estimation module to output a predicted value of the three-dimensional human skeleton pose. In the training process of the three-dimensional human skeleton pose estimation model, in each iteration, a current three-dimensional human skeleton pose estimation model is used for rapid fitting, and a single-modal contribution value of the four types of modal data is calculated based on the Shannon value method, and the four types of modal data are divided into a superior modal class and an inferior modal class through a preset clustering algorithm. Single-modal data whose single-modal contribution value decreases to a set condition from a previous batch of samples is defined as noise modal data. The current batch of samples is used for simulation training, and the noise modal data belonging to the superior modal class and the noise modal data belonging to the inferior modal class are fitted respectively, and the root mean square error generated by the simulation training is used to establish two Gaussian distributions with different means based on the Gaussian mixture model, and the noise data corresponding to the Gaussian distribution with the higher mean is discarded. The three-dimensional human skeleton pose estimation model is trained using the current batch of samples after denoising, a joint loss is constructed based on the root mean square error of the predicted value and the true value of the three-dimensional human skeleton pose and the introduction of a plastic weight to consolidate the loss, and the three-dimensional human skeleton pose estimation model is updated in parameter by minimizing the joint loss.

[0140] Further, the feature extraction module parameters corresponding to the superior modal class at a preset iteration are fused with the initial feature extraction module parameters thereof according to a first weight, and the feature extraction module parameters corresponding to the inferior modal class are fused with the initial feature extraction module parameters thereof according to a second weight through an adaptive relearning method.

[0141] The embodiment of the present application also provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the edge computing server deployment method. The computer readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0142] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0143] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0144] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0145] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations to the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A 3D human skeleton posture estimation model training method, characterized in that: The method comprises the following steps: Obtaining a training sample set, wherein each sample in the training sample set includes four types of modal data generated for the sample object, including image data, lidar data, millimeter-wave radar data, and wireless channel data, and the sample includes a label of a true value of a three-dimensional human skeleton posture; The three-dimensional human skeleton pose estimation model is trained using the training sample set, and the three-dimensional human skeleton pose estimation model includes an image feature extraction module, a first point cloud feature extraction module, a second point cloud feature extraction module, a wireless channel feature extraction module, an inter-modal feature fusion module and a three-dimensional coordinate estimation module; the image feature extraction module is used to extract the image features of the image data, the first point cloud feature extraction module is used to extract the laser radar features of the laser radar data, the second point cloud feature extraction module is used to extract the millimeter wave radar features of the millimeter wave radar data, and the wireless channel feature extraction module is used to extract the wireless channel features of the wireless channel data; the inter-modal feature fusion module is used to fuse the image features, the laser radar features, the millimeter wave radar features and the wireless channel features, and input them into the three-dimensional coordinate estimation module to output a three-dimensional human skeleton pose prediction value, based on the root mean square error between the three-dimensional human skeleton pose prediction value and the true value, and introducing a plastic weight consolidation loss to construct a joint loss, minimize the joint loss to update the parameters of the three-dimensional human skeleton pose estimation model, and obtain a target three-dimensional human skeleton pose estimation model; Among them, in each round of iteration, the current three-dimensional human skeleton posture estimation model is used for fast fitting and the single modal contribution values ​​of the four types of modal data are calculated based on the Shapley value method. The four types of modalities are divided into superior modal classes and inferior modal classes according to the single modal contribution values ​​through a preset clustering algorithm; the single modal data whose single modal contribution values ​​of the current batch of samples decrease to the set conditions compared with the previous batch of samples are defined as noise modal data; the current batch of samples are used for simulation training, in which the fitting of the corresponding feature extraction module is suspended for the noise modal data belonging to the superior modal class, and the noise modal data belonging to the inferior class is suspended. The noise modal data of the isomodal class is fitted with the corresponding feature extraction module based on resampling enhancement, and two Gaussian distributions with different means are established for the root mean square error generated by the simulation training based on the Gaussian mixture model, and the noise data corresponding to the one with the higher mean is discarded; the three-dimensional human skeleton posture estimation model is trained using the denoised current batch samples; the feature extraction module parameters corresponding to the superior modal class are fused with the initial feature extraction module parameters according to the first weight, and the feature extraction module parameters corresponding to the inferior modal class are fused with the initial feature extraction module parameters according to the second weight.

2. The method for training a 3D human skeleton posture estimation model according to claim 1, wherein: The image feature extraction module adopts four convolutional networks and three dilated convolutional networks; the first point cloud feature extraction module adopts a first five-layer self-attention network, each layer of the first five-layer self-attention network is connected to a downsampling layer and a first average pooling layer is connected to the final output of the first five-layer self-attention network; the second point cloud feature extraction module adopts a second five-layer self-attention network, and the output of the second five-layer self-attention network is connected to a second average pooling layer; the wireless channel feature extraction module adopts three residual neural networks, a self-attention network and a two-dimensional average pooling layer fusion network.

3. The method for training a 3D human skeleton posture estimation model according to claim 1, wherein: The step of the inter-modal feature fusion module fusing the image feature, the lidar feature, the millimeter-wave radar feature, and the wireless channel feature includes: The image features, the lidar features, the millimeter-wave radar features, and the wireless channel features are uniformly mapped to the same feature space through the fully connected layer of the inter-modal feature fusion module and unified to a preset dimension through dimensional transformation; The image features, the lidar features, the millimeter-wave radar features, and the wireless channel features that are in the same feature space and have the same dimension are spliced ​​to obtain spliced ​​features, and the spliced ​​features are input into the four-layer self-attention layer of the inter-modal feature fusion module for feature fusion and average pooling is performed to output multimodal fusion features.

4. The three-dimensional human skeleton posture estimation model training method according to claim 1, characterized in that: The expression for calculating the single-mode m contribution value of the four types of modal data based on the Shapley value method is: V(S,m)=s(p 3D ,MM(S∪{m}))-s(p 3D ,MM(S)); in, Represents the set of modes m, S is expressed as subset of S, V(S,m) represents the additional benefit obtained by merging mode m into subset S, p 3D Represents the true value of the 3D human skeleton posture, represents the three-dimensional human skeleton posture prediction value, MM(·) represents the three-dimensional human skeleton posture estimation model, Indicates the Pearson correlation coefficient between the true value of the 3D human skeleton posture and the predicted value of the 3D human skeleton posture when the 3D human skeleton point is k. j represents the number of 3D human body posture skeleton points.

5. The three-dimensional human skeleton posture estimation model training method according to claim 1, characterized in that: The expression of the root mean square error generated by the simulation training is: Among them, j represents the number of 3D human body posture skeleton points, It represents the true value of the posture of the 3D human skeleton point k when the training batch sample is i. It represents the posture prediction value of the 3D human skeleton point k when the training batch sample is i.

6. The method for training a 3D human skeleton posture estimation model according to claim 1, wherein: The root mean square error generated by the simulation training is used to establish two Gaussian distributions with different means based on the Gaussian mixture model and discard the noise data corresponding to the one with the higher mean. The noise sample composed of the noise data corresponding to the one with the higher mean is A normal sample consisting of normal data corresponding to the one with a lower mean The expression is: in, represents the complete sample, represents the sample points in the complete sample, represents the posterior probability distribution when the training batch sample is i, g represents the distribution component with the lower mean in the Gaussian mixture model, represents the root mean square error when the training batch sample is i, represents a sample without noise data, Represents a sample containing noisy data.

7. The 3D human skeleton posture estimation model training method according to claim 1, characterized in that: Based on the root mean square error of the predicted value and the true value of the three-dimensional human skeleton posture, and introducing the plastic weight consolidation loss to construct the expression of the joint loss is: in, represents the root mean square error generated by the simulation training, λ represents the preset hyperparameter, represents the plastic weight consolidation loss, j represents the number of 3D human posture skeleton points, i represents the i-th sample in the training batch, It represents the true value of the posture of the 3D human skeleton point k when the training batch sample is i. It represents the posture prediction value when the training batch sample is i and the 3D human skeleton point is k. Indicated by Sample of the moment The calculated adaptive Fisher information matrix, θ t,m express The parameters of the mth feature extraction module at time, express The feature extraction module parameters at the moment, represents the information matrix calculated when the training batch sample is i, Represents the feature extraction module parameter θ t The gradient of , β represents the preset hyperparameter.

8. According to the 3D human skeleton posture estimation model training method of claim 1, the feature extraction module parameters corresponding to the superior modality class are fused with the initial feature extraction module parameters according to a first weight, and the feature extraction module parameters corresponding to the inferior modality class are fused with the initial feature extraction module parameters according to a second weight. The expression is: in, represents the first weight, α I represents the second weight, represents the initial feature extraction module parameters of mode m, represents the feature extraction module parameters of mode m when the preset iteration round is l, represents the superior modal class, Represents the secondary modal class.

9. A three-dimensional human skeleton posture estimation method, characterized in that: The method comprises the following steps: Obtain four types of modal data to be predicted; The four types of modal data to be predicted are input into the target three-dimensional human skeleton pose estimation model obtained by the three-dimensional human skeleton pose estimation model training method as described in any one of claims 1 to 8 to obtain the three-dimensional human skeleton pose.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images