Human skeleton sample feature diversification classification method, device and equipment based on diffusion model

Abstract

The application provides a human skeleton sample feature diversification classification method and device based on a diffusion model, and an equipment. The joint coordinate data of a human body is collected from an image device and input into a trained feature extraction network to extract joint features by using the feature extraction network, and the joint features are input into a trained classification network for classification to obtain a human behavior state. Since the trained feature extraction network and the classification network are both obtained by using a diffusion model for auxiliary training, and in the training process, the diffusion model randomly introduces noise into each input sample to obtain a plurality of corresponding noise feature samples, which diversifies the types of feature samples and increases the number of samples. Then, the plurality of noise feature samples are inversely denoised to obtain virtual feature samples, and the parameters of each layer of the feature extraction network and the classification network are adjusted according to the loss of the virtual feature samples. Therefore, the application can improve the accuracy of classifying the joint data to obtain the human behavior state.

Description

Technical Field

[0001] This invention belongs to the field of virtual skeleton sample generation technology, specifically relating to a classification method, apparatus, and device for human skeleton sample feature diversification based on a diffusion model. Background Technology

[0002] Behavior recognition is a crucial task in computer vision, aiming to infer human or animal behavior by analyzing image or video data. Skeleton behavior recognition is an important branch of behavior recognition, inferring human behavior by analyzing and understanding the movement patterns and postures of human joints.

[0003] Virtual feature generation is a method that utilizes computer algorithms and techniques to generate new sample data. It is widely used in machine learning and data science. Virtual feature generation aims to improve the performance, generalization ability, and data diversity of machine learning models by generating new sample data. It has significant application value in data science, machine learning, and artificial intelligence. Common virtual feature generation techniques include Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs). Virtual feature generation can assist in behavior recognition tasks, especially when data is limited or imbalanced. By generating virtual feature samples, the training dataset can be expanded, improving the model's generalization ability and robustness.

[0004] In recent years, with the development of diffusion model theory, diffusion models have been used as a data generation method to generate virtual feature samples. By adjusting the parameters of the diffusion model, the propagation and diffusion process of matter or information in space can be simulated, thereby generating new sample data with similar characteristics. This method has already found some applications in data generation tasks in certain fields.

[0005] In their paper (Spatial-temporal Transformer-guidedDiffusion based Data Augmentation for Efficient Skeleton-based ActionRecognition), Yifan Jiang et al. used a diffusion model to generate virtual feature samples to assist in the training of a feature extraction model. This approach employs a diffusion model based on a Transformer structure to generate complete virtual skeleton data samples.

[0006] Patent document (CN117422783A) discloses a training method, image sample generation method, and apparatus for an image sample generation model. This scheme uses multiple textual constraints to guide a diffusion model in generating forged images, including: acquiring at least one bank card image as the original image; determining the forged region and forged content in the original image based on the forgery task; determining positional constraint text based on the forgery region; determining content constraint text to describe the forged content; determining content guidance text to describe the forgery task; inputting the original image into a pre-trained diffusion model for image sample generation; and fine-tuning the diffusion model during backdiffusion using the positional constraint text, the content constraint text, and the content guidance text.

[0007] Patent document (CN117173607A) discloses a multi-level fusion multi-target tracking method, system, and computer-readable storage medium. This scheme includes sub-method one: extracting target re-identification features using a joint zero-order segmentation network; sub-method two: target relationship graph neural network processing; and sub-method three: multi-level long-range trajectory fusion. Sub-method one includes: pre-segmenting human targets using a transformer-based segmentation model (SAM); inputting the pre-segmented target image boxes into a pre-trained re-identification network to extract re-identification features of the human targets; sub-method two includes: given a continuous video segment and its corresponding set of detection boxes, constructing a target relationship graph model using a GNN; and sub-method three includes: training multiple target relationship graph models described in sub-method two in different order of levels, and linking video segments of different sizes using this target relationship graph model. This method improves the accuracy of feature recognition in multi-target tracking.

[0008] Patent document (CN117372524A) discloses a highly realistic ground-to-air image matching method and system based on conditional generation networks. This scheme uses a global alignment algorithm to preprocess the input image. The preprocessed image is then imported into a first conditional generation network to generate a cross-view image; this generator network is named G1. The synthesized image output by generator G1 is input into a second conditional generation network, and a segmentation map is synthesized in the same view; this generator network is denoted as G2 and named the segmenter based on its function. G1 and G2 are trained simultaneously. Discriminator networks D1 and D2 are constructed for G1 and G2 respectively. G1, G2, D1, and D2 networks are trained on a cross-view dataset platform. Based on the cross-view dataset platform, corresponding sub-datasets are constructed according to specific application scenarios and domains, and the network parameters are fine-tuned. For the input street view image, the network trained in the aforementioned steps is used for retrieval to achieve geolocation of the input image.

[0009] The existing diffusion model methods described above only add textual guidance during the reverse diffusion process. However, using only one textual guidance is insufficient to constrain the generation process, resulting in a significant semantic deviation between the generated samples and the real samples. Furthermore, using generative models such as GANs or VAEs to generate virtual feature samples to expand the dataset increases generation time and training costs exponentially by directly generating complete original training samples. Summary of the Invention

[0010] To address the aforementioned problems in the existing technology, this invention provides a classification method, apparatus, and device for human skeleton samples based on a diffusion model to address feature diversity. The technical problem to be solved by this invention is achieved through the following technical solution:

[0011] In a first aspect, the present invention provides a classification method for the diversity of human skeleton sample features based on a diffusion model, including:

[0012] S100, acquires the coordinate data of the joints of the human body from the imaging device;

[0013] S200, the joint coordinate data is input into the trained feature extraction network to extract joint features using the feature extraction network, and the joint features are input into the trained classification network for classification to obtain the human behavior state;

[0014] The trained feature extraction network and the trained classification network are both trained using a diffusion model. During the training process, the diffusion model randomly introduces noise into each input sample to obtain multiple corresponding noise feature samples, and then reversely denoises the multiple noise feature samples to obtain virtual feature samples. The parameters of each layer of the feature extraction network and the classification network are adjusted according to the loss of the virtual feature samples.

[0015] Secondly, the present invention provides a classification device for human skeleton samples with diverse features based on a diffusion model, comprising:

[0016] The acquisition module is configured to acquire the coordinate data of the human body's joints from the imaging device;

[0017] The classification module is configured to input the joint coordinate data into a trained feature extraction network to extract joint features using the feature extraction network, and then input the joint features into a trained classification network for classification to obtain the human behavior state.

[0018] The trained feature extraction network and the trained classification network are both trained using a diffusion model. During the training process, the diffusion model randomly introduces noise into each input sample to obtain multiple corresponding noise feature samples, and then reversely denoises the multiple noise feature samples to obtain virtual feature samples. The parameters of each layer of the feature extraction network and the classification network are adjusted according to the loss of the virtual feature samples.

[0019] In a second aspect, the present invention provides a device including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0020] Memory, used to store computer programs;

[0021] When a processor executes a program stored in memory, it implements the steps of the method described in the first aspect.

[0022] Beneficial effects:

[0023] This invention provides a method, apparatus, and device for classifying human skeleton samples based on a diffusion model, which diversifies the features of these samples. The method involves acquiring joint coordinate data of the human body from an image processing device and inputting it into a trained feature extraction network. The feature extraction network extracts joint features, which are then input into a trained classification network for classification to obtain the human behavioral state. Since both the feature extraction and classification networks are trained using a diffusion model, and during training, the diffusion model randomly introduces noise into each input sample to obtain multiple corresponding noisy feature samples, diversifying the types of feature samples and increasing the number of samples, the multiple noisy feature samples are then de-denoised to obtain virtual feature samples. The parameters of each layer of the feature extraction and classification networks are adjusted based on the loss of the virtual feature samples. Therefore, this invention can improve the accuracy of classifying human behavioral states from joint data.

[0024] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0025] Figure 1 A flowchart illustrating a classification method for human skeleton samples based on a diffusion model, provided in an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of the flow of joint point data provided in an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram illustrating the process of input data passing through a feature extraction network to a diffusion model, as provided in an embodiment of the present invention.

[0028] Figure 4 This is a schematic diagram of the diffusion model, feature extraction network, and classification network provided in the embodiments of the present invention;

[0029] Figure 5 This is a schematic diagram of the overall training process provided in an embodiment of the present invention;

[0030] Figure 6 This is a structural diagram of a diffusion-based classification device for human skeleton samples with diverse features, provided in an embodiment of the present invention. Detailed Implementation

[0031] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0032] Combination Figure 1 and Figure 2 This invention provides a classification method for human skeleton samples based on a diffusion model to address feature diversity, including:

[0033] S100, acquires the coordinate data of the joints of the human body from the imaging device;

[0034] This invention can collect human joint coordinate data from depth cameras or other methods such as video capture and pose estimation algorithms.

[0035] S200, the joint coordinate data is input into the trained feature extraction network to extract joint features using the feature extraction network, and the joint features are input into the trained classification network for classification to obtain the human behavior state;

[0036] The trained feature extraction network and the trained classification network are both trained using a diffusion model. During the training process, the diffusion model randomly introduces noise into each input sample to obtain multiple corresponding noise feature samples, and then reversely denoises the multiple noise feature samples to obtain virtual feature samples. The parameters of each layer of the feature extraction network and the classification network are adjusted according to the loss of the virtual feature samples.

[0037] This invention provides a method, apparatus, and device for classifying human skeleton samples based on a diffusion model, which diversifies the features of these samples. The method involves acquiring joint coordinate data of the human body from an image processing device and inputting it into a trained feature extraction network. The feature extraction network extracts joint features, which are then input into a trained classification network for classification to obtain the human behavioral state. Since both the feature extraction and classification networks are trained using a diffusion model, and during training, the diffusion model randomly introduces noise into each input sample to obtain multiple corresponding noisy feature samples, diversifying the types of feature samples and increasing the number of samples, the multiple noisy feature samples are then de-denoised to obtain virtual feature samples. The parameters of each layer of the feature extraction and classification networks are adjusted based on the loss of the virtual feature samples. Therefore, this invention can improve the accuracy of classifying human behavioral states from joint data.

[0038] Combination Figures 3-5 In a specific embodiment of the present invention, the step of using a diffusion model to assist in training to obtain a trained feature extraction network and a trained classification network includes:

[0039] S1, multiple sequence samples are formed by acquiring the three-dimensional coordinates of human joints from an image device, and each sequence sample is labeled with a category label to form a dataset; each element in each sequence sample is a topology graph; the topology graph consists of nodes and edges connecting the nodes, and each node represents the spatial coordinates of human joints;

[0040] This step uses a depth camera, or other methods such as video capture and pose estimation algorithms, to collect a dataset of 3D coordinates of human joints and label the dataset. The labeled dataset is then divided into training, validation, and test sets. The validation and test sets are used to validate and test the trained network.

[0041] It is worth noting that each sample in the dataset described in this invention is a sequence, and each element in the sequence is a topological graph. Specifically, the nodes in the graph represent the spatial coordinates of human joints, the edges in the graph are used to connect adjacent joints, and the topological structure of the graph is predefined.

[0042] S2, each sequence sample in the dataset is input into a pre-constructed feature extraction network to pre-train the feature extraction network and obtain a pre-trained feature extraction network;

[0043] The feature extraction network includes a hierarchical decomposition graph convolutional network, a spatiotemporal graph convolutional network, a channel topology refinement graph convolutional network, a deep learning network based on the Transformer structure, and a convolutional neural network.

[0044] It is worth noting that while this embodiment can use a hierarchical decomposition graph convolutional network as a feature extraction network, in other embodiments, other graph neural networks such as Spatio-Temporal Graph Convolutional Networks (ST-GCN) and Channel-wise Topology Refinement Graph Convolution Networks (CTR-GCN), or deep learning networks based on the Transformer structure, or other deep learning networks commonly used in the field based on convolutional neural networks can also be used as the feature extraction network.

[0045] In this embodiment, each sequence sample in the dataset is input into a pre-built feature extraction network to pre-train the feature extraction network. The cross-entropy loss function is used as the objective function to calculate the distance metric between each sequence sample and the optimization target after forward propagation. The parameters of the feature extraction network are updated using the batch gradient descent algorithm to obtain the pre-trained feature extraction network.

[0046] It is worth noting that the cross-entropy loss function and batch gradient descent algorithm are the tool functions and algorithms used in this embodiment. In other embodiments, other widely used loss functions and objective optimization algorithms can be used to replace the tool functions and algorithms in the training process.

[0047] S3 generates action descriptions and synonym descriptions for each category label;

[0048] This step generates action descriptions for each category label based on the category labels. Specifically, the action descriptions are complete and detailed statements describing the corresponding category action. This generation process can be implemented manually or using a large language model. This step also generates synonym descriptions for each category label based on the category labels. Specifically, the synonym descriptions consist of the action label and its synonyms. More specifically, 10 synonyms are generated for each category label text. This generation process can also be implemented manually or using a large language model.

[0049] S4, freeze the parameters of the pre-trained feature extraction network, and input all sequence samples into the feature extraction network with the frozen parameters in sequence to obtain the corresponding skeleton feature samples;

[0050] This step involves inputting sequence samples into a feature extraction network for forward propagation to obtain skeleton feature samples, which are then used as input to the diffusion model.

[0051] S5, the skeleton feature sample is input into the diffusion model so that the diffusion model obtains multiple noise feature samples by gradually introducing random noise and removes noise in reverse to obtain denoised feature samples that are semantically consistent with the input skeleton feature sample but not completely identical in appearance.

[0052] The diffusion model includes a noise introduction network and a noise prediction network. The noise introduction network is used to introduce random noise into the input sample to obtain a noisy feature sample. The noise prediction network is used to predict the random noise in the noisy feature sample and remove the random noise in reverse to obtain a denoising feature sample that is semantically consistent with the input sample but not completely identical in appearance. This sample is the virtual feature sample.

[0053] Combination Figure 3 and Figure 4 In a specific embodiment of the present invention, S5 includes:

[0054] S51, the skeleton feature sample is input into the noise introduction network, so that the noise introduction network starts from the input skeleton feature sample and gradually adds random noise to the skeleton feature sample to gradually form multiple noise feature samples corresponding to the skeleton feature sample.

[0055] S52, the plurality of noise feature samples and the action description are input into the noise prediction network so that the noise prediction network predicts the mean and variance of the noise contained in the plurality of noise feature samples, and uses the noise mean and variance to reverse the noise of the noise feature samples to obtain denoised feature samples that are semantically consistent with the input skeleton feature samples but not completely the same in form.

[0056] Combination Figure 4 and Figure 5 In a specific embodiment of the present invention, S52 includes:

[0057] S521, input both the action description and the synonym description into the text encoder to obtain the text features of the action description and the text features of the synonym description;

[0058] This embodiment first constructs a text encoder based on Transformer blocks. In this embodiment, a CLIP (Contrastive Language-Image Pre-training) pre-trained model is used as the text encoder. Specifically, the text encoder architecture in this embodiment is VIT-B / 32. It should be noted that in other embodiments, other text encoders commonly used in the art, such as BERT (Bidirectional Encoder Representation from Transformers), can also be used as the text encoder.

[0059] S522, the plurality of noise feature samples and the action description are input into the noise prediction network so that the noise prediction network predicts the mean and variance of the noise contained in the plurality of noise feature samples, and uses the noise mean and variance to reverse the noise feature samples. In the denoising process, the text features described by the synonyms are used to add the category attribute of the skeleton feature samples, so as to obtain denoised feature samples that are semantically consistent with the input skeleton feature samples but not completely the same in form.

[0060] It's worth noting that the diffusion model is based on Markov chains and generates skeleton data by progressively adding noise. This process aims to simulate the noise and uncertainties that may exist in real-world scenarios. Specifically, starting with the original skeleton feature sample x0, random (Gaussian) noise is added to the sample in each step to gradually form a noisy feature sample x. t Subsequently, in the reverse process, the goal of the diffusion model is to progressively denoise the samples x. t This approximates the denoised feature sample x0 in the target data distribution. Specifically, in this embodiment, the input to the diffusion model is the output of the feature extraction network, and the feature vector x0 has a length of 256.

[0061] Specifically, the above diffusion process can be represented by the following formula:

[0062]

[0063] Where t is the number of noise-adding steps, x t x represents the feature representation when the number of noise addition steps is t. t-1 Let β represent the feature representation when the number of noise addition steps is t-1. t Let t represent the noise variance when the number of noise addition steps is t, and I represent the standard normal distribution.

[0064] To obtain intermediate samples x more quickly t This invention defines Where β mLet x represent the noise variance when the number of noise addition steps is m, where m ranges from 1 to t. Then, x is obtained by sampling. t The sampling process can be represented by the following formula:

[0065]

[0066] The prediction process of a noise prediction network can be viewed as an inverse denoising process that introduces noise into the skeleton feature data. Its goal is to progressively denoise the noisy features x. t This process approximates the clean original sample x0, thereby generating feature samples that are semantically consistent with x0 but not entirely identical in appearance. The specifically generated denoised feature samples can be represented by the following formula:

[0067]

[0068] In practice, because x cannot be directly derived through mathematical reasoning... t Get x t-1 Therefore, a noise prediction network is introduced. The noise prediction network uses x... t Using t as input to predict x t-1 mean μ θ (x t ,t) and variance Specifically, the noise prediction network architecture used in this embodiment is a U-Net structure.

[0069] Wherein, mean μ θ (x t (t) can be expressed using the following formula:

[0070]

[0071] Where, ∈ θ (x t ,t) represents the mean noise predicted by the noise prediction network, while the variance It is usually predefined.

[0072] At the same time, by controlling the noise intensity added to the action description, the generation process of the diffusion model can be modulated to a certain extent, making it more adaptable to specific tasks and data characteristics.

[0073] In the reverse diffusion process of the diffusion model, the acquired action description information is added to the noise prediction network, and the noise prediction network f is trained. θ (x t E t E f This invention uses E. f E is the embedded feature representation of the action description obtained after the action description has been encoded by the text encoder. tThis represents the embedding feature representation of the noisy step number t obtained after passing through two fully connected layers.

[0074] S6. Using the synonym description, action description, and denoised feature samples, calculate the loss function of the diffusion model;

[0075] In a specific embodiment of the present invention, S6 includes:

[0076] S61, using the denoised feature samples and the skeleton feature samples, calculate the first loss of the noise prediction network; S62, using the synonym description, the denoised feature samples and the skeleton feature samples, calculate the second loss to optimize the skeleton feature samples; S63, using the first loss and the second loss, calculate the loss function of the diffusion model.

[0077] The diffusion model of this invention introduces an action description to optimize the noise prediction network. The loss function (first loss) during the optimization process can then be expressed by the following formula:

[0078]

[0079] For action recognition tasks, to ensure semantic consistency of the model, after diffusion, synonym descriptions are further used to enhance the category attributes of the skeleton action features. The skeleton feature representation and the embedded feature representation of the synonym description are compared in two directions within the same batch of data to jointly optimize the skeleton-text contrast loss. Specifically, this embodiment uses KL divergence as the skeleton-text contrast loss. The loss function (second loss) in this optimization process can be expressed by the following formula:

[0080]

[0081]

[0082] Where s and l represent the feature representations of the skeleton feature sample and the synonym description, respectively, and the subscript i represents the sample index, sim(l i ,s i ) represents the cosine similarity, τ is the temperature parameter, and B is the batch size of the skeleton feature samples. Representing the entire dataset, y s2l and y l2s y is the label value of the similarity score. If s and l correspond, then y... s2l The value of is 1, and KL represents the KL divergence function.

[0083] Therefore, the sampling process of the diffusion model uses the following function as the loss function:

[0084] L diff =L recon (Es E f )+λL con (E s E l (8)

[0085] Among them, E s E represents the skeleton action feature representation obtained from the GCN network. f E is the embedded feature representation obtained after the action description is encoded by a text encoder. l The embedded feature representation is obtained after the synonym description is encoded by the text encoder, and λ is a hyperparameter. Specifically, in this embodiment, the value of λ is 0.01.

[0086] S7. Adjust the parameters of each layer of the diffusion model according to the direction of decrease of the total loss function of the diffusion model to obtain the trained diffusion model; freeze the parameters of each layer of the trained diffusion model and unfreeze the parameters of each layer of the trained feature extraction network.

[0087] S8, input the sequence sample into the thawed feature extraction network to obtain skeleton feature samples; input the skeleton feature samples into the diffusion model with frozen parameters of each layer to generate multiple input skeleton feature samples that are semantically consistent but have different representations.

[0088] S9, input the virtual feature samples and the skeleton feature samples into the classification network and calculate the output loss function; update the parameters of each layer of the feature extraction network and the classification network according to the direction of descent of the output loss function to obtain the trained feature extraction network and the trained classification network.

[0089] This invention freezes the parameters of the diffusion model and unfreezes the parameters of the feature extraction network. Training set samples are input into the feature extraction network to obtain skeleton feature samples. These skeleton features are then input into the diffusion model to generate more skeleton feature samples. Specifically, in this embodiment, the number of generated samples is twice the batch size. The skeleton feature samples output by the feature extraction network and those generated by the diffusion model are input together into the classification network. The loss function is calculated and backpropagation is performed to update the parameters of both the feature extraction network and the classification network. Specifically, in this embodiment, the loss function used is the cross-entropy loss function, and the optimization algorithm is the batch gradient update algorithm.

[0090] To illustrate the technical effects of this invention, the experimental results of this embodiment on the existing public datasets NTU RGB+D and NTU RGB+D 120 are shown in the table below:

[0091]

[0092]

[0093] Secondly, the present invention provides a classification device for human skeleton samples with diverse features based on a diffusion model, comprising:

[0094] The acquisition module is configured to acquire the coordinate data of the joints of the human body from the imaging device;

[0095] The classification module is configured to input the joint coordinate data into a trained feature extraction network to extract joint features using the feature extraction network, and then input the joint features into a trained classification network for classification to obtain the human behavior state.

[0096] The trained feature extraction network and the trained classification network are both trained using a diffusion model. During the training process, the diffusion model randomly introduces noise into each input sample to obtain multiple corresponding noise feature samples, and then reversely denoises the multiple noise feature samples to obtain virtual feature samples. The parameters of each layer of the feature extraction network and the classification network are adjusted according to the loss of the virtual feature samples.

[0097] refer to Figure 6 The diffusion-based classification device for human skeleton samples with diverse features further includes an input device, a processor, a memory, and an output device. The input device, the processor, the memory, and the output device are connected via a bus.

[0098] The input devices include keyboards for text input, mice for user operation input, and image input devices such as cameras.

[0099] The output devices include display devices such as monitors and projectors for image display output, as well as other common output devices in the art such as printers.

[0100] Thirdly, the present invention also provides a device, characterized in that it includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0101] Memory, used to store computer programs;

[0102] When a processor executes a program stored in memory, it implements the steps of the method described in the first aspect.

[0103] In some embodiments, the processor may be a central processing unit (CPU), but it can also be other processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components. The processor typically serves as the control center of the electronic device, connecting different parts of the electronic device via data interfaces such as buses.

[0104] The memory primarily includes a program storage area and a data storage area. In some embodiments, the memory may be flash memory, a hard disk, a multimedia card, a card-type memory, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory, programmable read-only memory, magnetic storage, a hard disk, an optical disk, etc. The memory may also be an external storage device, such as a plug-in hard disk, a secure data card, a flash memory card, etc. The memory is used to store computer programs, which are executed by a processor to perform the functions described in this invention.

[0105] The terms "first" and "second" in this invention are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0106] Although this application has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the appended claims in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality.

[0107] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A classification method for human skeleton samples based on diffusion model feature diversity, characterized in that, include: S100, acquires the coordinate data of the joints of the human body from the imaging device; S200, the joint coordinate data is input into the trained feature extraction network to extract joint features using the feature extraction network, and the joint features are input into the trained classification network for classification to obtain the human behavior state; The trained feature extraction network and the trained classification network are both trained using a diffusion model. During the training process, the diffusion model randomly introduces noise into each input sample to obtain multiple corresponding noise feature samples, and reversely denoises the multiple noise feature samples to obtain virtual feature samples. The parameters of each layer of the feature extraction network and the classification network are adjusted according to the loss of the virtual feature samples. The steps for obtaining a trained feature extraction network and a trained classification network using a diffusion model-assisted training method include: S1, multiple sequence samples are formed by acquiring the three-dimensional coordinates of human joints from an image device, and each sequence sample is labeled with a category label to form a dataset; each element in each sequence sample is a topology graph; the topology graph consists of nodes and edges connecting the nodes, and each node represents the spatial coordinates of human joints; S2, each sequence sample in the dataset is input into a pre-constructed feature extraction network to pre-train the feature extraction network and obtain a pre-trained feature extraction network; S3 generates action descriptions and synonym descriptions for each category label; S4, freeze the parameters of the pre-trained feature extraction network, and input all sequence samples into the feature extraction network with the frozen parameters in sequence to obtain the corresponding skeleton feature samples; S5, the skeleton feature sample is input into the diffusion model so that the diffusion model obtains multiple noise feature samples by gradually introducing random noise and removes noise in reverse to obtain denoised feature samples that are semantically consistent with the input skeleton feature sample but not completely identical in appearance. S6. Using the synonym description, action description, and denoised feature samples, calculate the loss function of the diffusion model; S7. Adjust the parameters of each layer of the diffusion model according to the direction of decrease of the total loss function of the diffusion model to obtain the trained diffusion model; freeze the parameters of each layer of the trained diffusion model and unfreeze the parameters of each layer of the trained feature extraction network. S8, input the sequence sample into the thawed feature extraction network to obtain skeleton feature samples; input the skeleton feature samples into the diffusion model with frozen parameters of each layer to generate multiple input skeleton feature samples that are semantically consistent but have different representations. S9, input the virtual feature samples and the skeleton feature samples into the classification network and calculate the output loss function; update the parameters of each layer of the feature extraction network and the classification network according to the direction of descent of the output loss function to obtain the trained feature extraction network and the trained classification network.

2. The classification method for human skeleton samples based on diffusion model feature diversity according to claim 1, characterized in that, S2 include: Each sequence sample in the dataset is input into a pre-built feature extraction network to pre-train the feature extraction network. The cross-entropy loss function is used as the objective function to calculate the distance metric between each sequence sample and the optimization target after forward propagation. The parameters of the feature extraction network are updated using the batch gradient descent algorithm to obtain the pre-trained feature extraction network.

3. The classification method for human skeleton samples based on diffusion model feature diversity according to claim 1, characterized in that, The diffusion model includes a noise introduction network and a noise prediction network; The noise introduction network is used to introduce random noise into the input sample to obtain a noisy feature sample. The noise prediction network is used to predict the random noise in the noisy feature sample and remove the random noise in reverse to obtain a virtual feature sample that is semantically consistent with the input sample but not exactly the same in form.

4. The classification method for human skeleton samples based on diffusion model feature diversity according to claim 3, characterized in that, S5 include: S51, the skeleton feature sample is input into the noise introduction network, so that the noise introduction network starts from the input skeleton feature sample and gradually adds random noise to the skeleton feature sample to gradually form multiple noise feature samples corresponding to the skeleton feature sample. S52, the plurality of noise feature samples and the action description are input into the noise prediction network so that the noise prediction network predicts the mean and variance of the noise contained in the plurality of noise feature samples, and uses the noise mean and variance to reverse the noise of the noise feature samples to obtain denoised feature samples that are semantically consistent with the input skeleton feature samples but not completely the same in form.

5. The classification method for human skeleton samples based on diffusion model feature diversity according to claim 4, characterized in that, S52 includes: S521, input both the action description and the synonym description into the text encoder to obtain the text features of the action description and the text features of the synonym description; S522, the plurality of noise feature samples and the action description are input into the noise prediction network so that the noise prediction network predicts the mean and variance of the noise contained in the plurality of noise feature samples, and uses the noise mean and variance to reverse the noise feature samples. In the denoising process, the text features described by the synonyms are used to add the category attribute of the skeleton feature samples, so as to obtain denoised feature samples that are semantically consistent with the input skeleton feature samples but not completely the same in form.

6. The classification method for human skeleton samples based on diffusion model feature diversity according to claim 5, characterized in that, S6 include: S61, using the denoised feature samples and the skeleton feature samples, calculate the first loss of the noise prediction network; S62, using the synonym description, the denoised feature samples and the skeleton feature samples, calculate the second loss to optimize the skeleton feature samples; S63, calculate the loss function of the diffusion model using the first loss and the second loss.

7. The classification method for human skeleton samples based on diffusion model feature diversity according to claim 1, characterized in that, The feature extraction network includes a hierarchical decomposition graph convolutional network, a spatiotemporal graph convolutional network, a channel topology refinement graph convolutional network, a deep learning network based on the Transformer structure, and a convolutional neural network.

8. A classification device for human skeleton samples with diverse features based on a diffusion model, characterized in that, include: The acquisition module is configured to acquire the coordinate data of the joints of the human body from the imaging device; The classification module is configured to input the joint coordinate data into a trained feature extraction network to extract joint features using the feature extraction network, and then input the joint features into a trained classification network for classification to obtain the human behavior state. The trained feature extraction network and the trained classification network are both trained using a diffusion model. During training, the diffusion model randomly introduces noise into each input sample to obtain multiple corresponding noisy feature samples, and then reverse-denoises these multiple noisy feature samples to obtain virtual feature samples. The parameters of each layer of the feature extraction network and the classification network are adjusted based on the loss of the virtual feature samples. The steps of training the trained feature extraction network and the trained classification network using a diffusion model include: S1, multiple sequence samples are formed by acquiring the three-dimensional coordinates of human joints from an image device, and each sequence sample is labeled with a category label to form a dataset; each element in each sequence sample is a topology graph; the topology graph consists of nodes and edges connecting the nodes, and each node represents the spatial coordinates of human joints; S2, each sequence sample in the dataset is input into a pre-constructed feature extraction network to pre-train the feature extraction network and obtain a pre-trained feature extraction network; S3 generates action descriptions and synonym descriptions for each category label; S4, freeze the parameters of the pre-trained feature extraction network, and input all sequence samples into the feature extraction network with the frozen parameters in sequence to obtain the corresponding skeleton feature samples; S5, the skeleton feature sample is input into the diffusion model so that the diffusion model obtains multiple noise feature samples by gradually introducing random noise and removes noise in reverse to obtain denoised feature samples that are semantically consistent with the input skeleton feature sample but not completely identical in appearance. S6. Using the synonym description, action description, and denoised feature samples, calculate the loss function of the diffusion model; S7. Adjust the parameters of each layer of the diffusion model according to the direction of decrease of the total loss function of the diffusion model to obtain the trained diffusion model; freeze the parameters of each layer of the trained diffusion model and unfreeze the parameters of each layer of the trained feature extraction network. S8, input the sequence sample into the thawed feature extraction network to obtain skeleton feature samples; input the skeleton feature samples into the diffusion model with frozen parameters of each layer to generate multiple input skeleton feature samples that are semantically consistent but have different representations. S9, input the virtual feature samples and the skeleton feature samples into the classification network and calculate the output loss function; update the parameters of each layer of the feature extraction network and the classification network according to the direction of descent of the output loss function to obtain the trained feature extraction network and the trained classification network.

9. A device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-7.

Images