Unsupervised skeleton behavior recognition method and system based on multi-modal key semantics

Abstract

The invention provides an unsupervised skeleton behavior recognition method and system based on multi-modal key semanteme, and the method comprises the steps: collecting a pre-trained multi-modal skeleton video sequence, carrying out the preprocessing, and constructing a skeleton data set; performing time sequence understanding on each type of action images in the skeleton data set by using a multi-modal large language model, generating key description, and generating a text description feature cluster through a semantic encoder; capturing a skeleton action feature cluster of the picture in the skeleton data set through a hybrid enhancement strategy and a space-time diagram convolution ST-GCN model; and mapping the text description feature cluster and the skeleton action feature cluster to the same representation space by using a GLIP semantic alignment strategy, carrying out similarity comparison on different actions, and training a skeleton encoder by using back propagation to realize skeleton behavior recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of human motion behavior analysis, and in particular, to an unsupervised skeleton behavior recognition method and system based on multimodal key semantics. Background Art

[0002] As a key topic in the field of computer vision, the importance of behavior recognition is self-evident. The recognition and understanding of human actions is not only related to human visual perception and cognitive ability, but also has wide applications in video retrieval, human-computer interaction, sign language recognition and other fields. Especially in high-precision scenarios such as autonomous driving, the accuracy and robustness of behavior recognition technology are particularly important. In recent years, with the rapid development of depth sensors and posture estimation algorithms, three-dimensional skeleton data can be obtained more conveniently. Skeleton data, with its compactness, resistance to environmental interference, and accurate reflection of human joint position and motion information, significantly improves the performance of action recognition in noisy scenes.

[0003] Although existing unsupervised contrastive learning skeleton action recognition methods have made some progress, they still face many challenges. On the one hand, existing skeleton data augmentation strategies fail to effectively provide rich difficult contrast pairs to learn more discriminative spatiotemporal skeleton motion representations. On the other hand, the lack of contextual information and interacting objects in skeleton data leads to small differences between similar action classes, making the direct skeleton spatiotemporal feature modeling method have limited performance when dealing with ambiguous behaviors. In addition, similar actions within a fuzzy action group are difficult to distinguish, which limits the classification performance of the skeleton action recognition model. Therefore, how to effectively capture the most discriminative and representative motion features and semantic features between similar action classes is still a problem that needs to be solved in this field.

[0004] Patent CN116343338A proposes a human skeleton action recognition method based on a hierarchical spatiotemporal attention network, which extracts multi-level spatiotemporal features by dividing body parts and time segments. However, when processing skeleton data, this method is affected by sensor accuracy and posture prediction algorithms, and there are problems with noise and data confusion. Another patent, CN114170683A, proposes a behavior recognition method based on a multimodal attention fusion network, which extracts RGB and skeleton features through ResNeXt101 and Shift-GCN networks, and performs modal fusion. Although this method combines RGB and skeleton data to improve the recognition effect, it requires a large number of annotation labels, and RGB data occupies a large storage space and is easily affected by the environment. Although the two methods have their own advantages, they both face challenges such as data confusion and environmental influences, and need to be further optimized to improve recognition accuracy. Summary of the invention

[0005] According to the technical problems raised above, an unsupervised skeleton behavior recognition method and system based on multimodal key semantics are provided. The present invention mainly utilizes the unsupervised skeleton behavior recognition method based on multimodal key semantics, combines multimodal and semantic alignment training to develop an unsupervised contrastive learning framework, obtains semantic cues through action keyframes, uses a large-scale multimodal pre-trained language model (GPT-4-O) as a perception engine, generates a set of key dynamic descriptions with temporal significance for each type of action, uses CLIP pre-training technology to fuse additional action semantic information for mixed enhanced positive and negative sample pairs, and assists the model in learning more discriminative action representations for unsupervised multimodal skeleton action recognition, which has the advantages of accurate semantics, strong reproducibility, and good robustness. .

[0006] The technical means adopted by the present invention are as follows:

[0007] An unsupervised skeleton behavior recognition method based on multimodal key semantics, comprising:

[0008] S1, collect pre-trained multimodal skeleton video sequences, perform preprocessing, and construct a skeleton dataset;

[0009] S2. Use a multimodal large language model to understand the temporal sequence of each type of action image in the skeleton dataset and generate key descriptions, and generate text description feature clusters through a semantic encoder;

[0010] S3, capturing skeleton action feature clusters of images in the skeleton dataset through a hybrid enhancement strategy and spatiotemporal graph convolution ST-GCN model;

[0011] S4. Use the GLIP semantic alignment strategy to map the text description feature cluster and the skeleton action feature cluster to the same representation space, compare the similarities of different actions, and use back-propagation to train the skeleton encoder to achieve skeleton behavior recognition.

[0012] Furthermore, the step S1 specifically includes:

[0013] Preprocess the video sequences containing human skeleton movements, extract the first frame, middle frame and final frame of each type of movement original video as skeleton data, and construct a skeleton dataset;

[0014] The preprocessing includes: extracting original skeleton data in the video sequence, eliminating noise and normalizing the viewing angle.

[0015] Furthermore, the step S2 specifically includes:

[0016] The multimodal large language model describes the specific behaviors of the skeleton in the action image in order, and organizes the description of the specific behaviors of the skeleton into description sentences;

[0017] The specific behaviors include: objects in hand, hand posture, hand dynamics, torso dynamics, leg dynamics, and foot dynamics;

[0018] The description sentence is input into the semantic encoder E Text (·) generates text description feature clusters:

[0019] E i =E Text (d i )

[0020] Cluster(D)=f(BN(E1,E2,...,E m ))

[0021] Among them, d i represents the key description of the i-th action generated by the multimodal large language model, E i What does it mean? BN(·) means batch data normalization, f(·) means feature mapping, Cluster(·) means feature aggregation, and D means the generated text description feature cluster.

[0022] Furthermore, the hybrid enhancement strategy specifically includes:

[0023] Based on the topological information of the skeleton data, the skeleton joints are divided into five parts, and parts are randomly selected for cropping to form a cropped view; the cropped view is randomly combined with the truncated view composed of the remaining skeleton sequence to generate a mixed skeleton sequence;

[0024] Extract features from the mixed skeleton sequence and separate the features of the cropped view and the truncated view through spatiotemporal mask pooling. The formula of spatiotemporal mask pooling is as follows:

[0025]

[0026] Among them, p represents the crop view, g represents the truncated view, M r represents the mask matrix used to indicate the cropped skeleton fragments, represents the remaining skeleton sequence, I represents the identity matrix, represents the skeleton encoded vector, T′ represents the time dimension after time downsampling, and V represents the number of nodes in the skeleton data.

[0027] Furthermore, the spatiotemporal graph convolution ST-GCN model specifically includes:

[0028] The spatiotemporal graph convolutional network ST-GCN represents the dynamic information of the human skeleton as a spatiotemporal graph G = (V, E), where V = (v1, v2, ..., v N) represents a joint set consisting of N vertices, and E represents a set of edges connecting vertices; in the spatiotemporal graph convolutional network ST-GCN, spatial convolution and temporal convolution capture the spatial features and temporal features of skeleton data respectively;

[0029] The spatial convolution is used to capture the spatial relationship between different joints in the same time frame; the spatial convolution is expressed as:

[0030]

[0031] Among them, f out (v i ) represents node v i The output features, f in (v j ) represents node v j Input features of node v i The set of neighbor nodes, w ij Represents node v i and its neighbor node v j The weight between

[0032] The temporal convolution is used to capture the dynamic changes of the same joint between different time frames; the temporal convolution is expressed as:

[0033]

[0034] Among them, f out (v t ) represents the output features of time t and node v, f in (v τ ) represents the input features of time τ and node v, w(τ) represents the time weight function, and Δ represents the time window size;

[0035] The spatiotemporal graph convolutional network ST-GCN combines spatial convolution and temporal convolution and works alternately to capture spatiotemporal features:

[0036]

[0037] Among them, f out (v ti ) represents time t and node v i The output features, f in (y(v ti ,v tj )) is the neighbor node v sampled by the sampling function y tj Input features, B(v ti ) represents node v ti The neighbor combination of Z ti (v tj) represents the normalization term, w(v ti ,v tj ) represents the spatiotemporal weight function.

[0038] Furthermore, the cropped view p and the truncated view g are feature mapped through the spatiotemporal graph convolutional network ST-GCN:

[0039] v' i =(p' i ,g' i )=ST-GCN(p i ,g i )

[0040] Cluster(V)=f(P,G)=f(v'1,v'2,...,v' m )

[0041] Among them, v' i represents the mixed enhancement set of the i-th action, p' i represents the cropped view feature after the i-th action encoding, g' i represents the truncated view feature after the i-th action encoding, p i represents the cropped view feature of the i-th action, g i Represents the truncated view feature of the i-th action, v' m represents the hybrid enhancement set of the last action. P represents the set of all action cropped views, G represents the set of all action truncated views, f(·) represents feature mapping, Cluster(·) represents feature aggregation, and V is the generated skeleton action feature cluster.

[0042] Furthermore, the step S4 specifically includes:

[0043] Through the CLIP semantic alignment strategy, all action clipping view sets P and all action truncated view sets G are embedded with the text description feature cluster D for auxiliary fusion, and the association relationship learning method is expressed as:

[0044]

[0045] Among them, P' represents the multimodal feature vector of the feature vector after action cropping view data enhancement and the fusion of text feature vector, G' represents the multimodal feature vector of the feature vector after action truncation view data enhancement and the fusion of text feature vector, Similarity is used to calculate the cosine similarity of the normalized input features, Represents matrix multiplication;

[0046] The multimodal feature vector P' is embedded with the corresponding key in the memory queue Q Form a positive sample pair, the multimodal feature vector G' is embedded with the corresponding key in the memory queue Q Form a negative sample pair, and convert the loss L of the multimodal feature vectors P' and G' P' and L G' Respectively expressed as:

[0047]

[0048] in, Used to represent Z(P') and memory queues The similarity between

[0049] The loss L P' and L G' The average value of is taken as the overall contrast loss, and the calculation method of the overall contrast loss L is as follows:

[0050] L=L info +λ(L P' +L G' )

[0051] Among them, L info represents the general InfoNCE loss, λ is a hyperparameter used to balance the scale of easy contrast and difficult contrast;

[0052] The parameters of the skeleton encoder are updated according to the overall contrast loss L through a gradient descent algorithm, and iterative training is performed until a set number of training times is reached, and the trained skeleton encoder is used to implement action classification.

[0053] Corresponding to an unsupervised skeleton behavior recognition method based on multimodal key semantics, the present application also provides an unsupervised skeleton behavior recognition system based on multimodal key semantics, including: an image acquisition unit, an action semantics generation unit, a skeleton action feature extraction unit and a multimodal feature fusion unit, wherein:

[0054] The image acquisition unit is used to acquire a pre-trained multi-modal skeleton video sequence, perform pre-processing, and construct a skeleton data set;

[0055] The action semantic generation unit is used to use a multimodal large language model to perform temporal understanding on each type of action image in the skeleton data set, generate key descriptions, and generate text description feature clusters through a semantic encoder;

[0056] The skeleton action feature extraction unit is used to capture the skeleton action feature clusters of the pictures in the skeleton dataset through a hybrid enhancement strategy and a spatiotemporal graph convolution ST-GCN model;

[0057] The multimodal feature fusion unit is used to map the text description feature cluster and the skeleton action feature cluster to the same representation space using the GLIP semantic alignment strategy, compare the similarities of different actions, and use back-propagation to train the skeleton encoder to achieve skeleton behavior recognition.

[0058] The present invention also includes a storage medium, which includes a stored program, wherein when the program is run, the unsupervised skeleton behavior recognition method based on multimodal key semantics is executed.

[0059] The present invention also includes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the unsupervised skeletal behavior recognition method based on multimodal key semantics through the computer program.

[0060] Compared with the prior art, the present invention has the following advantages:

[0061] The unsupervised skeleton behavior recognition method and system based on multimodal key semantics provided by the present invention use a concise action prompt method for multiple parts of the human body to perform image-level semantic work on the action when acquiring key action semantics. It generates key dynamic descriptions through GPT-4-O, which greatly reduces the probability of acquiring redundant semantics and improves the efficiency of the model in processing effective discriminative semantics.

[0062] The unsupervised skeleton behavior recognition method and system based on multimodal key semantics provided by the present invention generates difficult contrast samples by introducing a skeleton hybrid enhancement strategy. The difficult positive sample pairs (such as truncated views and masked views) and difficult negative sample pairs (such as trimmed views and truncated views) provided by the hybrid skeleton sequence are used to learn better global and local spatiotemporal action representations, and the spatiotemporal graph convolutional network is used to replace the traditional 2D convolutional network, thereby improving the feature extraction capability of the encoder.

[0063] The unsupervised skeleton action recognition method and system based on multimodal key semantics provided by the present invention develops a new unsupervised multimodal contrastive learning framework for skeleton action recognition. Combining CLIP multimodal semantic alignment pre-training technology, a new multimodal fusion contrastive learning framework is developed, which uses the key description of the action to add additional textual semantic information to the mixed enhanced positive and negative sample pairs, enhancing the model's ability to distinguish confusing actions from a semantic level.

[0064] The present invention adopts an unsupervised learning method, which does not require labeled data at all, reducing the cost of data collection and labeling. At the same time, by comparing learning strategies, it can effectively utilize unlabeled data to generate a large number of training samples, thereby improving data utilization efficiency. Compared with the prior art, the present invention can better adapt to scenarios where data is scarce and reduce dependence on data quality and quantity.

[0065] By integrating multimodal key semantics, hybrid data enhancement, spatiotemporal graph convolutional networks, and multimodal semantic alignment contrast learning, the present invention can understand and recognize human actions from different levels, especially when dealing with complex and subtle actions, showing higher accuracy and efficiency. This method is universal in theory and shows significant performance advantages in practical applications.

[0066] The present invention can learn more discriminative action representations through the strategy of key semantic generation and multimodal feature comparison and fusion, thereby improving the model's recognition accuracy for easily confused actions. At the same time, the present invention adopts an unsupervised learning method, which can better adapt to unseen data and complex scenarios, thereby improving the generalization ability of the model. Compared with the prior art, the present invention can be applied in a wider range of scenarios and achieve better recognition effects.

[0067] The present invention adopts a multimodal mechanism and shows good universality in different data sets, without the need for personalized adjustments for each data set. By focusing on key action semantics, the present invention can more effectively identify human body movements and automatically adapt to the feature differences of different data sets. Compared with traditional unsupervised skeletal behavior recognition methods, the present invention pays more attention to learning the key semantic features of actions from data, and solves the problem of distinguishing similar actions through a semantically guided multimodal fusion module, thereby improving the ability to recognize actions in real-world scenes, achieving high-precision skeletal action classification results, and providing efficient and reliable solutions for applications in multiple fields.

[0068] The unsupervised skeleton action recognition method and system based on multimodal key semantics provided by the present invention has been proven to be effective through a series of detailed ablation experiments, thereby improving the practicality and reproducibility of the present invention. After comparative experiments with current methods based on graph convolutional network (GCN) technology and Transformer architecture, the present invention demonstrated its superior performance on three independent data sets, indicating that the proposed multimodal unsupervised model achieves better results in action classification tasks. The present invention also proposes an innovative skeleton action recognition technology, which significantly improves the accuracy and robustness of action classification through the strategy of key semantic generation and multimodal feature comparison and fusion.

[0069] Based on the above reasons, the present invention can be widely promoted in the technical field of human motion behavior analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0071] Figure 1 This is a flow chart of the unsupervised skeletal behavior recognition method based on multimodal key semantics in the present invention.

[0072] Figure 2 Generate a flow chart for the key description of actions in the present invention.

[0073] Figure 3 This is the unsupervised contrastive learning model diagram in the present invention. DETAILED DESCRIPTION

[0074] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0075] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0076] like Figure 1 As shown, the present invention provides an unsupervised skeleton behavior recognition method based on multimodal key semantics, comprising:

[0077] S1, collect pre-trained multimodal skeleton video sequences, perform preprocessing, and construct a skeleton dataset;

[0078] In specific implementation, as a preferred embodiment of the present invention, the step S1 specifically includes:

[0079] Preprocess the video sequences containing human skeleton movements, extract the first frame, middle frame and final frame of each type of movement original video as skeleton data, and construct a skeleton dataset;

[0080] The preprocessing includes: extracting original skeleton data in the video sequence, eliminating noise and normalizing the viewing angle.

[0081] S2. Use a multimodal large language model to understand the temporal sequence of each type of action image in the skeleton dataset and generate key descriptions, and generate text description feature clusters through a semantic encoder;

[0082] In specific implementation, as a preferred embodiment of the present invention, step S2 specifically includes:

[0083] The multimodal large language model describes the specific behaviors of the skeleton in the action image in order, and organizes the description of the specific behaviors of the skeleton into description sentences;

[0084] The specific behaviors include: items in hand (if there is no item, generate None), hand gestures (victory / OK / thumbs up / pointing forward / none), hand dynamics (mild / moderate / intense / none), trunk dynamics (mild / moderate / intense / none), leg dynamics (mild / moderate / intense / none), and foot dynamics (mild / moderate / intense / none). Then, organize this group of action text descriptions into a complete description sentence, for example, the action "take off the hat": there is a hat in the hand, no hand gestures, intense hand dynamics, slight trunk dynamics, no leg dynamics, and no foot dynamics.

[0085] The description sentence is input into the semantic encoder E Text (·) generates text description feature clusters:

[0086] E i =E Text (d i )

[0087] Cluster(D)=f(BN(E1,E2,...,E m ))

[0088] Among them, d i represents the key description of the i-th action generated by the multimodal large language model, E i What does it mean? BN(·) means batch data normalization, f(·) means feature mapping, Cluster(·) means feature aggregation, and D means the generated text description feature cluster.

[0089] S3, capturing skeleton action feature clusters of images in the skeleton dataset through a hybrid enhancement strategy and spatiotemporal graph convolution ST-GCN model;

[0090] In specific implementation, as a preferred embodiment of the present invention, the hybrid enhancement strategy specifically includes:

[0091] Based on the topological information of the skeleton data, the skeleton joints are divided into five parts, and parts are randomly selected for cropping to form a cropped view; the cropped view is randomly combined with the truncated view composed of the remaining skeleton sequence to generate a mixed skeleton sequence;

[0092] Extract features from the mixed skeleton sequence and separate the features of the cropped view and the truncated view through spatiotemporal mask pooling. The formula of spatiotemporal mask pooling is as follows:

[0093]

[0094] Among them, p represents the crop view, g represents the truncated view, M r represents the mask matrix used to indicate the cropped skeleton fragments, represents the remaining skeleton sequence, I represents the identity matrix, represents the skeleton encoded vector, T′ represents the time dimension after time downsampling, and V represents the number of nodes in the skeleton data.

[0095] In specific implementation, as a preferred embodiment of the present invention, the spatiotemporal graph convolution ST-GCN model specifically includes:

[0096] The spatiotemporal graph convolutional network ST-GCN represents the dynamic information of the human skeleton as a spatiotemporal graph G = (V, E), where V = (v1, v2, ..., v N ) represents a joint set consisting of N vertices, and E represents a set of edges connecting vertices; in the spatiotemporal graph convolutional network ST-GCN, spatial convolution and temporal convolution capture the spatial features and temporal features of skeleton data respectively;

[0097] The spatial convolution is used to capture the spatial relationship between different joints in the same time frame; given a graph, where nodes represent joints and edges represent natural connections between joints, the spatial convolution is represented as:

[0098]

[0099] Among them, f out (v i ) represents node v i The output features, f in (v j ) represents node v jInput features of node v i The set of neighbor nodes, w ij Represents node v i and its neighbor node v j The weight between

[0100] The temporal convolution is used to capture the dynamic changes of the same joint between different time frames; the temporal convolution is expressed as:

[0101]

[0102] Among them, f out (v t ) represents the output features of time t and node v, f in (v τ ) represents the input features of time τ and node v, w(τ) represents the time weight function, and Δ represents the time window size;

[0103] The spatiotemporal graph convolutional network ST-GCN combines spatial convolution and temporal convolution and works alternately to capture spatiotemporal features:

[0104]

[0105] Among them, f out (v ti ) represents time t and node v i The output features, f in (y(v ti ,v tj )) is the neighbor node v sampled by the sampling function y tj Input features, B(v ti ) represents node v ti The neighbor combination of Z ti (v tj ) represents the normalization term, w(v ti ,v tj ) represents the spatiotemporal weight function.

[0106] In specific implementation, as a preferred embodiment of the present invention, the spatiotemporal graph convolutional network ST-GCN can simultaneously capture the spatial structure and temporal dynamics of the skeleton data, thereby more effectively mapping the skeleton features. The cropped view p and the truncated view g are feature mapped through the spatiotemporal graph convolutional network ST-GCN:

[0107] v' i =(p' i ,g' i )=ST-GCN(p i ,g i )

[0108] Cluster(V)=f(P,G)=f(v'1,v'2,...,v' m )

[0109] Among them, v' i represents the mixed enhancement set of the i-th action, p' i represents the cropped view feature after the i-th action encoding, g' i represents the truncated view feature after the i-th action encoding, p i represents the cropped view feature of the i-th action, g i Represents the truncated view feature of the i-th action, v' m represents the hybrid enhancement set of the last action. P represents the set of all action cropped views, G represents the set of all action truncated views, f(·) represents feature mapping, Cluster(·) represents feature aggregation, and V is the generated skeleton action feature cluster.

[0110] S4. Use the GLIP semantic alignment strategy to map the text description feature cluster and the skeleton action feature cluster to the same representation space, compare the similarities of different actions, and use back-propagation to train the skeleton encoder to achieve skeleton behavior recognition.

[0111] In specific implementation, as a preferred embodiment of the present invention, step S4 specifically includes:

[0112] Through the CLIP semantic alignment strategy, all action cropped view sets P and all action truncated view sets G are embedded with the text description feature cluster D for auxiliary fusion, and the loss function is set to distinguish action samples. The association relationship learning method is expressed as:

[0113]

[0114] Among them, P' represents the multimodal feature vector of the feature vector after action cropping view data enhancement and the fusion of text feature vector, G' represents the multimodal feature vector of the feature vector after action truncation view data enhancement and the fusion of text feature vector, Similarity is used to calculate the cosine similarity of the normalized input features, Represents matrix multiplication;

[0115] The multimodal feature vector P' is embedded with the corresponding key in the memory queue Q Form a positive sample pair, the multimodal feature vector G' is embedded with the corresponding key in the memory queue Q Form a negative sample pair, and convert the loss L of the multimodal feature vectors P' and G' P' and L G' Respectively expressed as:

[0116]

[0117] in, Used to represent Z(P') and memory queues The similarity between

[0118] The loss L P' and L G' The average value of is taken as the overall contrast loss, and the calculation method of the overall contrast loss L is as follows:

[0119] L=L info +λ(L P' +L G' )

[0120] Among them, L info represents the general InfoNCE loss, λ is a hyperparameter used to balance the scale of easy contrast and difficult contrast;

[0121] By using the gradient descent algorithm, the parameters of the skeleton encoder are updated according to the overall contrast loss L, and iterative training is performed until the set number of training times is reached, and the trained skeleton encoder is used to implement action classification. The encoder is retained as a feature extractor for subsequent downstream tasks to implement action classification.

[0122] Corresponding to an unsupervised skeleton behavior recognition method based on multimodal key semantics, the present application also provides an unsupervised skeleton behavior recognition system based on multimodal key semantics, including: an image acquisition unit, an action semantics generation unit, a skeleton action feature extraction unit and a multimodal feature fusion unit, wherein:

[0123] The image acquisition unit is used to acquire a pre-trained multi-modal skeleton video sequence, perform pre-processing, and construct a skeleton data set;

[0124] The action semantic generation unit is used to use a multimodal large language model to perform temporal understanding on each type of action image in the skeleton data set, generate key descriptions, and generate text description feature clusters through a semantic encoder;

[0125] The skeleton action feature extraction unit is used to capture the skeleton action feature clusters of the pictures in the skeleton dataset through a hybrid enhancement strategy and a spatiotemporal graph convolution ST-GCN model;

[0126] The multimodal feature fusion unit is used to map the text description feature cluster and the skeleton action feature cluster to the same representation space using the GLIP semantic alignment strategy, compare the similarities of different actions, and use back-propagation to train the skeleton encoder to achieve skeleton behavior recognition.

[0127] The present invention also includes a storage medium, which includes a stored program, wherein when the program is run, the unsupervised skeleton behavior recognition method based on multimodal key semantics is executed.

[0128] The present invention also includes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the unsupervised skeletal behavior recognition method based on multimodal key semantics through the computer program.

[0129] Example

[0130] like Figure 1 As shown, the present invention provides an unsupervised skeleton behavior recognition method and system based on multimodal key semantics, and optimizes the model hyperparameters through experimental analysis. Three public large-scale datasets, NTU-RGB+D, NTU-RGB+D 120, and PKU-MMD, are used. Extensive experimental results demonstrate the effectiveness of the multimodal unsupervised skeleton behavior recognition model of the present invention and show that the present invention achieves advanced performance under various evaluation protocols.

[0131] The operating system used in the experiment is Ubuntu 22.04.1, and the deep learning framework used is PyTorch. The specific configurations involved in the experiment are shown in Table 1.

[0132] Table 1 Experimental configuration table

[0133] project Configuration Processor (CPU) Intel(R)Core(TM)i9-12900KF cpu@3.20GHz Graphics card (GPU) GeForce RTX 3090 24G×2 operating system Ubuntu 22.04.1 frame PyTorch

[0134] Through sufficient ablation experiments, the effectiveness of the innovative submodules proposed in the present invention is demonstrated. Table 2 shows the linear evaluation results of the semantic components of unsupervised contrastive training, Table 3 lists the linear evaluation results of the comparison of parameters, floating point numbers and efficiency, and Table 4 shows the results of the linear evaluation of the model of the present invention compared with advanced methods on multiple action recognition benchmarks.

[0135] Table 2 Linear evaluation results of semantic components of unsupervised contrastive training on NTU-RGB+D 60

[0136] method C-sub C-view Basic Model 76.6 83.0 +General semantics 79.2 85.4 +Key semantics proposed by this invention 81.5 87.4

[0137] Table 3 Linear evaluation results of parameters, floating point numbers and efficiency comparison on NTU-RGB+D 60

[0138] method Accuracy (%) Parameter quantity Floating point numbers Time / round 3s-CrosSCLR 77.8 2.45M 2.10G 34s±2s 3s-AimCLR 78.9 1.88M 1.57G 47s±2s The present invention proposes a method 85.6 2.30M 2.01G 59s±3s

[0139] Table 4 Linear evaluation results of different modalities under different settings on human skeleton datasets NTU RGB+D 60, NTU RGB+D 120 and PKU-MMD

[0140]

[0141] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0142] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0143] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0144] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0145] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0146] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An unsupervised skeleton behavior recognition method based on multimodal key semantics, characterized in that: include: S1, collect pre-trained multimodal skeleton video sequences, perform preprocessing, and construct a skeleton dataset; S2. Use a multimodal large language model to understand the temporal sequence of each type of action image in the skeleton dataset and generate key descriptions, and generate text description feature clusters through a semantic encoder; S3, capturing skeleton action feature clusters of images in the skeleton dataset through a hybrid enhancement strategy and spatiotemporal graph convolution ST-GCN model; S4. Use the GLIP semantic alignment strategy to map the text description feature cluster and the skeleton action feature cluster to the same representation space, compare the similarities of different actions, and use back-propagation to train the skeleton encoder to achieve skeleton behavior recognition.

2. The unsupervised skeleton behavior recognition method based on multimodal key semantics according to claim 1 is characterized in that: The step S1 specifically includes: Preprocess the video sequences containing human skeleton movements, extract the first frame, middle frame and final frame of each type of movement original video as skeleton data, and construct a skeleton dataset; The preprocessing includes: extracting original skeleton data in the video sequence, eliminating noise and normalizing the viewing angle.

3. The unsupervised skeleton behavior recognition method based on multimodal key semantics according to claim 1 is characterized in that: The step S2 specifically includes: The multimodal large language model describes the specific behaviors of the skeleton in the action image in order, and organizes the description of the specific behaviors of the skeleton into description sentences; The specific behaviors include: objects in hand, hand posture, hand dynamics, torso dynamics, leg dynamics, and foot dynamics; The description sentence is input into the semantic encoder E Text (·) generates text description feature clusters: AND i =And Text (d i ) Cluster(D)=f(BN(E1,E2,...,E m )) Among them, d i represents the key description of the i-th action generated by the multimodal large language model, E i What does it mean? BN(·) means batch data normalization, f(·) means feature mapping, Cluster(·) means feature aggregation, and D means the generated text description feature cluster.

4. The unsupervised skeleton behavior recognition method based on multimodal key semantics according to claim 1 is characterized in that: The hybrid enhancement strategy specifically includes: Based on the topological information of the skeleton data, the skeleton joints are divided into five parts, and parts are randomly selected for cropping to form a cropped view; the cropped view is randomly combined with the truncated view composed of the remaining skeleton sequence to generate a mixed skeleton sequence; Extract features from the mixed skeleton sequence and separate the features of the cropped view and the truncated view through spatiotemporal mask pooling. The formula of spatiotemporal mask pooling is as follows: Among them, p represents the crop view, g represents the truncated view, M r represents the mask matrix used to indicate the cropped skeleton fragments, represents the remaining skeleton sequence, I represents the identity matrix, represents the skeleton encoded vector, T′ represents the time dimension after time downsampling, and V represents the number of nodes in the skeleton data.

5. The unsupervised skeleton behavior recognition method based on multimodal key semantics according to claim 1 is characterized in that: The spatiotemporal graph convolution ST-GCN model specifically includes: The spatiotemporal graph convolutional network ST-GCN represents the dynamic information of the human skeleton as a spatiotemporal graph G = (V, E), where V = (v1, v2, ..., v N ) represents a joint set consisting of N vertices, and E represents a set of edges connecting vertices; in the spatiotemporal graph convolutional network ST-GCN, spatial convolution and temporal convolution capture the spatial features and temporal features of skeleton data respectively; The spatial convolution is used to capture the spatial relationship between different joints in the same time frame; the spatial convolution is expressed as: Among them, f out (v i ) represents node v i The output features, f in (v j ) represents node v j Input features of node v i The set of neighbor nodes, w ij Represents node v i and its neighbor node v j The weight between The temporal convolution is used to capture the dynamic changes of the same joint between different time frames; the temporal convolution is expressed as: Among them, f out (v t ) represents the output features of time t and node v, f in (v τ ) represents the input features of time τ and node v, w(τ) represents the time weight function, and Δ represents the time window size; The spatiotemporal graph convolutional network ST-GCN combines spatial convolution and temporal convolution and works alternately to capture spatiotemporal features: Among them, f out (v ti ) represents time t and node v i The output features, f in (y(v ti ,v tj )) is the neighbor node v sampled by the sampling function y tj Input features, B(v ti ) represents node v ti The neighbor combination of Z ti (v tj ) represents the normalization term, w(v ti ,v tj ) represents the spatiotemporal weight function.

6. The unsupervised skeleton behavior recognition method based on multimodal key semantics according to claim 4 or 5, characterized in that: The cropped view p and the truncated view g are feature mapped through the spatiotemporal graph convolutional network ST-GCN: v i '=(p i ',g i ')=ST-GCN(p i ,g i ) Cluster(V)=f(P,G)=f(v1',v'2,...,v' m ) Among them, v i ' represents the mixed enhancement set of the i-th action, p i ' represents the cropped view feature after the i-th action encoding, g i ' represents the truncated view feature after the i-th action encoding, p i represents the cropped view feature of the i-th action, g i Represents the truncated view feature of the i-th action, v' m represents the hybrid enhancement set of the last action; P represents the set of all action cropped views, G represents the set of all action truncated views, f(·) represents feature mapping, Cluster(·) represents feature aggregation, and V is the generated skeleton action feature cluster.

7. The unsupervised skeleton behavior recognition method based on multimodal key semantics according to claim 1 is characterized in that: The step S4 specifically includes: Through the CLIP semantic alignment strategy, all action clipping view sets P and all action truncated view sets G are embedded with the text description feature cluster D for auxiliary fusion, and the association relationship learning method is expressed as: Among them, P' represents the multimodal feature vector of the feature vector after action cropping view data enhancement and the fusion of text feature vector, G' represents the multimodal feature vector of the feature vector after action truncated view data enhancement and the fusion of text feature vector, Similarity is used to calculate the cosine similarity of the normalized input features, Represents matrix multiplication; The multimodal feature vector P' is embedded with the corresponding key in the memory queue Q Form a positive sample pair, the multimodal feature vector G' is embedded with the corresponding key in the memory queue Q Form a negative sample pair, and convert the loss L of the multimodal feature vectors P' and G' P' and L G' Respectively expressed as: in, Used to represent Z(P') and memory queues The similarity between The loss L P' and L G' The average value of is taken as the overall contrast loss, and the calculation method of the overall contrast loss L is as follows: L=L info +λ(L P' +L G' ) Among them, L info represents the general InfoNCE loss, λ is a hyperparameter used to balance the scale of easy contrast and difficult contrast; The parameters of the skeleton encoder are updated according to the overall contrast loss L through a gradient descent algorithm, and iterative training is performed until a set number of training times is reached, and the trained skeleton encoder is used to implement action classification.

8. An unsupervised skeleton behavior recognition system based on multimodal key semantics implemented by the unsupervised skeleton behavior recognition method based on multimodal key semantics according to any one of claims 1 to 7, characterized in that: include: Image acquisition unit, action semantic generation unit, skeleton action feature extraction unit and multimodal feature fusion unit, where: The image acquisition unit is used to acquire a pre-trained multi-modal skeleton video sequence, perform pre-processing, and construct a skeleton data set; The action semantic generation unit is used to use a multimodal large language model to perform temporal understanding on each type of action image in the skeleton data set, generate key descriptions, and generate text description feature clusters through a semantic encoder; The skeleton action feature extraction unit is used to capture the skeleton action feature clusters of the pictures in the skeleton dataset through a hybrid enhancement strategy and a spatiotemporal graph convolution ST-GCN model; The multimodal feature fusion unit is used to map the text description feature cluster and the skeleton action feature cluster to the same representation space using the GLIP semantic alignment strategy, compare the similarities of different actions, and use back-propagation to train the skeleton encoder to achieve skeleton behavior recognition.

9. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the method according to any one of claims 1 to 7 is executed.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor executes the method according to any one of claims 1 to 7 by running the computer program.

Images