A method and system for intelligently recognizing the movements of a Yao long drum dance

Abstract

The application provides a Yao nationality long drum dance action intelligent recognition method and system, and the method comprises the following steps: collecting original perception data sets of dancer actions; extracting visual space-time features, extracting auditory rhythm features, modeling joint motion trajectories through ST-GCN, and forming multi-dimensional motion feature vectors; using a channel attention mechanism and uncertainty modeling to dynamically allocate modal weights and calculate uncertainty; combining the Yao nationality long drum dance knowledge graph and GNN to inject cultural characteristics, and performing semantic alignment through a cultural constraint loss function; using a Transformer fusion classifier and a confidence calibration module to perform cross-modal feature fusion through a multi-head self-attention mechanism, and generating an action category recognition result. Through high-precision space-time alignment and noise suppression technology, the application solves the space-time alignment difficulty problem caused by the difference between the video frame rate and the audio sampling rate, the spatial coordinate drift and the environmental noise interference, and improves the action recognition accuracy and robustness.

Description

Technical Field

[0001] This invention relates to the field of Yao ethnic group's long drum dance movement recognition technology, and in particular to a method and system for intelligent recognition of Yao ethnic group's long drum dance movements. Background Technology

[0002] The Yao ethnic group's long drum dance, a traditional dance art, has long relied on manual observation and experience for motion recognition. Existing technologies mostly employ single-modal data acquisition and analysis. For example, high-speed cameras capture dancers' limb trajectories, combined with manually annotated motion frames for keypoint localization; or audio spectrum analyzers extract drum frequency features and use threshold segmentation to identify the "dong-guang" two-tone rhythm. Some studies attempt to integrate video and audio data, but these often employ traditional machine learning methods. For instance, OpenPose and other skeleton extraction tools are used to obtain joint coordinates, combined with manually designed geometric features (such as limb angles and motion trajectory slopes) for motion classification; audio processing relies on Fourier transform to extract fundamental and harmonic features, using Support Vector Machines (SVM) or Hidden Markov Models (HMM) for rhythm pattern recognition. These methods can achieve basic motion classification in standard scenarios, but require significant manual intervention in feature design and are sensitive to environmental noise and individual dancer differences.

[0003] Existing technologies face challenges in spatiotemporal alignment of multimodal data. Due to the inherent difference between video frame rates and audio sampling rates, traditional methods often employ fixed time windows for forced alignment, leading to temporal misalignment between actions and drum beats (e.g., drum beats appearing earlier or later than the corresponding actions). This misalignment is particularly pronounced in dynamic performance scenarios, where rapid dancer movement causes video spatial coordinate drift, and audio echo interference further exacerbates the temporal misalignment, ultimately resulting in distorted action-rhythm feature extraction and a significant decrease in recognition accuracy. Summary of the Invention

[0004] This invention aims to at least solve the technical problems existing in the prior art, and in particular, it innovatively proposes a method and system for intelligent recognition of Yao ethnic group's long drum dance movements.

[0005] To achieve the above-mentioned objectives of this invention, this invention provides a method for intelligent recognition of Yao ethnic group's long drum dance movements, the method comprising: S1. Collect video stream data of dancers' movements, audio signals of drum beats, and joint kinematic data to form a raw perception dataset; S2. Preprocess the original sensing dataset to generate spatiotemporally aligned and noise-suppressed multimodal input data; S3. Based on the multimodal input data, visual spatiotemporal features are extracted using a CNN+ConvLSTM hybrid architecture. Auditory rhythm features are generated by combining the Mel spectrum + MFCC dual-channel audio feature extraction module. Joint motion trajectories are modeled through the ST-GCN graph convolutional network to form a multidimensional motion feature vector. S4. Based on the multi-dimensional motion feature vector, the channel attention mechanism and uncertainty modeling module are used to dynamically allocate the weights of each modality and calculate the cognitive / accidental uncertainty to generate a fusion feature vector with weight coefficients. S5. Based on the fused feature vector, the Yao ethnic group's long drum dance knowledge graph and GNN cultural feature injection system are used to perform semantic alignment between motion features and cultural features through a cultural constraint loss function, thereby generating a culturally enhanced feature representation. S6. Based on the culturally enhanced feature representation, the Transformer fusion classifier and confidence calibration module are used to perform cross-modal feature fusion through a multi-head self-attention mechanism to generate action category recognition results.

[0006] On the other hand, the present invention also provides a Yao ethnic group long drum dance movement intelligent recognition system, the system comprising: processor; Memory used to store processor-executable instructions; The processor is configured to implement the Yao ethnic group's long drum dance intelligent recognition method when executing the executable instructions.

[0007] The beneficial effects of this invention are as follows: This invention effectively solves the problem of spatiotemporal alignment difficulties caused by differences in video frame rate and audio sampling rate, spatial coordinate drift in dynamic performance scenarios, and environmental noise interference in existing technologies through high-precision spatiotemporal alignment and noise suppression technology for multimodal data. Specifically, it uses a timestamp and ArUco marker automatic spatial registration algorithm to achieve strict alignment in both time and space dimensions. Combined with bilateral filtering, wavelet threshold denoising, and Kalman filtering models, it performs targeted noise suppression on video, audio, and joint kinematic data respectively, ensuring that the multimodal input data is accurately synchronized in time, precisely registered in space, and with minimal noise interference. This preprocessing mechanism avoids the problem of action-drum timing misalignment caused by fixed time window forced alignment in traditional methods, and eliminates the negative impact of video spatial coordinate drift and audio echo interference on feature extraction in dynamic performance scenarios. This enables subsequent visual spatiotemporal feature extraction based on CNN+ConvLSTM, Mel spectrum+MFCC dual-channel audio feature extraction, and ST-GCN joint motion trajectory modeling to be carried out based on high-precision aligned and low-noise input data. Finally, through multi-dimensional motion feature vector fusion, cultural feature semantic alignment, and Transformer cross-modal fusion classification technology, the accuracy of action recognition is greatly improved, while significantly enhancing robustness to individual differences among dancers, environmental noise, and performance dynamics.

[0008] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0009] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a flowchart of a method for intelligent recognition of Yao ethnic group's long drum dance movements according to the present invention. Detailed Implementation

[0010] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0011] Example 1 like Figure 1 As shown, a method for intelligent recognition of Yao ethnic group's long drum dance movements is described, the method comprising: S1. Collect video stream data of dancers' movements, audio signals of drum beats, and joint kinematic data to form a raw perception dataset; In step S1, it is necessary to explain in detail that the selection of acquisition equipment must meet the requirements for synchronous acquisition of multimodal data. The video stream data uses a high-speed industrial camera with a resolution of 2560×1440 and a frame rate of 120fps, equipped with an 8mm fixed-focus lens and an infrared fill light to ensure that the details of the dancer's body can still be clearly captured under complex stage lighting. The drum beat audio signal is acquired through an omnidirectional condenser microphone with a sampling rate of 48kHz and a bit depth of 24bit. The microphone is fixed 30cm directly above the drumhead and records the vibration of the drumhead and the sound of the dancer's footsteps. The joint kinematic data uses a 10-point inertial motion capture system. The sensors are respectively attached to the dancer's bilateral shoulder, elbow, wrist, hip, knee, and ankle joints and the lumbar spine. The sampling frequency is set to 200Hz, and the three-dimensional acceleration, angular velocity and magnetic field strength data are transmitted in real time via Bluetooth 5.0 protocol. All acquisition devices achieve microsecond-level time synchronization through GPS timing modules, generating raw data streams containing device IDs, acquisition timestamps, and data type tags. These raw data streams are ultimately stored as raw sensing datasets. Each data record contains video frame encoded data, audio waveform arrays, six-degree-of-freedom kinematic parameters for 10 joints, and metadata about the acquisition environment (such as light intensity, room temperature, and humidity).

[0012] S2. Preprocess the original sensing dataset to generate spatiotemporally aligned and noise-suppressed multimodal input data; S3. Based on the multimodal input data, visual spatiotemporal features are extracted using a CNN+ConvLSTM hybrid architecture. Auditory rhythm features are generated by combining the Mel spectrum + MFCC dual-channel audio feature extraction module. Joint motion trajectories are modeled through the ST-GCN graph convolutional network to form a multidimensional motion feature vector. S4. Based on the multi-dimensional motion feature vector, the channel attention mechanism and uncertainty modeling module are used to dynamically allocate the weights of each modality and calculate the cognitive / accidental uncertainty to generate a fusion feature vector with weight coefficients. S5. Based on the fused feature vector, the Yao ethnic group's long drum dance knowledge graph and GNN cultural feature injection system are used to perform semantic alignment between motion features and cultural features through a cultural constraint loss function, thereby generating a culturally enhanced feature representation. S6. Based on the culturally enhanced feature representation, the Transformer fusion classifier and confidence calibration module are used to perform cross-modal feature fusion through a multi-head self-attention mechanism to generate action category recognition results.

[0013] As an optional embodiment of the present invention, optionally, preprocessing the original sensing dataset in step S2 to generate spatiotemporally aligned and noise-suppressed multimodal input data includes: S201. Timestamp the original sensor dataset to generate an intermediate dataset aligned with the time dimension; In step S201, the following steps are detailed: First, the timestamp format of the original sensory dataset is parsed and uniformly converted to Unix timestamps (accurate to the microsecond level). A time index table is then established, containing timestamp sequences of video frame timestamps, audio sampling point timestamp sequences, and joint kinematic data sampling point timestamp sequences. The Dynamic Time Warping (DTW) algorithm is used to initially match the time series of different modalities. Using the frame rate of the video stream data as the baseline time axis, linear interpolation is used to resample the audio signal and joint kinematic data in the time dimension, ensuring uniform sampling intervals for each modality on the time axis. For individual data points with missing or abnormal timestamps, a moving average of five adjacent valid timestamps is used to complete the timestamps, ensuring that the intermediate dataset with time dimension alignment is continuous and without jumps on the time axis, with time synchronization errors controlled within ±0.5ms.

[0014] S202. Based on the intermediate dataset aligned in the time dimension, the ArUco marker automatic spatial registration algorithm is used to calculate the spatial transformation matrix by identifying the coordinates of preset marker points, thereby generating an intermediate dataset with coordinate system spatial alignment. In step S202, the ArUco marker automatic spatial registration algorithm and the calculation of the spatial transformation matrix require detailed explanation. Five ArUco markers with different IDs (15cm x 15cm in size, encoded using a 4x4 dictionary) are pre-pasted at the four corners of the stage performance area and the center of the drumhead to ensure the industrial camera can capture all markers within its field of view. The video frames are then marked using OpenCV's aruco module to obtain the coordinates of the four corner points of each marker in the image pixel coordinate system. Simultaneously, an inertial motion capture system is used during the static calibration phase to record the three-dimensional coordinates of the center point of each ArUco marker in the global inertial coordinate system, establishing the correspondence between the marker pixel coordinates and the global coordinates. The DLT (Direct Linear Transformation) algorithm is used to solve for a 3x4 camera intrinsic parameter matrix (including focal length, principal point coordinates, and distortion coefficients), a 3x3 rotation matrix R, and a 3x1 translation vector T based on the corresponding coordinates of at least four non-collinear marker points, constructing the spatial transformation matrix M from the image pixel coordinate system to the global inertial coordinate system. For each frame in the video stream, the pixel coordinates of the dancer's limb area are converted into three-dimensional coordinates in the global inertial coordinate system using the M-matrix. For joint kinematic data, the original data is already in the global inertial coordinate system and requires no further transformation. For drum audio signals, the audio acquisition point is associated with the spatial position of the drum surface by marking the global coordinates of the center of the drumhead with ArUco. This spatial registration process unifies the video visual features, joint motion trajectories, and audio spatial positions to the same global coordinate system.

[0015] S203. Noise suppression processing is performed on the spatially aligned intermediate dataset. The video stream data uses a bilateral filtering algorithm to remove dynamic background noise and salt-and-pepper noise; the drum audio signal uses a wavelet threshold denoising algorithm to suppress environmental clutter and echo interference; the joint kinematic data uses a Kalman filter model to smooth the jitter noise in the motion trajectory; and multimodal input data with spatiotemporal alignment and noise suppression is generated.

[0016] In step S203, it is necessary to explain in detail that for the bilateral filtering of the video stream data, the specific parameters are set as follows: the spatial domain standard deviation is set to 5, the grayscale domain standard deviation is set to 20, and the filtering window size is 7×7 pixels. This parameter combination can effectively filter out dynamic noise caused by the flickering lights in the stage background and salt-and-pepper noise introduced by the camera sensor while preserving the details of the dancer's limb edges. The processing flow is as follows: each frame of color image is first converted to the YCrCb color space, and bilateral filtering is performed only on the luminance channel Y, and then converted back to the RGB space to avoid color information distortion. For wavelet thresholding of the drum audio signal, the db4 wavelet basis function is selected, and 5-level wavelet decomposition is performed. The high-frequency coefficients of each level are processed by a hard thresholding function. The threshold calculation formula is λ=σ√(2lnN), where σ is the noise standard deviation (estimated by dividing the median absolute value of the first-level high-frequency coefficients after decomposition by 0.6745), and N is the signal length. During processing, the main frequency band information of drum beats between 80-2000Hz is preserved. Low-frequency vibration noise below 50Hz and high-frequency environmental noise above 5000Hz are suppressed through wavelet reconstruction. For Kalman filtering of joint kinematic data, the state equation is established as Xk=AXk-1+Buk+wk, and the observation equation is Zk=HXk+vk. The state vector Xk contains the three-dimensional position, velocity, and acceleration of the joint. A is a 3×3 state transition matrix, B is the control matrix (set to 0), H is the observation matrix, and uk, wk, and vk represent the control input, process noise, and observation noise, respectively. The process noise covariance Q is set as a diagonal matrix with diagonal elements of [0.01,0.01,0.01,0.001,0.001,0.001,0.0001,0.0001,0.0001,0.0001] (corresponding to position, velocity, and acceleration noise). The observation noise covariance R is set to 0.005 according to the sensor's factory parameters. By employing a Kalman filter prediction-update iterative process, the six-DOF data for each joint are smoothed in real time, significantly reducing sensor jitter noise during rapid motion and lowering the root mean square error (RMSE) of the joint motion trajectory to within 0.5 mm. After these three preprocessing steps, the multimodal input data shows significant improvements in time synchronization accuracy, spatial coordinate consistency, and signal-to-noise ratio.

[0017] As an optional embodiment of the present invention, optionally, in step S3, visual spatiotemporal features are extracted based on the multimodal input data using a CNN+ConvLSTM hybrid architecture, auditory rhythm features are generated by combining the Mel spectrum + MFCC dual-channel audio feature extraction module, and joint motion trajectories are modeled through an ST-GCN graph convolutional network to form a multi-dimensional motion feature vector including: S301. Based on the video stream data in the multimodal input data, a ConvLSTM two-layer network is used to extract spatiotemporal features, and a visual spatiotemporal feature vector is generated by convolution kernel and ReLU activation function. In step S301, the specific configuration of the two-layer ConvLSTM network is as follows: the first ConvLSTM unit has 3 input channels (corresponding to RGB video frames), a 3×3 kernel size, 64 hidden state feature maps, a stride of 1, and "same" padding; the second ConvLSTM unit has 64 input channels (following the output of the first layer), a 3×3 kernel size, 128 hidden state feature maps, a stride of 2, and "valid" padding. The network uses 20 video frames as the input sequence. The first ConvLSTM layer extracts the spatial features of each frame and models short-term temporal dependencies between frames, generating a 64-channel spatiotemporal feature map. The second ConvLSTM layer further expands the receptive field, capturing long-term temporal motion patterns while extracting higher-level spatial features, outputting a 128-channel feature map. Each ConvLSTM unit is followed by a batch normalization layer and a dropout layer to prevent overfitting. Finally, global average pooling is used to convert the 128-channel feature map into a 128-dimensional visual spatiotemporal feature vector, which contains the dynamic change trend and spatial morphological features of the dancer's body movements.

[0018] S302. Based on the drum beat audio signal in the multimodal input data, generate an auditory feature vector using the Mel spectrum + MFCC dual-channel module; In step S302, it should be noted that the preprocessed drum audio signal is first subjected to frame-by-frame windowing, with a frame length of 2048 sampling points and a frame shift of 512 sampling points. Hamming windows are used to suppress spectral leakage. For the Mel spectrum channel, each frame's time-domain signal is converted into a spectrum using a 512-point Fast Fourier Transform. Then, a Mel spectrum diagram with a size of 40×128 (frequency domain dimension × time frame dimension) is generated using a filter bank containing 40 Mel filters (frequency range 20Hz-20kHz). After logarithmic power conversion, a 128-dimensional Mel spectrum feature vector is extracted. For the MFCC channel, 13 MFCC coefficients are calculated based on the Mel spectrum, including both first-order and second-order difference coefficients, forming a 39-dimensional MFCC feature vector. The Mel spectrum features and MFCC features are concatenated, and feature dimensionality reduction and fusion are performed using a 1×1 convolutional layer, ultimately generating a 256-dimensional auditory rhythm feature vector. This vector can simultaneously represent the pitch, intensity, rhythmic pattern, and timbre variation information of the drum beat.

[0019] S303. Based on the joint kinematic data in the multimodal input data, the ST-GCN graph convolutional network is used to define the connection edges between joint nodes and bones, and the joint motion trajectory feature vector is generated by Chebyshev multinomial spatial graph convolution and TCN temporal convolution. In step S303, it is necessary to explain in detail that the topology construction of the ST-GCN graph convolutional network adopts the standard human skeleton connection method. Ten joint sensors are defined as the node set V={v1,v2,...,v10} of the graph. Skeletal connections such as v1-v2 (shoulder-elbow), v2-v3 (elbow-wrist), v4-v5 (hip-knee), and v5-v6 (knee-ankle) are defined as the undirected edge set E, forming a human motion topology graph containing 10 nodes and 12 edges. The network input is preprocessed joint kinematic data. Each node's feature vector contains nine dimensions: three-dimensional position (x,y,z), three-dimensional velocity (vx,vy,vz), and three-dimensional acceleration (ax,ay,az). The input sequence consists of 50 consecutive sampling points in the time dimension. The ST-GCN network structure comprises four spatiotemporal convolutional modules. Each module consists of a spatial graph convolutional layer, a temporal convolutional layer, a batch normalization layer, and a ReLU activation function. The spatial graph convolutional layer employs a spectral graph convolution operator approximated by Chebyshev polynomials, with a filter order of 2. It captures spatial dependencies between joints by learning the adjacency matrix weights of skeletal connections. The temporal convolutional layer uses a 1×3 convolutional kernel with a stride of 1 and the same padding method to model the temporal sequence features of joint motion. The first spatiotemporal module has 9 input channels and 64 output channels; the second to fourth modules have 128, 256, and 512 output channels, respectively. Each module is followed by a max-pooling layer (with a temporal pooling kernel size of 2) to reduce temporal resolution. Finally, the network compresses the spatiotemporal features into a 512-dimensional joint motion trajectory feature vector through global average pooling. This vector accurately represents the amplitude of motion, velocity changes, and coordination relationships between the dancer's joints. Through feature extraction in steps S301-S303, a multi-dimensional motion feature vector is finally formed, which includes visual spatiotemporal features, auditory rhythm features, and joint motion trajectory features.

[0020] S304. Based on the visual spatiotemporal feature vector, auditory feature vector, and joint motion trajectory feature vector, feature splicing and dimensionality reduction fusion are performed with the fully connected layer using channel shuffling operation to generate multi-dimensional motion feature vectors.

[0021] In step S304, it is necessary to explain in detail that, firstly, dimensionality consistency processing is performed on the visual spatiotemporal feature vector, auditory rhythm feature vector, and joint motion trajectory feature vector. A 1×1 convolutional kernel is used to adjust the dimensionality of the visual features (128 input channels, 256 output channels) and auditory features (256 input channels, 256 output channels) to unify the three feature dimensions to 256. Then, a channel shuffling operation is performed: the three 256-dimensional feature vectors are stacked according to channel dimensions to form a 768-dimensional feature matrix, which is then divided into three groups (each group being 256-dimensional) along the channel axis. The i-th channel of each group is then rearranged into the i×3+0, i×3+1, and i×3+2 channels of the new feature matrix, achieving cross-modal feature channel shuffling and recombination, enhancing information interaction between different modal features. The shuffled feature matrix is ​​fused and dimensionality reduced using a fully connected network with two hidden layers. The first fully connected layer has an input dimension of 768 and an output dimension of 512, using the LeakyReLU activation function; the second fully connected layer has an input dimension of 512 and an output dimension of 256, using the ReLU activation function. Each layer is followed by a batch normalization layer and a dropout layer (dropout rate = 0.4). The resulting 256-dimensional multi-dimensional motion feature vector retains the unique information of each modality, including visual, auditory, and joint motion, and achieves deep intermodal correlation through channel shuffling and fully connected fusion.

[0022] As an optional embodiment of the present invention, optionally, in step S4, based on the multi-dimensional motion feature vector, the channel attention mechanism and uncertainty modeling module are used to dynamically allocate the weights of each modality and calculate the cognitive / accidental uncertainty to generate a fusion feature vector with weighted coefficients, including: S401. Based on the multi-dimensional motion feature vector, the SE-Net channel attention mechanism is used to compress spatial information through global average pooling and process it through a fully connected layer to generate the initial weight allocation result. In step S401, the specific implementation process of the SE-Net channel attention mechanism is as follows: The 256-dimensional multi-dimensional motion feature vector is treated as 256 feature channels. First, a global average pooling operation is performed on each channel, compressing the features of each channel into a 1×1 scalar value. This scalar value reflects the global importance of the corresponding channel's features. These scalar values ​​are then input into a bottleneck structure containing two fully connected layers: the first fully connected layer reduces the input dimension from 256 to 64 (64 neurons) and introduces a non-linear transformation using the ReLU activation function; the second fully connected layer restores the dimension from 64 to 256 and normalizes the output value to the 0-1 range using the Sigmoid activation function, obtaining the initial weight coefficients for each feature channel. This initial weight allocation result can initially highlight feature channels that contribute significantly to action recognition, such as channels representing joint motion amplitude or audio feature channels with strong drum rhythms.

[0023] S402. Based on the initial weight allocation results, the Monte Carlo Dropout and output entropy calculation modules are used to quantify the cognitive uncertainty and accidental uncertainty of each modality, forming an uncertainty measurement index. The expression for the output entropy calculation module is: in, This represents the output entropy value, with a range of [0, ...]. The larger the value, the higher the uncertainty of chance. Indicates the total number of action categories. The model represents the first The predicted probability of a class of actions is obtained through the output of the Softmax layer, satisfying the following conditions: .

[0024] In step S402, the cognitive uncertainty and random uncertainty require detailed explanation. Cognitive uncertainty mainly stems from the uncertainty of model parameters, which is quantified by introducing Monte Carlo Dropout (dropout rate = 0.5) into the feature fusion fully connected layer. Specifically, the process involves performing T = 20 forward propagations on the same multi-dimensional motion feature vector (randomly discarding some neurons in each propagation), resulting in T sets of predicted probability distributions. The mean and variance of all predicted probability distributions are calculated, and cognitive uncertainty is represented by the mean of the variance. This metric reflects the degree of cognitive ambiguity of the model regarding unknown action patterns; a higher value indicates lower confidence in the model's prediction of the current action, requiring reduction through increasing training data or optimizing the network structure. Random uncertainty, on the other hand, originates from the noise and variability of the data itself, and is quantified through the output entropy calculation module. Random uncertainty is taken as the mean of the output entropy of T Monte Carlo propagations. This metric reflects the inherent variability in the action execution process (such as different forces and speeds of the same action), and its value is not affected by the amount of training data, serving as an objective measure of the reliability of action recognition results. By weighting and summing cognitive uncertainty and accidental uncertainty (with weight coefficients determined to be 0.3 and 0.7 through cross-validation), a comprehensive uncertainty metric is formed to dynamically adjust the weight allocation of each modality feature.

[0025] S403. Based on the uncertainty metric, a dynamic weight adjustment strategy is used, and the weight allocation ratio is optimized by combining learnable parameters to generate modal weight coefficients that adapt to different action complexities. In step S403, it is necessary to explain in detail that the core of the dynamic weight adjustment strategy lies in establishing a mapping relationship between the uncertainty metric and the modal weight coefficients. First, the comprehensive uncertainty metric U is normalized so that its value range is mapped to the [0,1] interval, denoted as Unorm. For the visual modality, auditory modality, and joint motion modality, initial weight coefficients wv, wa, and wj are set respectively (each with an initial value of 1 / 3), and learnable parameters αv, αa, and αj (initialized to 0.5) are introduced to adjust the sensitivity of each modality to uncertainty. The dynamic weight adjustment formula is defined as: wv' = wv (1+αv Unorm),wa'=wa (1+αa Unorm), wj'=wj (1+αj When the overall uncertainty (Unorm) is high (e.g., in complex movements or scenarios with high data noise), the weight allocation for uncertainty-sensitive modalities is enhanced by a learnable parameter α. For example, joint motion modalities are more sensitive to changes in movement details, and their αj value will adaptively increase through backpropagation during training, giving this modality a higher weight under high uncertainty. The adjusted weights are then L2 normalized to ensure wv'+wa'+wj'=1. By minimizing the joint loss function of action classification loss and uncertainty-weighted loss during training, the learnable parameters αv, αa, and αj are optimized, allowing the modality weight coefficients to dynamically adjust according to the complexity of the movement (e.g., rapid changes like "short-stance steps" or continuous movements like "long drum circles"), achieving adaptive fusion of different modal features.

[0026] S404. Based on the modal weighting coefficients, the visual, auditory and kinematic feature vectors are weighted and fused using element-wise multiplication to generate a weighted fused feature vector. In step S404, it is necessary to explain in detail that this embodiment uses element-wise multiplication to achieve weighted fusion: the multi-dimensional motion feature vector is decomposed into visual sub-vectors, auditory sub-vectors, and joint motion sub-vectors according to the modality source, and then multiplied element-wise with the modality weight coefficients wv', wa', and wj' respectively. Subsequently, these three weighted sub-vectors are re-concatenated according to the channel dimension to form a weighted fusion feature vector, wherein the channel proportions of visual, auditory, and joint motion features are dynamically adjusted according to the weight coefficients. The final weighted fusion feature vector not only retains the core information of each modality, but also achieves dynamic adaptation of modality importance under different action scenarios through the weight coefficients.

[0027] S405. Based on the weighted fusion feature vector, the feature dimensionality reduction module integrates the feature dimensions and retains key information through a fully connected layer to generate a fusion feature vector with weight coefficients.

[0028] In step S405, it is necessary to explain in detail that the feature dimensionality reduction module adopts a fully connected network structure with one hidden layer. The input is the weighted fusion feature vector (256 dimensions) generated in step S404. The number of neurons in the hidden layer is set to 128, and the ELU (Exponential Linear Unit) activation function is used to alleviate the gradient vanishing problem. At the same time, a batch normalization layer and a Dropout layer are connected after the hidden layer to further enhance the model's generalization ability. The number of neurons in the output layer is 64, and no activation function is set. The 64-dimensional fusion feature vector with weighted coefficients is directly output. This dimensionality reduction process reduces the feature dimensionality while ensuring that the key features crucial to the recognition of Yao ethnic group's long drum dance are preserved, such as the rhythmic features of the long drum, the trajectory features of the limb joints, and the visual dynamic features of the dance movements, by minimizing the joint optimization objective of feature reconstruction error and action classification loss.

[0029] As an optional embodiment of the present invention, optionally, in step S5, based on the fused feature vector, the Yao ethnic group's long drum dance knowledge graph and GNN cultural feature injection system are used to perform semantic alignment of motion features and cultural features through a cultural constraint loss function to generate a culturally enhanced feature representation, including: S501. Based on the fused feature vector, the knowledge graph of the Yao ethnic group's long drum dance is used to generate a knowledge graph association feature vector through action-rule-culture triplet association query. In step S501, regarding the Yao Long Drum Dance knowledge graph and the action-rule-culture ternary association query, it is necessary to explain in detail that the Yao Long Drum Dance knowledge graph adopts a construction method combining top-down and bottom-up approaches, with "action units" as the core nodes, integrating knowledge from multiple fields such as dance, ethnology, and musicology. The ontology structure of the knowledge graph includes six core entity types: action units (such as "short stance step" and "long drum circle"), cultural connotations (such as "praying for blessings" and "exorcising evil"), action rules (such as "knee flexion and extension range not less than 30°" and "drum rhythm is 4 / 4 time"), costumes and props (such as "long drum" and "embroidered Yao brocade"), scene context (such as "initiation ceremony" and "Panwang Festival celebration"), and inheritors (such as "national intangible cultural heritage inheritors"). Entities are connected through 12 semantic relationships, among which the "action-rule" relationship includes attributes such as "execution standard", "strength requirement", and "rhythmic characteristics", and the "action-culture" relationship includes attributes such as "symbolic meaning", "historical origin", and "regional variation". The action-rule-culture triple association query is implemented using the SPARQL language. The returned results include triple information such as "the drumhead rotation trajectory must be a perfect circle" (rule) and "symbolizing sun worship and the cycle of life" (cultural connotation). The rule parameters (such as angle threshold and rhythm cycle) and cultural tags (such as "praying for blessings" and "harvest") in the query results are transformed into 32-dimensional vectors through a word embedding model. After feature concatenation, a knowledge graph association feature vector is formed, which contains the normative constraints of actions and cultural semantic information.

[0030] S502. Based on the knowledge graph associated feature vector, the GNN cultural feature injection system is used to generate cultural feature vectors through the GraphSAGE graph convolutional network and cultural attention mechanism. In step S502, it is necessary to explain in detail that the core architecture of the GNN cultural feature injection system includes three parts: a graph construction module, a GraphSAGE convolutional layer, and a cultural attention mechanism. First, the graph construction module transforms the knowledge graph's associated feature vectors into a heterogeneous graph structure: using the joint motion trajectory features in the fused feature vectors as the central node, action units, cultural connotations, and action rule entities in the knowledge graph as adjacent nodes, and semantic relationships between entities as directed edges (e.g., "short stance - execution standard - knee flexion / extension 30°"). Edge weights are dynamically assigned according to the relationship type, with the "action-rule" relationship weight set to 0.6 (emphasizing action standardization) and the "action-culture" relationship weight set to 0.4 (highlighting cultural connotation). The GraphSAGE convolutional layer employs a sampling aggregation strategy: randomly sampling 20 adjacent nodes for each central node, fusing the features of the adjacent nodes with the central node features through a mean aggregator, and learning the associated features between entities through two layers of graph convolution operations (hidden layer dimensions of 64 and 32 respectively). The cultural attention mechanism sets dynamic weights for different cultural attributes: for word vectors of cultural attributes such as "symbolic meaning" and "historical origin," the semantic similarity between them and action features is calculated using Bahdanau attention. Cultural attributes with high similarity (such as the "harvest" connotation in the "Panwang Festival" scenario) receive higher attention weights. Finally, the graph features output by GraphSAGE are concatenated with the attribute features weighted by cultural attention to generate a 32-dimensional cultural feature vector. This vector contains both the structured association information of the knowledge graph and highlights the cultural contextual features of the action.

[0031] S503. Based on the cultural feature vector, a semantic alignment loss value is generated by using a cultural constraint loss function and the dual constraints of motion feature loss and cultural feature loss. In step S503, it is necessary to explain in detail that the cultural constraint loss function is composed of a weighted sum of the motion feature loss Lmotion and the cultural feature loss Lculture. The total loss function Ltotal = β·Lmotion + (1-β)·Lculture, where β is the balance coefficient (set to 0.6 after experimental verification). The motion feature loss Lmotion adopts cross-entropy loss, calculating the difference between the predicted probability of the action category output by the fused feature vector through the Softmax layer and the true label, ensuring that the model learns accurate action recognition features. The cultural feature loss Lculture achieves semantic alignment between motion features and cultural features through cosine similarity loss, and its expression is: Lculture = 1 - cos(Ffusion, Fculture), where Ffusion is a 64-dimensional fused feature vector, and Fculture is a 32-dimensional cultural feature vector (filled with zeros to match the dimensionality to 64). cos(·) represents the cosine similarity function, and the closer the value is to 1, the higher the semantic consistency between the two feature vectors. By minimizing Lculture, the fused feature vector is forced to not only contain the physical motion information of the action, but also to be consistent with the cultural semantic information in the knowledge graph. For example, when recognizing the "long drum circling" action, the model needs to simultaneously learn the trajectory features of the drum surface rotation (motion features) and its symbolic cultural meaning (cultural features), bringing the two closer together in the feature space. Furthermore, to enhance the constraint strength of cultural features, a cultural attribute weight vector, Wculture, is introduced into Lculture. This vector weights different cultural sub-features based on the importance of cultural attributes in the knowledge graph (e.g., "symbolic meaning" weight is set to 0.3, and "historical origin" weight is set to 0.2), further increasing the contribution of key cultural information to semantic alignment.

[0032] S504. Based on the semantic alignment loss value, a dynamic weight adjustment strategy is used to optimize the fusion ratio of motion features and cultural features through learnable parameters, and an adjusted fusion feature vector is generated. In step S504, it is necessary to explain in detail that the core of the dynamic weight adjustment strategy in the semantic alignment stage is to dynamically optimize the fusion ratio of motion features and cultural features based on the semantic alignment loss value. First, the semantic alignment loss value Ltotal is normalized to obtain the normalized loss value Lnorm (with a value range of [0,1]). Initial values ​​are set for the motion feature weight wm and the cultural feature weight wc (wm is initially set to 0.7, wc is initially set to 0.3, prioritizing the fundamental role of motion features), and learnable parameters γm and γc (initialized to 0.4) are introduced to control the degree of influence of the loss value on the weights. The dynamic adjustment formula is defined as: wm' = wm (1+γm (1-Lnorm)), wc'=wc (1+γc When the semantic alignment loss value Lnorm is high (indicating a large semantic deviation between motion features and cultural features), the cultural feature weight wc' is increased by increasing γc, thus strengthening the role of cultural features in the fusion. Conversely, when Lnorm is low (indicating good semantic alignment), the motion feature weight wm' is kept dominant by increasing γm. The adjusted weights need to be L1 normalized to ensure wm' + wc' = 1. Subsequently, the fused feature vector Ffusion (64-dimensional) and the cultural feature vector Fculture (zero-padded to 64-dimensional) are multiplied element-wise with wm' and wc' respectively, and then added according to the channel dimension to generate the adjusted fused feature vector. This process minimizes the total loss function Ltotal during training and optimizes the learnable parameters γm and γc through backpropagation, enabling the model to dynamically balance the fusion ratio of motion features and cultural features according to the importance of cultural attributes of different actions.

[0033] S505. Based on the adjusted fused feature vector, the feature fusion module performs cross-modal feature fusion through contrastive learning and gating fusion unit to generate a culturally enhanced feature representation.

[0034] In step S505, the feature fusion module, contrastive learning, and gating fusion unit require detailed explanation. The feature fusion module employs a two-layer fusion architecture combining contrastive learning and gating fusion units, aiming to achieve deep coupling between motion features and cultural features. First, the contrastive learning module optimizes the feature space distribution by constructing positive and negative sample pairs: positive sample pairs consist of an adjusted fused feature vector (motion-dominant) and a cultural feature vector (culture-dominant) for the same Yao ethnic group's long drum dance movement; negative sample pairs randomly select feature vectors from different movements for pairing. The InfoNCE loss function is used to calculate the similarity difference between positive and negative sample pairs. By minimizing this loss, the motion and cultural features of the same movement are clustered in the feature space, while features of different movements are separated, enhancing the semantic consistency of the features.

[0035] The gated fusion unit is designed based on the gating mechanism of the existing gated recurrent unit (GRU) to dynamically control the fusion weights of motion features and cultural features. Specifically, in use, the gated fusion unit denotes the adjusted fused feature vector (motion features dominant) as Fm and the cultural feature vector (cultural features dominant) as Fc. First, the reset gate r is used to calculate: r = σ(Wr·[Fm,Fc] + br), where Wr is the weight matrix, br is the bias term, and σ is the sigmoid activation function. The value range of the reset gate r is [0,1], which is used to determine the degree of preservation of cultural feature information in Fc. When r is close to 1, it indicates that the cultural details of Fc are fully preserved; when r is close to 0, it tends to ignore the information of Fc. Next, the update gate z is used to calculate: z = σ(Wz·[Fm,Fc] + bz), where Wz and bz are the weight matrix and bias term of the update gate, respectively. The update gate z controls the contribution ratio of Fm and Fc in the final fused features. The larger the z value, the higher the proportion of motion features of Fm; conversely, the higher the proportion of cultural features of Fc. Subsequently, the candidate fused feature vector h is calculated: h = tanh(Wh·[r⊙Fc,Fm] + bh), where Wh and bh are the weight matrix and bias term of the candidate features, and ⊙ represents element-wise multiplication. Finally, the culturally enhanced feature representation h is generated: h = (1-z)⊙Fc + z⊙h. This gating mechanism can adaptively adjust the fusion strategy according to the cultural complexity of the action. For example, for the culturally rich "ritual of rites of passage," the gating unit will reduce the z value and increase the r value to enhance the fusion weight of cultural features, ensuring that cultural semantics such as "praying for blessings" and "exorcising evil" are fully reflected in the feature representation; while for the "short-stance step," which has high requirements for action standardization, the z value will be increased to highlight the accuracy of the movement trajectory. By combining comparative learning to optimize feature distribution with dynamic weighting of gating units, the resulting culturally enhanced feature representation not only has accurate action recognition capabilities but also deeply integrates the cultural connotations of the Yao ethnic group's long drum dance.

[0036] As an optional embodiment of the present invention, optionally, in step S6, based on the culturally enhanced feature representation, a Transformer fusion classifier and a confidence calibration module are used to perform cross-modal feature fusion through a multi-head self-attention mechanism to generate action category recognition results, including: S601. Based on the culturally enhanced feature representation, the Transformer encoder is used to capture long-distance dependencies of cross-modal features through a self-attention mechanism, generating a cross-modal fusion feature vector. In step S601, it is necessary to explain in detail that the Transformer encoder consists of four stacked Transformer blocks, each containing a multi-head self-attention sublayer and a feedforward neural network sublayer. The culturally enhanced feature representation first adds temporal positional information (using sine and cosine positional encoding) through a positional encoding module, transforming it into a 64×64 feature matrix (sequence length 64, feature dimension 64). The multi-head self-attention sublayer sets up eight attention heads, each with a feature dimension of 8 (64-dimensional features are evenly divided into 8 heads). By calculating the scaled dot product attention of the query Q, key K, and value V matrices, it captures the long-distance dependencies between different action frames in the feature sequence, such as the temporal correlation between the drum rhythm and limb swing in the "long drum loop" action. After concatenation and linear transformation, the multi-head attention output is processed through residual connections and layer normalization (LayerNorm) before being input into the feedforward neural network sublayer. The feedforward neural network adopts a two-layer fully connected structure with a hidden layer dimension of 256 and an activation function of GELU (GaussianErrorLinearUnit) to perform nonlinear transformation and dimension mapping on the attention-fused features. The final output is a cross-modal fusion feature vector with a dimension of 64×64, which integrates the kinematic features of the action, cultural semantic features, and temporal dynamic features.

[0037] S602. Based on the cross-modal fusion feature vector, the confidence calibration module is used to adjust the prediction probability distribution through the temperature scaling parameter to generate a calibrated prediction probability vector. In step S602, it is necessary to explain in detail that, in this embodiment, the confidence calibration module addresses the overconfidence problem in the predicted probability of the cross-modal fusion feature vector output by the Transformer encoder by using a temperature scaling method. First, the cross-modal fusion feature vector is input into a fully connected classification layer to obtain the original logits vector (dimension equal to the number of Yao ethnic group long drum dance movement categories, such as 20 dimensions). Temperature scaling is applied by introducing a learnable parameter temperature T (initial value set to 1.0) to scale the original logits. The calibration formula is: calibrated logits = original logits / T. Subsequently, the calibrated logits are input into the Softmax function to generate a calibrated predicted probability vector. During training, the temperature parameter T is optimized by minimizing the negative log-likelihood loss. Specifically, the NLLLoss is calculated between the calibrated predicted probability vector and the true action category label, and the T value is adjusted through backpropagation to make the probability distribution output by the model closer to the true uncertainty level. For example, when the model's initial predicted probability for the "long drum circling" action is 0.98 (overconfidence), lowering the temperature T (e.g., to 0.7) can reduce the calibrated probability to 0.85, better reflecting the actual recognition difficulty. Conversely, for the simple action "short stump step," if the initial predicted probability is 0.8, the temperature T might be fine-tuned to 1.1, slightly increasing the probability to 0.83 and balancing the overall confidence distribution. The calibrated predicted probability vector not only improves the reliability of the model's output.

[0038] S603. Based on the calibrated prediction probability vector, the Transformer decoder is used to perform semantic mapping between features and action categories through a multi-head self-attention mechanism to generate initial action category prediction results. In step S603, it is necessary to explain in detail that the Transformer decoder adopts the same 4-layer stacked structure as the encoder, but each layer additionally includes an encoder-decoder attention sublayer to establish the association between the cross-modal fusion feature vector and the action category semantics. First, the action category embedding corresponding to the calibrated predicted probability vector is used as the initial input sequence of the decoder, and temporal information is added through positional encoding. The encoder-decoder attention sublayer uses the cross-modal fusion feature vector output by the encoder as the key K and value V, and the hidden state of the current input sequence of the decoder as the query Q, to calculate the attention weights, thereby dynamically matching the action features with the category semantics. For example, when the cross-modal feature "long drum looping" is input, the decoder focuses on the dimensions related to "rotation trajectory" and "drum surface orientation" in the feature through the attention mechanism, and associates them with the semantic embedding of the "long drum looping" category. The multi-head self-attention sublayer (also with 8 attention heads) captures the dependencies within the action category sequence, such as the similarity or repulsion between different action categories. After linear transformation and softmax activation, the decoder output generates the initial action category prediction results, i.e., the predicted probability distribution of each action category.

[0039] S604. Based on the initial prediction results, the dynamic threshold filtering module is used to filter low-confidence predictions by adaptive threshold adjustment, and a candidate action category set is generated. In step S604, it is necessary to explain in detail that the dynamic threshold filtering module aims to solve the problems of class imbalance and misidentification of complex movements in the Yao ethnic group's long drum dance. It filters low-confidence predictions and retains high-reliability candidate movement categories through an adaptive threshold adjustment mechanism. The specific implementation process is as follows: First, for the 20 movement categories of the Yao ethnic group's long drum dance (such as "long drum circle," "short stake step," and "ritual ceremony"), an initial threshold θi (i=1,2,...,20) is set for each category. The initial threshold is initialized based on the F1-score of that category in the training set. The higher the F1-score, the lower the difficulty of the movement recognition, and the lower the initial threshold θi is set (e.g., the F1-score for "short stake step" is 0.92, and θi is initially set to 0.6; the F1-score for "ritual ceremony" is 0.75, and θi is initially set to 0.75). Subsequently, a dynamic adjustment factor α is introduced, which is updated according to the deviation between the average confidence of each category in the current batch prediction results and the historical average confidence. If the average confidence score of a category in the current batch is more than 15% lower than the historical average confidence score (indicating increased difficulty in recognizing the action in this batch), its threshold θi is reduced by α (α=0.05) to avoid missed detections. Conversely, if the average confidence score is more than 20% higher than the historical average confidence score (indicating improved recognition stability), the threshold θi is increased by α to reduce false detections. Furthermore, to prevent over-adjustment of the threshold, upper and lower limits are set (θmin=0.5, θmax=0.9). For the initial action category prediction results, the predicted probability of each category is compared with the corresponding dynamic threshold θi. When the predicted probability P≥θi, the category is included in the candidate action category set. If the predicted probabilities of all categories for a given input action are lower than the corresponding threshold (e.g., fuzzy actions or undefined actions), the top 3 categories with the highest predicted probabilities are selected as candidates to ensure the completeness of the results. For example, when identifying the "long drum circling" action, if its predicted probability is 0.82 and the corresponding dynamic threshold θi is adjusted to 0.78, then this category is included in the candidate set; while the "butterfly spreading its wings" action, with a predicted probability of 0.65 and a threshold θi of 0.7, is filtered out.

[0040] S605. Based on the candidate set, a non-maximum suppression algorithm is used to filter the optimal action category through the IOU threshold, and an action category recognition result is generated.

[0041] In step S605, it is necessary to explain in detail that the Non-Maximum Suppression (NMS) algorithm is used to select the optimal action category with the highest confidence and no redundancy from the candidate action category set. Specifically, the candidate action category set is first sorted from highest to lowest predicted probability. For the Yao ethnic group's long drum dance action recognition scenario, since each input action usually corresponds to a single category, the Intersection over Union (IOU) between candidate categories needs to be calculated to measure the overlap between categories. Here, IOU is defined as the similarity between two action categories in the feature space, approximated by calculating the cosine similarity of their cultural enhancement feature representations (the higher the cosine similarity, the larger the IOU value, indicating that the two categories are more similar at the feature level). An IOU threshold is set (e.g., 0.5), and the category with the highest probability is selected from the sorted candidate set as the baseline category. Then, the IOU values ​​of subsequent categories with the baseline category are calculated sequentially. If the IOU value of a category with the baseline category is greater than the set threshold, it is considered highly similar to the baseline category and belongs to redundant prediction, so it is suppressed; otherwise, the category is retained as a new baseline category, and subsequent comparisons continue. For example, the candidate set includes "long drum circling" (probability 0.82), "single drum rotation" (probability 0.75, IOU=0.6 with "long drum circling"), and "short stump step" (probability 0.70, IOU=0.3 with "long drum circling"). If the IOU threshold is set to 0.5, "single drum rotation" is suppressed because its IOU with "long drum circling" is greater than 0.5, and "long drum circling" and "short stump step" are retained. However, considering that a single movement of the Yao ethnic group's long drum dance usually corresponds to a unique category, the "long drum circling" movement with the highest probability in the retained set is further selected as the final movement category recognition result. This achieves accurate screening of the optimal movement category in complex movement sequences and effectively avoids multi-class misjudgment caused by interference from similar movement features.

[0042] Example 2 A Yao ethnic group long drum dance movement intelligent recognition system includes: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the Yao ethnic group long drum dance movement intelligent recognition method when executing the executable instructions. It should be noted that the computer device includes a processor and a memory, and may also include one or more of the following: multimedia components, input / output (I / O) interfaces, and communication components. The processor controls the overall operation of the computer device, completing all or part of the steps of the Yao ethnic group long drum dance movement intelligent recognition method. The memory stores various types of data to support device operation and can be implemented by volatile or non-volatile storage devices or combinations thereof, such as SRAM, EEPROM, etc. The multimedia components include a screen (such as a touch screen) and an audio component. The audio component has a microphone to receive external audio signals and also includes at least one speaker to output audio signals. The I / O interface provides an interface for the processor and other interface modules (such as a keyboard, mouse, buttons, etc.). The communication component is used for wired or wireless communication between devices. Wireless communication includes Wi-Fi, Bluetooth, etc., and the communication component includes a Wi-Fi module, etc. As a preferred embodiment, the computer device can be implemented using electronic components such as ASICs and DSPs to execute the Yao ethnic group long drum dance movement intelligent recognition method.

[0043] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for intelligent recognition of Yao ethnic group's long drum dance movements, characterized in that, The method includes: Collect video stream data of dancers' movements, audio signals of drum beats, and joint kinematic data to form a raw perception dataset; The original sensor dataset is preprocessed to generate spatiotemporally aligned and noise-suppressed multimodal input data; Based on the multimodal input data, visual spatiotemporal features are extracted using a CNN+ConvLSTM hybrid architecture. Auditory rhythm features are generated by combining the Mel spectrum + MFCC dual-channel audio feature extraction module. Joint motion trajectories are modeled through the ST-GCN graph convolutional network to form a multidimensional motion feature vector. Based on the multi-dimensional motion feature vector, the channel attention mechanism and uncertainty modeling module are used to dynamically allocate the weights of each modality and calculate the cognitive / accidental uncertainty to generate a fusion feature vector with weight coefficients. Based on the fused feature vector, the Yao ethnic group's long drum dance knowledge graph and GNN cultural feature injection system are used to semantically align motion features and cultural features through a cultural constraint loss function, generating culturally enhanced feature representations. Based on the culturally enhanced feature representation, the Transformer fusion classifier and confidence calibration module are used to perform cross-modal feature fusion through a multi-head self-attention mechanism to generate action category recognition results.

2. The intelligent recognition method for Yao ethnic group's long drum dance movements as described in claim 1, characterized in that, The generated spatiotemporally aligned and noise-suppressed multimodal input data includes: The original sensor dataset is timestamped to generate an intermediate dataset aligned with the time dimension; Based on the intermediate dataset aligned in the time dimension, the ArUco marker automatic spatial registration algorithm is used to calculate the spatial transformation matrix by identifying the coordinates of preset marker points, thereby generating an intermediate dataset with coordinate system spatial alignment. Noise suppression processing is performed on the spatially aligned intermediate dataset. The video stream data uses a bilateral filtering algorithm to remove dynamic background noise and salt-and-pepper noise; the drum audio signal uses a wavelet thresholding algorithm to suppress environmental clutter and echo interference; the joint kinematic data uses a Kalman filter model to smooth the jitter noise in the motion trajectory; and spatiotemporally aligned and noise-suppressed multimodal input data is generated.

3. The intelligent recognition method for Yao ethnic group's long drum dance movements as described in claim 1, characterized in that, The formation of multi-dimensional motion feature vectors includes: Based on the video stream data in the multimodal input data, a two-layer ConvLSTM network is used to extract spatiotemporal features, and a visual spatiotemporal feature vector is generated by convolution kernels and ReLU activation function. Based on the drum beat audio signal in the multimodal input data, an auditory feature vector is generated using the Mel spectrum + MFCC dual-channel module; Based on the joint kinematic data in the multimodal input data, the ST-GCN graph convolutional network is used to define the connection edges between joint nodes and bones, and the joint motion trajectory feature vector is generated by Chebyshev multinomial spatial graph convolution and TCN temporal convolution. Based on the aforementioned visual spatiotemporal feature vector, auditory feature vector, and joint motion trajectory feature vector, a multi-dimensional motion feature vector is generated by using channel shuffling operation and fully connected layer for feature splicing and dimensionality reduction fusion.

4. The intelligent recognition method for Yao ethnic group's long drum dance movements as described in claim 1, characterized in that, Generating a fused feature vector with weighted coefficients includes: Based on the multi-dimensional motion feature vector, the SE-Net channel attention mechanism is used to compress spatial information through global average pooling and process it through a fully connected layer to generate the initial weight allocation result. Based on the initial weight allocation results, the Monte Carlo Dropout and output entropy calculation modules are used to quantify the cognitive uncertainty and accidental uncertainty of each modality, forming an uncertainty measurement index. Based on the aforementioned uncertainty metric, a dynamic weight adjustment strategy is used, combined with learnable parameters to optimize the weight allocation ratio, thereby generating modal weight coefficients that adapt to different action complexities. Based on the modal weight coefficients, the visual, auditory, and kinematic feature vectors are weighted and fused using element-wise multiplication to generate a weighted fused feature vector. Based on the weighted fusion feature vector, the feature dimensionality reduction module integrates the feature dimensions and retains key information through a fully connected layer to generate a fusion feature vector with weight coefficients.

5. The intelligent recognition method for Yao ethnic group's long drum dance movements as described in claim 4, characterized in that, The expression for the output entropy calculation module is: in, This represents the output entropy value, with a range of [0, ...]. The larger the value, the higher the uncertainty of chance. Indicates the total number of action categories. The model represents the first The predicted probability of a class of actions is obtained through the output of the Softmax layer, satisfying the following conditions: .

6. The intelligent recognition method for Yao ethnic group's long drum dance movements as described in claim 1, characterized in that, Generative culture-enhanced feature representations include: Based on the fused feature vector, the Yao ethnic group's long drum dance knowledge graph is used to generate a knowledge graph association feature vector through action-rule-culture triplet association query. Based on the knowledge graph associated feature vector, the GNN cultural feature injection system is used to generate cultural feature vectors through the GraphSAGE graph convolutional network and cultural attention mechanism. Based on the cultural feature vector, a semantic alignment loss value is generated by using a cultural constraint loss function and subjecting it to dual constraints of motion feature loss and cultural feature loss. Based on the semantic alignment loss value, a dynamic weight adjustment strategy is used to optimize the fusion ratio of motion features and cultural features through learnable parameters, thereby generating an adjusted fusion feature vector. Based on the adjusted fused feature vector, the feature fusion module performs cross-modal feature fusion through contrastive learning and gating fusion unit to generate a culturally enhanced feature representation.

7. The intelligent recognition method for Yao ethnic group's long drum dance movements as described in claim 1, characterized in that, The generated action category recognition results include: Based on the culturally enhanced feature representation, the Transformer encoder is used to capture long-distance dependencies of cross-modal features through a self-attention mechanism, generating a cross-modal fused feature vector. Based on the cross-modal fusion feature vector, the confidence calibration module is used to adjust the prediction probability distribution through temperature scaling parameters to generate a calibrated prediction probability vector. Based on the calibrated predicted probability vector, the Transformer decoder is used to perform semantic mapping between features and action categories through a multi-head self-attention mechanism to generate initial action category prediction results. Based on the initial prediction results, a dynamic threshold filtering module is used to filter low-confidence predictions through adaptive threshold adjustment, generating a candidate action category set. Based on the candidate set, a non-maximum suppression algorithm is used to filter the optimal action category through an IOU threshold, and action category recognition results are generated.

8. A smart recognition system for Yao ethnic group's long drum dance movements, characterized in that, The system includes: processor; Memory used to store processor-executable instructions; The processor is configured to implement the Yao ethnic group's long drum dance intelligent recognition method according to any one of claims 1 to 7 when executing the executable instructions.

Images