A misaction recognition method for a VR / AR device

Abstract

This application discloses a method for error action recognition in VR / AR devices, comprising the following steps: Step 1, collecting the original audio / video dataset of playing errors; Step 2, performing data augmentation on the original samples; Step 3, constructing a contrastive learning model; inputting the original samples and augmented data into the contrastive learning model for feature extraction, and simultaneously performing interpolation to obtain positive sample pairs; Step 4, inputting the original dataset and the augmented dataset into the contrastive learning model to train the model, obtaining a trained contrastive learning model. Based on interpolation, more reliable positive sample pairs are constructed. By augmenting the action sequence through interpolation, the duration of the error action is lengthened, making it easier to capture the error action. A novel contrastive loss function is proposed to mask instances in the same cluster, and instance weights are designed to avoid interference from invalid samples in the contrastive learning process.

Description

TECHNICAL FIELD

[0001] The present application relates to an interpolation-based cross-modal contrast learning algorithm, in particular, a momentum coding-based contrast learning framework is used, a method of interpolation and extrapolation is designed for data enhancement, and a method of reconstructing negative samples in contrast learning is designed. The action and audio double modal input can be used to identify the action errors in the process of playing music. BACKGROUND

[0002] Playing music has become the hobby and profession of many people. In the process of playing music, it is easy to have some difficult to detect error habits. For example, inaccurate finger placement, wrong high and low pitch, etc. It is difficult for the learner himself or the teacher to correct these errors. This potential error playing habit will affect the quality of the performance and will cause damage to the hand joints to some extent.

[0003] In the technical aspect, the commonly used method for hand action recognition is supervised learning on a labeled data set, but this requires a large amount of labeled data and manual labor, and the labeling process is very tedious and subjective. Self-supervised learning is a method of learning the features of the data itself rather than the label, which can effectively reduce the demand for labeled data. Therefore, insufficient data labels have become the main challenge of action recognition. In order to reduce the dependence on labels, researchers have developed a self-supervised paradigm called contrast learning, and indeed can achieve excellent performance in computer vision tasks.

[0004] The main process of contrast learning is to obtain a large number of pseudo-labels through data augmentation in the pre-training process, so that the model can distinguish which augmented instances are positive pairs and which are negative pairs. After pre-training, the model can use a small amount of labeled data to fine-tune in downstream tasks and can obtain the same competitiveness as supervised learning. There are many types of contrast learning for pre-training tasks, including MoCo (Momentum Contrast), MoCo.V2 and SimCLR (Simple Framework for Contrastive Learning of Visual Representations) use example discrimination as the task, NNCLR (Nearest-Neighbor Contrastive Learning of Visual Representations), MSF (Mean Shift Feature) uses a method based on positive and negative sample redefinition.

[0005] Data augmentation is considered very critical in contrastive learning, it can design various motion patterns for the same semantic representation, and directly affects the quality of the feature representation learned by the model, unlike RGB human action recognition, 3D skeleton data has more complex modal representation, and is more sensitive to morphological changes, so it is more difficult to design an enhancement method. The core content of contrastive learning for skeleton action recognition lies in the design of data augmentation and the construction method of positive and negative samples. And in the process of data set construction, when collecting music playing, it is more difficult to collect data of playing music with wrong habits than to collect action data based on class-based action recognition. SUMMARY

[0006] Based on the existing deep learning technology, an error action recognition method for VR / AR equipment is proposed, a method of interpolation and extrapolation is designed for data augmentation based on the momentum coding contrastive learning framework, and a method of reconstructing negative samples in contrastive learning is designed. The action and audio dual modal input can automatically correct the action error in the process of playing music.

[0007] The error action recognition method for VR / AR equipment comprises the following steps:

[0008] Step 1, collect the original audio and video data set HM(b,n) and AS(b,n) of playing wrong music, wherein HM(b,n) is a hand action data set, AS(b,n) is a corresponding audio data set, the original data set includes original samples x, including original hand action data and corresponding audio data.

[0009] Step 2, data augmentation is performed on the original sample;

[0010] Step 3, constructing a contrastive learning model;

[0011] Interpolation value calculation is performed after entering the encoder. Especially in the extrapolation process, inter-frame and intra-frame interpolation is used, which reduces the data collection of the application and improves the data quality.

[0012] Step 4, input the original data set and the data set after data augmentation into the contrastive learning model, train the model, and obtain the trained contrastive learning model.

[0013] Further, the original sample is data augmented in step 2, specifically including the following:

[0014] The original sample x in the original action data set HM(b,n) is flipped, rotated and randomly masked to obtain sample enhanced data x en ;

[0015] The sample enhanced data x enExtrapolation augmentation is performed to obtain extrapolated augmented data x. ex .

[0016] Furthermore, this invention uses the original samples and weakly enhanced data to perform extrapolation calculations before entering the encoder. The extrapolation includes two steps: intra-frame interpolation and inter-frame interpolation.

[0017] (2) Intraframe interpolation:

[0018] Combine the original sample x and the augmented sample data x en Perform frame-by-frame fusion; after fusion, perform inter-frame interpolation, and expand the fused data length to m times the original.

[0019] The interpolated frame sequence M generated by interpolation:

[0020]

[0021] Where N is the temporal length of the original sample x, set to 50, and i represents the i-th frame.

[0022] (2) Use the intra-frame interpolation generation sequence M to perform inter-frame interpolation, that is, generate several frames between frames to expand the data, and finally obtain extrapolated enhanced data.

[0023] Specifically, inter-frame interpolation is performed on sequence M in the time domain:

[0024]

[0025] The i-th frame of sequence M is mapped to extrapolated augmented data x ex In the m*i-th frame, the interpolated frames between the i-th frame and the i+1-th frame are sequentially mapped to the m*i+1, m*i+2, ..., m*i+m-1-th frames, where j = m-1;

[0026] Furthermore, a first-order Markov chain is used to describe the state transitions of the skeleton data to augment the generated extrapolated data x. ex Smoothing is performed as follows:

[0027]

[0028] Where delta(i′,j′) is the Kronecker delta function, p is a parameter that controls the smoothness, and d is the time interval between adjacent frames;

[0029] Furthermore, the contrastive learning model includes a first query encoder, a second query encoder, a third query encoder, and an MLP projection head g(.) set after each query encoder;

[0030] It also includes a first key value encoder and a first momentum encoder;

[0031] The first query encoder, the second query encoder, and the third query encoder are ST-GCN query encoders;

[0032] The original sample x, the sample augmented data x en , and the extrapolated augmented data x ex are input into the first query encoder, the second query encoder, and the third query encoder respectively, and the first query encoder, the second query encoder, and the third query encoder output the original sample x, the sample augmented data x en , and the extrapolated augmented data x ex respectively. en , z ex ; the representation is extracted by the query encoder and the mlp projection head g(.), wherein the query encoder and the mlp projection head g(.) are two key components for extracting a feature vector and mapping the feature vector to a low-dimensional space, so as to calculate a similarity and perform subsequent contrastive learning.

[0033] That is, the first query encoder outputs the encoded feature z

[0034] The second query encoder outputs the encoded feature z en .

[0035] The third query encoder outputs the encoded feature z ex .

[0036] Then, after normalization processing:

[0037] z = F(x),

[0038] z en = F(x en ),

[0039] In the above formula, F(.) includes a query encoder and an mlp projection head, that is, F(.) = g(f(.)), wherein f(.) represents the query encoder.

[0040] The encoded feature z and the encoded feature z en are interpolated to define the interpolated value z in :

[0041] z in = λ * z + (1 - λ) * z en = λ * F(x) + (1 - λ) * F(x en )

[0042] In the formula, λ is a Beta distribution.

[0043] The positive sample pair [zin ,z ex ](extrapolation,interpolation);

[0044] The original sample is input into the first key-value encoder to obtain an encoded key z ak , and the first key-value encoder is updated in parameters by the first momentum encoder

[0045] Further, the contrast learning model further comprises a second key-value encoder, a second momentum encoder and a fourth query encoder.

[0046] The audio data is input into the second key-value encoder, and the output of the second key-value encoder is input into the second momentum encoder to obtain an encoded key z sk .

[0047] The audio data is input into the fourth query encoder to obtain an encoded query z sq of the audio.

[0048] The present application uses MoCo.V2 as a contrast learning model for cross-modal contrast learning. MoCo.V2 is a contrast learning framework using a memory caching mechanism and a dynamic negative sample generation strategy. In contrast learning, each action and audio segment collected by VR / AR generates a pair of query and key, and contrast learning is achieved by comparing the similarity of the query and the key. In this process, the query is a feature vector extracted from data-enhanced data, while the key is a feature vector obtained by feature extraction and caching processing, and is uniformly stored in a dynamically updated memory bank. In the present application, a hand action encoding query (a, action; q, query) is first given in and an encoded key (k, key), as well as an audio encoding query (s, sound; q, query) and an encoded key

[0049] Further, the gradient stop encoding feature z ex is operated, and the consistency constraint is expressed as:

[0050] lossC=sim(z in ,stopgrad(z ex ))

[0051] Further, the batch of encoded keys and Embedded separately in two dynamically updated memory banks, the size of which is adjusted according to the amount of data. Within the memory banks, new encoding keys... and The first key to enter the memory is the encoded key, which is then removed to reduce redundant calculations. and and As two pairs of positive samples and Samples in the memory bank are used as negative samples for training.

[0052] Referring to the contrastive loss function in MoCo.V2[2], the contrastive loss function of the cross-model infoNce of this invention is designed and written as follows:

[0053]

[0054] Where τ is the temperature hyperparameter, and the dot product is used for calculation. and For similarity of positive sample pairs, click and The similarity of negative sample pairs is used to calculate the similarity of positive sample pairs. The contrastive loss function aims to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs.

[0055] After applying the contrastive loss function in equation (18), the encoders corresponding to the query and key—the query encoder and the key encoder—need to be updated accordingly. The query encoder updates its parameters using gradients, while the key encoder updates using the moving average of the query encoder, i.e., momentum updates. The parameters of the query encoder are set to θ. q Set the parameters of the key-value encoder to θ k Therefore, the parameter update formula for the key-value encoder is:

[0056] θ k ←mθ k +(1-m)θ q

[0057] Where m∈[0,1] is the momentum coefficient, which is generally chosen as m=0.999, indicating that the key-value encoder will update slowly, which will ensure the consistency of the key-value representation in the memory.

[0058] Furthermore, each encoded key value stored in the memory is clustered. By calculating the cluster to which the encoded key value belongs and comparing it with the cluster to which the key value belongs in the memory, a Boolean mask is obtained.

[0059] Specifically, each encoded query q has a corresponding encoded key value k as a positive sample;

[0060] Set memory bank M a and memory bank M s Two clustering memories are used to store hand motion data and audio data, respectively. Clustering is performed on the two memory databases and each key-value pair k. Two clustering labels are generated for each key-value pair and each key-value pair k. The clustering labels for key-value pair k are compared sequentially with the clustering labels in the memory databases, and identical labels are masked. The contrastive loss function based on the clustering mask is constructed as follows:

[0061]

[0062] Where, N B This indicates the total number of samples collected; The query encoding feature representing the i″-th interpolation value; The m represents the query encoding feature of the i″-th audio data; j″ M represents the j″-th data in the memory bank; s and M a Two memory banks are maintained to store negative samples.

[0063] Furthermore, the framework of this invention uses clustering to reconstruct the definition of negative samples. However, additional standards are needed to quantify the performance of clustering. If the clustering results differ too much from the true data distribution, true negative pairs will need to be eliminated, which will severely impact the pre-training performance of the model. Therefore, this invention argues that in clustering methods, samples closer to the cluster boundary are less stable in their classification, while samples closer to the cluster center are more likely to belong to the same class. Therefore, this invention designs a hyperparameter β to adjust the negative sample reconstruction scheme. Specifically, the hyperparameter β represents the proportion of samples close to the cluster center. Samples within this proportion are reconstructed using the negative sample scheme, while samples outside the negative sample category are reconstructed using an instance-based judgment scheme. Therefore, the strategy proposed in this invention uses L... info-cluster The contrastive loss function L in the MoCo.V2 framework info This invention can be adjusted multiple times to find the optimal cluster boundary. Therefore, the contrast function loss of this invention can be written as:

[0064]

[0065]

[0066] Where thr(β,i″) represents a value of 1 if sample i″ is within the cluster center β, and 0 otherwise. When β is small, each cluster will receive fewer samples. The opposite is true when β increases. When β is 0, the model's contrastive loss function degenerates into the standard MoCo.V2 loss, and when β is 1, the model degenerates into the negative sample reconstruction loss proposed in this invention.

[0067] Traditional contrastive learning, especially when using strong enhancements to augment positive samples, suffers significant loss of semantic information, particularly for more nuanced hand skeletons. Furthermore, erroneous hand movements appear only briefly in the entire sample, making them difficult to capture accurately using contrastive learning methods. To address this issue, this invention attempts to amplify erroneous movements to augment the samples, thereby capturing hand errors more accurately. Additionally, in the contrastive learning process of selecting positive and negative samples, instances are treated as negations of each other, distancing themselves from each other during training, implying potential over-segregation of instances of the same class. The innovative aspects of this invention address these two problems as follows:

[0068] [1]. Interpolation-based methods construct more reliable positive sample pairs. By interpolating the action sequence, the duration of erroneous actions is lengthened, making it easier to capture them. Furthermore, both intrinsic and extrapolation methods are used, with constraints applied to ensure consistency in feature representation between the two methods.

[0069] [2]. A method for reconstructing negative samples is designed, which treats samples from different classes in a cluster as negative samples. A new contrastive loss function is proposed to mask instances in the same cluster, and instance weights are designed to avoid interference from invalid samples on contrastive learning. Attached Figure Description

[0070] Figure 1 This is the internal and external interpolation model of the present invention;

[0071] Figure 2 This is the overall network model of the present invention;

[0072] Figure 3 This is a schematic diagram of the clustering confidence assessment of the present invention;

[0073] Figure 4 This is a flowchart of the learning phase of a specific embodiment of the present invention;

[0074] Figure 5 This is a flowchart of the practical stages of a specific embodiment of the present invention. Specific Implementation

[0075] The following is a detailed explanation. Figures 1-5 The specific embodiments of the present invention will be further explained below.

[0076] In summary, this invention is based on contrastive learning for skeleton action recognition. Addressing the shortcomings of existing contrastive learning algorithms that do not consider negative sample construction, it proposes a contrastive learning algorithm based on interpolation-based negative sample reconstruction. The invention uses the classic ST-GCN network, the most widely used network in skeleton recognition, as the encoder, and combines it with the momentum coding framework of the contrastive learning MoCo.V2 to guide network training, ultimately achieving efficient skeleton recognition.

[0077] In the learning process, this invention first uses AR / VR technology to collect the hand movements and audio recordings of performers. Specifically, AR / VR cameras record audio data from multiple performers playing different instruments, ensuring that the collected audio includes both completely correct playing movements and audio, as well as incompletely correct playing movements and audio. The incompletely correct playing movements and audio are caused by the different playing habits of multiple performers. For later verification, the collected data needs to be detailedly labeled, including hand movement data with incorrect finger movements and abnormal notes in the audio data. Finally, the collected data is fused at the data level using time-aligned fusion, aligning the collected hand movement and audio data according to time to obtain a multi-channel dataset. Finally, cross-modal comparative learning is performed, and the trained model is packaged onto a VR device.

[0078] During use, the performer wears VR glasses, and the system records the performer's hand movements and audio data in real time, comparing and learning from the original database to determine the correctness of the performance. If the performer makes a similar error to the one in the database, such as missing a string or playing the wrong string, the error recognition system will display an error message in the glasses and show the correct playing method. For example, if the performer misses a string, the glasses will immediately display an error message indicating which string was not pressed in time. Furthermore, the augmented reality system in the glasses will display the performer's hand movements in real time, showing the correct playing action when incorrect movements occur. Depending on the user's needs, the system allows for self-correction during practice or the use of the error recognition method during performance, helping to improve the performer's playing skills.

[0079] The present invention provides a method for error action recognition in VR / AR devices, comprising two main stages: a learning stage and a practical stage. The learning stage includes steps such as collecting learning data, extracting and clustering features from the learning data, and training the model. The practical stage includes steps such as collecting practical data, extracting features from the practical data, and updating the data based on feedback.

[0080] Step 1, Learning Phase

[0081] like Figure 4As shown, during the learning phase, VRAR is used to collect the player's hand movements and audio data. Hand movements of the same error type are matched with their corresponding audio data and saved together for subsequent data analysis and model training. The error types include hand error types and audio error types; hand error types include missing strings, wrong strings, inaccurate finger pressing, excessive finger bending, uneven finger pressing force, and unstable playing rhythm; audio error types include pitch error, rhythm deviation, timbre error, and noise.

[0082] Then, perform normal enhancement on the hand motion data, construct the conditions for extrapolation, and perform the extrapolation operation.

[0083] Then, a Markov chain is used to smooth the interpolated sequence.

[0084] Then, the hand motion data obtained from extrapolation is combined with the original samples and sample augmentation data for feature extraction. To achieve efficient feature comparison and matching, this invention uses a multilayer perceptron (MLP) projector to map the extracted feature vectors into a low-dimensional space. Simultaneously, interpolation is performed to extract meaningful feature representations.

[0085] Then, the obtained interpolated data is subjected to consistency constraint processing.

[0086] Then, during the comparative learning process, the feature vector set in the memory is clustered to cover samples of the same type.

[0087] Then, assess the cluster quality and set the cluster range.

[0088] Then, the contrastive learning model is trained, and the trained contrastive learning model is used in the practical stage.

[0089] Its specific content is as follows:

[0090] Step 1.1 Learning Data Acquisition

[0091] The main task of this step is to collect the performer's hand movements and audio data. This is primarily implemented on VR / AR. First, the invention deploys a music keyboard or other stringed instrument in virtual space, allowing the user to perform in the virtual environment. Simultaneously, assuming the VR / AR's size and spacing are suitable for the user, and the center of the VR / AR's two lenses aligns with the center of the user's eyes, the origin O of the real coordinate system is defined as the center point of the line connecting the centers of the two eyes, parallel to the face plane. The X-axis of the real coordinate system extends from the origin O to the right eye, parallel to the face plane and perpendicularly downwards from the origin O, forming the Y-axis. The Z-axis extends perpendicularly from the origin O towards the eyes. Using the open-source library OpenPose, which can recognize hand postures, the performer uses the camera in the VR / AR glasses to extract hand data, including palm length and finger length. This data matches the instrument key positions in the virtual space with the hand positions. The VR / AR camera continuously captures hand video, collecting hand movements. Simultaneously, the microphone on the VR / AR records the audio of the music played, collecting audio data. This invention requires multiple performers to use the same hand position and movement to perform multiple times during the performance, in order to extract audio data of different pitches under the same movement conditions.

[0092] We obtain the audio and video datasets HM(b,n) and AS(b,n) for playing errors, where HM(b,n) is the hand movement dataset and AS(b,n) is the corresponding audio dataset, including HM... a (b1,n) and AS a (b1,n), HM b (b2,n) and AS b (b2,n), where b∈{b1,b2}.

[0093] HM a (b1,n) is the set of hand movements of the b1th type of error performed by the nth performer, AS a (b1,n) is the collection of audio signals corresponding to the b1th type of hand movement error by the nth performer;

[0094] AS b (b2,n) is the set of audio recordings containing the b2th error type from the nth performer, HM b (b2,n) is the set of actions corresponding to the b2th type of audio error for the nth performer.

[0095] There were a total of 50 performers, and a total of 20 types of hand movement errors. a (b1,n) and AS a(b1,n) A total of 1000 videos and corresponding audio recordings containing incorrect hand gestures were collected. The video format was AVI, and the audio format was MP3. There were a total of 15 types of audio errors. b (b2,n) and HM b (b2,n) A total of 750 data points containing erroneous audio and corresponding action videos were collected. The video storage format is AVI and the audio storage format is MP3.

[0096] Step 1.2 Learning Data Processing

[0097] Data augmentation and interpolation are performed on the audio-visual dataset of playing errors obtained by the performer in step 1.1. Prior to this, data of poor quality, such as incomplete hand movements or high audio noise, are filtered out to ensure that the collected data meets the standards required for the research. Then, data processing steps such as interpolation, feature extraction, etc., are performed on the collected data.

[0098] Step 1.2.1 Enhancement and extrapolation of the original samples

[0099] The sample x in the original action dataset HM(i,n) is flipped, rotated, and randomly occluded to obtain sample augmentation data x. en .

[0100] The extrapolation includes two steps: intra-frame interpolation and inter-frame interpolation.

[0101] (1) Perform intra-frame interpolation:

[0102] Combine sample x and sample augmentation data x en Frame-by-frame fusion is performed; after fusion, inter-frame interpolation is performed to expand the length of the fused data to m times the original. The original samples of this invention are uniformly sampled to a fixed length of 50 frames.

[0103] The interpolated frame sequence M generated by interpolation:

[0104]

[0105] M is . Where N is the temporal length of the original sample x, set to 50, and i represents the i-th frame.

[0106] (2) Use the intra-frame interpolation generation sequence M to perform inter-frame interpolation, that is, generate several frames between frames to expand the data, and finally obtain extrapolated enhanced data.

[0107] Specifically, inter-frame interpolation is performed on sequence M in the time domain:

[0108]

[0109] This invention maps the i-th frame of sequence M to extrapolation enhancement data x. ex In the m*i-th frame, the interpolated frames between the i-th and i+1-th frames are sequentially mapped to the m*i+1, m*i+2, ..., m*i+m-1-th frames, where j = m-1, and the extrapolation is finally represented as x. ex

[0110] Step 1.2.2: Smoothing the interpolated sequence using a Markov chain.

[0111] Since the interpolation method used in this invention generates new data that conforms to the chronological order, to reduce the dual noise introduced by interpolation and extrapolation, this invention uses a first-order Markov chain to describe the state transitions of the skeleton data. Before entering the encoder, the feature vectors are smoothed to reduce noise in the feature vectors. This invention uses the feature vector at the current time step as the current state and the feature vector at the previous time step as the previous state to smooth the generated sequence. The implementation is as follows:

[0112]

[0113] P[i′,j′] is the constructed transition probability matrix, representing the transition probability from state i′ to state j′. p is a parameter controlling the smoothness, taking a value of 0.5. d is the distance between states, set to 1 for the time interval between adjacent frames, representing the average distance between states. This ensures that the distance between states is consistent after interpolation. abs(i′-j′) is the distance between i′ and j′. This represents the relative distance between i′ and j′. delta(i′,j′) is the Kronecker delta function, where (1-p)*delta(i′,j′) represents the transition probability of state i′ transitioning to itself, which is 1 when i′=j′ and 0 when i′≠j′. N is the extrapolation augmentation data x. ex The number of frames. Its purpose is to ensure that each state has a certain transition probability, even if it is far away from other states. Therefore, the extrapolated vector x after Markov chain processing ex Each frame is obtained by calculating the transition probability of all other data points to that frame and adding the interpolation product of all other data points, resulting in a smoother frame. The sequence processed by the Markov chain will have better smoothness; this smoothness and stability can mitigate data fluctuations and noise generated by the interpolation method, thereby improving data quality and robustness. The processed x... ex and x, x en Perform feature extraction;

[0114] When the feature vectors obtained by this invention are processed by Markov chains, they exhibit better smoothness and stability, reducing data fluctuations and noise generated by interpolation methods. Then, this invention learns the representational consistency between adjacent branches by optimizing the similarity between two feature vectors. Since these two views inherently share more information than typical views, the model allows the target features to converge more gently or smoothly to potential cluster centers. It should be noted that this invention only performs inter-frame operations on extrapolation; for intrapolation, it only performs linear hybrid interpolation along the feature vector dimension. Compared to intrapolation, the feature vectors generated by extrapolation always have the same feature dimensions as the intrapolated vectors but contain more spatial and temporal information.

[0115] Step 1.2.3 Learning Data Feature Extraction and Interpolation

[0116] like Figure 1 As shown, the original sample x and the sample augmentation data x are... en and extrapolation augmentation data x ex The data are fed into the first ST-GCN query encoder, the second ST-GCN query encoder, and the third ST-GCN query encoder, respectively. Each encoder is followed by an MLP projection head g(.). After passing through two layers of MLP projection heads g(.), the encoded features z and z' are obtained. en z ex And for z and z en Normalization is performed:

[0117]

[0118] z ex =F(x) ex )

[0119] F(.) contains an ST-GCN encoder f(.) and an MLP projection head g(.).

[0120] The ST-GCN encoder is a deep learning model for skeleton action recognition that fully utilizes spatiotemporal information to capture the spatiotemporal relationships in action sequences. The ST-GCN encoder is pre-trained on the NTU-RGB 120 human skeleton dataset for feature extraction in subsequent tasks. The input to the ST-GCN encoder is skeleton data x, x', ... en and x ex The training batch size is (N, C, T, V, M), where N is the training batch size, C is the number of channels, T is the duration of the skeleton data, V is the number of skeleton joints, and M is the number of people appearing in the skeleton data. The output is (N, C), where N is the training batch size and C is the number of channels after feature extraction.

[0121] The MLP projector is a subsequent layer used for feature extraction. It consists of fully connected layers and its function is to map high-dimensional feature vectors to a low-dimensional space while retaining important feature information. Here, two MLP projectors are used, which means an MLP network composed of two fully connected layers.

[0122] The interpolation value is defined as z in this invention. in :

[0123] z in =λ*z+(1-λ)*z en =λ*F(x)+(1-λ)*F(x) en (7)

[0124] In the formula, λ represents the distribution of Beta. After steps 1.2.1, 1.2.2, and 1.2.3, the following is obtained: Figure 1 The positive sample pair shown is [z] in ,z ex ],

[0125] Step 1.2.4 Optimize the positive sample pairs and constrain the consistency of internal and external interpolation data.

[0126] Optimize positive sample pairs [z] in ,z ex The similarity between positive samples is used to learn the representation consistency between adjacent branches. Since positive sample pairs are extracted from the same view, they share more information. Furthermore, the encoded feature z... ex Compared to the interpolation sequence z in It contains more spatial and temporal information. This invention will utilize the gradient stopping encoding feature z. ex The operation is for the interpolation sequence z. in Choose a more trustworthy target. The consistency constraint can be expressed as:

[0127] lossC = sim(z) in ,stopgrad(z ex (8)

[0128] The formula fixes the coding feature z. ex This forces the interpolation sequence z in Reducing and encoding feature z ex Distance helps the target features converge more gently or smoothly to the potential cluster centers during subsequent clustering processes.

[0129] In this invention, the cosine similarity is chosen for the sim function. The stopgrad operation is used to select a more reliable target for similarity learning. By fixing the features of the target sample, the interference of features from other samples on the similarity calculation is avoided, thus improving model performance. The extrapolation branch Z... ex Constrained and reduced with the interpolation branch Z in The feature distance. This invention obtains positive sample pairs by interpolating two inputs and two features of the interpolated inputs, and each positive sample pair contains semantic information of the original sample and weakly augmented data. Papers have demonstrated that interpolation is effective in pushing the decision boundary away from the class boundary. This invention can add a conditional distribution based on two feature vectors in step 4.

[0130] Step 1.3 Contrastive learning based on negative sample reconstruction

[0131] This invention references the original MoCo.V2 contrastive learning framework in the contrastive learning part of negative sample reconstruction, maintaining the momentum encoder through momentum updates. For example... Figure 2 As shown, the present invention inputs enhanced hand movements and audio data into a contrastive learning model.

[0132] First, the hand motion encoding query z is obtained by querying the encoder and key-value encoder. in Use interpolation encoding to query z aq and encoding key z ak ; and query the encoder and key-value encoder to obtain the audio encoding query z sq and encoding key z sk .

[0133] The cross-modal contrastive loss function under the MoCo.V2 contrastive learning framework can be written as follows:

[0134]

[0135] Among them, small batches of data are used. x i″ and y i″ These represent the hand gestures and audio data after being added in small batches. The batch size in this invention is chosen to be 128, the momentum coefficient m = 0.999, and the temperature coefficient τ = 0.07. As can be seen from the contrast loss function, within a batch... Only for As a positive sample, Only for If a sample is considered positive, then similar samples will be treated as negative samples. (From the appendix...) Figure 3As can be seen, when a positive sample pair is selected, other negative sample pairs may contain samples of the same type as the positive sample, which are pushed aside. To solve this problem, this invention designs a method for feature representation clustering before sending the contrastive learning channel. This method performs clustering representation on the memory Q and each input encoded key value. By calculating the cluster to which the encoded key value belongs and comparing it with the clusters to which all key values ​​stored in memory Q belong, a Boolean mask is obtained. This invention assigns a large value to this mask, so that the contribution of samples of the same type to the similarity of negative samples is set to 0, meaning that samples with the same label no longer form negative pairs.

[0136] Specifically, this invention utilizes the fact that each query vector q has a corresponding key value k as a positive sample, and these keys have similar distributions. Each key value k is stored in a memory bank Q, which includes Q1 and Q2. Two memories, Q1 and Q2, are maintained to store hand movement data and audio data, respectively. The model then performs clustering on both memories and each key value k. Two cluster labels are generated for each memory and each key value k. The cluster labels for key value k are compared sequentially with the cluster labels in the memories, and identical labels are masked using a large constant 1e9. This minimizes the impact of negative samples that are similar to positive samples on contrastive learning. The contrastive loss function based on cluster masking in this invention is constructed as follows:

[0137]

[0138] Step 1.4 Clustering Validation of Negative Sample Reconstruction

[0139] Validating the performance of clustering is also a crucial process. If the clustering results deviate significantly from the true data distribution, true negative pairs will need to be eliminated, severely impacting the model's pre-training performance. Therefore, it is believed that in clustering methods, samples closer to the cluster boundary are less stable in their classification, while samples closer to the cluster center are more likely to belong to the same class. Furthermore, within each cluster set in the sample space, there exist false negative samples that cluster due to spatial proximity but do not match semantic information. Therefore, this invention chooses a nearest neighbor-based clustering method, fully utilizing the semantic information in the distance metric. In the nearest neighbor method, a representative point is used to represent the nominal cluster center of each cluster. Thus, this invention proposes a hyperparameter β to adjust the negative sample reconstruction scheme. Specifically, the hyperparameter β represents the proportion of samples close to the representative point; samples within this proportion are reconstructed using a negative sample scheme, while samples outside the negative sample proportion use an instance-based judgment scheme. Therefore, the proposed strategy uses L... info-a-cluster / L info-a-cluster MoCo.V2 uses normal L info-a / L info-b The optimal clustering boundary can be found through multiple adjustments. Therefore, the contrastive function loss of this invention can be written as:

[0140]

[0141] Where thr(β,i″) represents a value of 1 if sample i″ is within the representative point β, and 0 otherwise. For example... Figure 3 As shown, when β is small, each cluster will receive fewer samples, and vice versa when β increases. The model was optimized by adjusting β multiple times, and the model parameters were then saved.

[0142] Step 2 Practical Stage

[0143] like Figure 5 As shown, the practical application stage includes steps such as practical data collection, feature extraction of practical data, and feedback updates.

[0144] Step 2.1 Practical Data Collection

[0145] During the actual data acquisition phase, the VR / AR camera needs to continuously collect hand movement and audio data, extracting data containing individual hand errors for subsequent analysis and processing. To ensure data quality and accuracy, the data length is maintained between 5 and 8 seconds to include as much hand error information as possible. After data acquisition, the hand movement and audio data need to be aligned and manually labeled for further analysis and processing. To facilitate processing, all data is sampled, and the temporal length of all data is fixed at 50 frames. This effectively reduces the difficulty and complexity of data processing while ensuring data reliability and accuracy.

[0146] Step 2.2 Feature extraction and feedback update of practical data

[0147] The processed data is input into step 1.2 to obtain positive sample pairs of interpolation [z]. in ,z ex Then, the processed practical data is fed into the pre-trained contrastive learning model for training, and the parameters of the query encoder and key-value encoder are updated according to step 1.3. The latest contrastive learning model parameters are saved into the model to ensure the reusability of the model in the future.

[0148] Through the aforementioned work, this invention combines interpolation methods and contrastive learning to achieve contrastive learning across two different data modalities: skeletal motion and audio. The algorithm effectively identifies incorrect playing habits and automatically adjusts the pitch, achieving good results. This algorithm has the potential for practical application, helping players cultivate good playing habits and improve performance quality.

Claims

1. A method of identifying a wrong movement in playing a musical instrument, characterized by, Comprising the following steps: Step 1, collect the audio and video original data set of playing wrong, each original sample x includes hand action data and corresponding audio data; Step 2, data augmentation is carried out on the original sample, specifically comprising the following: Flipping, rotating and random masking on original sample x to get sample augmented data x en ; reinforcing data x of the sample again en extrapolation value increase, to obtain extrapolation value reinforcing data x ex ; The extrapolation value increase includes two steps, intra-frame interpolation and inter-frame interpolation: (1) Intra-frame interpolation: The original sample x and the sample enhancement data x en Frame-by-frame fusion is performed; after fusion, inter-frame interpolation is performed, and the length of the fusion data is expanded to m times the original length; The intra-frame interpolation sequence M generated by intra-frame interpolation is: Wherein, N is the time sequence length of the original sample x, which is set to 50, and i represents the i-th frame; (2) Inter-frame interpolation is carried out using the intra-frame interpolation generated sequence M, that is, several frames are generated between frames to expand the data, and finally the extrapolation value enhanced data is obtained; Specifically, inter-frame interpolation in time domain is carried out on the sequence M: The i-th frame of the sequence M is mapped to the extrapolation value enhancement data x ex In the m*i-th frame of the sequence M, for the interpolation frames between the i-th frame and the i+1-th frame are mapped to the m*i+1, m*i+2, …, m*i+m-1 in turn, where j = m-1; Step 3, constructing a contrast learning model; the original sample and the increased data are input into the contrast learning model for feature extraction, and at the same time, the interpolation value operation is realized to obtain a positive sample pair; Step 4, input the original data set and the data set after data augmentation into the contrast learning model, train the model, and obtain the trained contrast learning model.

2. The method of claim 1, wherein the method further comprises: using a first order Markov chain to describe the state transition of the skeleton data to the generated extrapolated value enhanced data x ex performing smoothing as follows: Wherein, P[i',j'] is the transition probability matrix constructed, representing the transition probability from state i' to state j', delta(i',j') is the Kronecker delta function, p is a parameter for controlling the smoothness, and d is the time interval between adjacent frames.

3. The method of claim 1, wherein the method further comprises: The contrast learning model comprises a first query encoder, a second query encoder, a third query encoder, and an MLP projection head g(.) arranged after each query encoder; Further comprising a first key-value encoder and a first momentum encoder; The first query encoder, the second query encoder and the third query encoder are ST-GCN query encoders; The original sample x, the sample enhanced data x en and the extrapolation enhanced data x ex are input into the first query encoder, the second query encoder and the third query encoder respectively, and the first query encoder, the second query encoder and the third query encoder output the original sample x, the sample enhanced data x en and the extrapolation enhanced data x ex corresponding encoding features z, z en , z ex respectively. for the coding features z and z en interpolating the interpolated value z in : z in = λ * z + (1 - λ) * z en = λ * F(x) + (1 - λ) * F(x en ) Wherein, lambda is the distribution of Beta, F(.) contains a query encoder and an MLP projection head, that is, F(.)=g(f(.)), wherein f(.) represents the query encoder; obtaining positive sample pairs [z in ,z ex ] The original sample x is input into the first key-value encoder to obtain an encoded key z ak The first key-value encoder is then updated by the first momentum encoder.

4. The method of claim 3, wherein the step of identifying the error in the playing of the musical piece is performed by a computer program. The contrast learning model further comprises a second key-value encoder, a second momentum encoder, a fourth query encoder, and an MLP projection head g(.) arranged after the third query encoder and the fourth query encoder; The audio data is input to the second key-value encoder, and the output of the second key-value encoder is input to the second momentum encoder; an encoded key z is obtained sk ; The audio data is input to the fourth query encoder to obtain an encoded query z of the audio sq .

5. The method of claim 3, wherein the step of identifying the error in the playing of the musical piece is further defined as: Encoding features z with gradient stop ex Operation, the consistency constraint is expressed as: ​ lossC = sim(z in , stopgrad(z ex )). lossC = sim(z in , stopgrad(z ex )). lossC 6. The method of claim 3, wherein the step of identifying the error in the playing of the musical piece is further defined as: The cross-modal contrast loss function in the contrast learning is written as: ​ where N B represents the total number of samples collected; represents the query code feature of the i" interpolation value; represents the query code feature of the i" audio data; m j″ represents the j" data in the memory; M s and M a are two maintained memories for storing negative samples, and τ represents a temperature coefficient.

7. The method of claim 3, wherein the step of identifying the error in the playing of the musical piece is further defined as: Each encoded query q has a corresponding encoded key-value k as a positive sample; ​ Memory bank M is set a and memory bank M s Respectively used to store hand action data and audio data, respectively, the two memory banks and each key value k are clustered, the memory bank and each key value k generate two cluster labels respectively, and the cluster labels of the key value k are compared in turn in the cluster labels of the memory bank, and the same labels are masked.

8. The method of claim 7, wherein the step of identifying the error in the performance of the musical piece is further defined as: The contrast loss function based on the clustering mask is constructed as: ​

Images