A system and method for capturing and identifying acupuncture operations

By converting the first-person view posture features into third-person view posture features and integrating them, the fine-grained analysis problem of acupuncture techniques evaluation in the prior art is solved, and the high accuracy evaluation of acupuncture operations is achieved, which is suitable for real and virtual Training of acupuncture techniques in the environment.

CN118658208BActive Publication Date: 2024-11-26BEIJING ANDING HOSPITAL CAPITAL MEDICAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411036694.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-26
Estimated Expiration
2044-07-31

AI Technical Summary

Technical Problem

The existing technology lacks a standardized acupuncture technique evaluation system. Sensor technology affects the experience of the technique and is costly. Third-person perspective analysis cannot perform fine-grained analysis, and cannot accurately evaluate acupuncture techniques in real and virtual environments.

Method used

A system for capturing and identifying needle-punching operations is designed, and posture features are extracted from the first person view through feature extraction units and converted into third person view posture features. After fusing, it is used to predict action quality evaluation and use space-time to guide cross-views. The conversion module and the cross-view space-time attention module perform feature conversion and matching.

Benefits of technology

It effectively reduces the obstruction of the back of the hand to the finger movement, enhances the fine-grained analysis ability of acupuncture operations, improves the accuracy of the movement quality evaluation of the acupuncture techniques, and can better capture the details of the movement quality in traditional Chinese medicine physical therapy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118658208B_ABST
    Figure CN118658208B_ABST
Patent Text Reader

Abstract

A system and method for capturing and identifying acupuncture operations, wherein a feature extraction unit extracts first-person view postures from a first-person view and projects the first-person view postures to latent space features to obtain first-person view posture features; a feature conversion unit converts the first-person view posture features into third-person view posture features; a feature fusion unit fuses the first-person view posture features with the third-person view posture features to obtain posture fusion features; and a prediction and evaluation unit predicts an action quality evaluation score based on the posture fusion features. The present invention effectively reduces the occlusion of the back of the hand on the actual finger action, thereby displaying more information required for effective first-person view action quality evaluation scores in traditional Chinese medicine physical therapy. At the same time, the ability to capture fine-grained details and reduce occlusion effects is enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of gesture recognition, and in particular to a system and method for capturing and recognizing acupuncture operations. Background Art

[0002] Acupuncture therapy can be used as an auxiliary intervention for a variety of diseases, and its efficacy is closely related to acupuncture techniques. However, there is still a lack of standardized evaluation systems for acupuncture techniques. Existing systems mainly rely on sensors or third-person perspectives to analyze movements. Sensor-related technologies will affect the experience of acupuncture techniques and increase costs. There are problems such as the difference between the third-person perspective analysis and the first-person perspective of the therapist in training and operation, and deeper fine-grained analysis cannot be performed. When establishing acupuncture technique training in real and virtual environments in the future, it is necessary to emphasize the accuracy of acupuncture techniques from a self-centered perspective.

[0003] Therefore, due to the limitations of existing technology, people cannot solve it. Summary of the invention

[0004] (I) Purpose of the invention: To solve the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a system and method for capturing and identifying acupuncture operations, which predicts the action quality evaluation of different skill subjects based on the input first-person view posture.

[0005] (II) Technical solution: In order to solve the above technical problems, the present technical solution provides a system for capturing and identifying acupuncture operations, including a feature extraction unit, a feature conversion unit, a feature fusion unit, and a prediction evaluation unit; the feature extraction unit extracts a first-person view posture from a first-person view, and projects the first-person view posture to a latent space feature to obtain a first-person view posture feature;

[0006] The feature conversion unit converts the first-person view posture feature into a third-person view posture feature;

[0007] The feature fusion unit fuses the first-person view posture feature and the third-person view posture feature to obtain a posture fusion feature;

[0008] The prediction and evaluation unit predicts the action quality evaluation score according to the posture fusion feature.

[0009] Furthermore, the feature extraction unit firstly uses a human posture estimation model to extract first-person postures representing hand bone joints frame by frame from the recorded first-person video. , ; Then the first-person view posture is encoded by a two-dimensional encoder and projected into the latent space feature to obtain the first-person view posture feature , ;

[0010] Where T represents the duration of the input first-person view pose sequence, K is the number of hand skeleton joints, express The number of channels in Represents the 2D coordinates of the hand skeleton joints.

[0011] Furthermore, the feature conversion unit includes a spatiotemporal guided cross-view conversion module; the spatiotemporal guided cross-view conversion module uses a space-time converter to convert the first-person view posture feature Convert to third person view pose feature .

[0012] Furthermore, the spatiotemporal guided cross-view transformation module consists of J space-time transformer blocks. For each space-time transformer block j, the following is processed by the self-attention mechanism:

[0013]

[0014] Among them, Q represents the query matrix, K represents the key matrix, and V represents the value matrix. The Q, K and V matrices are input features from the first-person view pose feature The matrices are derived from Q, K, V using a separate learnable weight matrix unique to each space-time transformer block j. Projected represents the dimensions of K keys and Q queries, which helps normalization and keeps the gradient stable during training; T represents the duration of the input first-person view pose sequence.

[0015] Furthermore, the output of each spatio-temporal transformer block j is passed through a first feed-forward neural network, which is applied to each position separately and consists of two linear transformations with a ReLU activation in between:

[0016]

[0017] Among them, x represents the feature vector of the first feedforward neural network; W1 represents the weight matrix for transforming the feature vector in the first layer of the first feedforward neural network; b1 represents the bias term added to the transformed feature vector of the first layer; W2 represents the weight matrix for transforming the feature vector in the second layer of the first feedforward neural network; b2 represents the bias term added to the transformed feature vector of the second layer.

[0018] Furthermore, the feature fusion unit includes a cross-view spatial-temporal attention module; the cross-view spatial-temporal attention module uses a mean square error to integrate the first-person view posture features and third person view pose features Matching is performed to find the first person view pose features and third person view pose features The most relevant feature between the first person view pose features and third person view pose features Perform feature fusion to obtain posture fusion features.

[0019] Furthermore, the mean squared error is used as an auxiliary loss to align the first-person view pose features. and third person view pose features ;

[0020] The most relevant features are identified by computing an attention matrix that quantifies the relevance of each pose feature in the first-person view and the third-person view according to its relationship, i.e., the first-person view pose feature and third person view pose features The attention mechanism highlights the most relevant features by focusing on the highest correlation values, thereby identifying the most relevant features.

[0021] Furthermore, the cross-view spatial-temporal attention module is composed of L cross-view spatial-temporal transformation blocks; for each cross-view spatial-temporal transformation block l, the first-person view posture features are effectively captured through the cross-attention mechanism and third person view pose features The interactions between spaces highlight the most predictive features for accurate quality assessment in action quality evaluation;

[0022] Each output of the cross-attention mechanism is further processed by a second feedforward neural network in each cross-view spatial-temporal transformation block l, acting on the position of the output of each cross-view spatial-temporal transformation block l:

[0023]

[0024] Wherein, x represents the feature vector processed by the second feedforward neural network; W1 represents the weight matrix for transforming the feature vector in the first layer of the second feedforward neural network; b1 represents the bias term added to the transformed feature vector of the first layer; W2 represents the weight matrix for transforming the feature vector in the second layer of the second feedforward neural network; b2 represents the bias term added to the transformed feature vector of the second layer;

[0025] The second feedforward neural network applies nonlinearity to integrate the information processed by the cross-attention mechanism to refine the posture fusion feature , , the posture fusion feature Used for subsequent action quality evaluation tasks.

[0026] Furthermore, the prediction and evaluation unit includes a cross-view enhanced self-centered action quality evaluation module; first, the cross-view enhanced self-centered action quality evaluation module is trained:

[0027]

[0028] in, Represents an estimated third-person view With true third person view The L1 norm between ; Represents the predicted label With the true label The cross entropy between Indicates that the The weight of Indicates that the The weight of .

[0029] A method for capturing and identifying acupuncture operations, applicable to the system for capturing and identifying acupuncture operations, specifically comprises the following steps:

[0030] Step 1: The feature extraction unit extracts a first-person view posture from the first-person view, and projects the first-person view posture to a latent space feature to obtain a first-person view posture feature;

[0031] Step 2, a feature conversion unit converts the first-person view posture feature into a third-person view posture feature;

[0032] Step 3: The feature fusion unit fuses the first-person view posture feature and the third-person view posture feature to obtain a posture fusion feature;

[0033] Step 4: The prediction evaluation unit predicts the action quality evaluation score based on the posture fusion features.

[0034] (III) Beneficial effects: The system and method for capturing and identifying acupuncture operations of the present invention effectively reduces the occlusion of the back of the hand on the actual finger movements by converting the first-person view posture features into the third-person view posture features, thereby displaying more information required for effective first-person view AQA in traditional Chinese medicine physical therapy. Then, by matching and fusing the first-person view posture features and the third-person view posture features, the ability to capture fine-grained details and reduce occlusion effects is enhanced, thereby obtaining the most relevant features to predict AQA. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1It is a structural diagram of a system for capturing and identifying acupuncture operations of the present invention;

[0036] Figure 2 It is a flow chart of a method for capturing and identifying acupuncture operations of the present invention;

[0037] Figure 3 It is a specific flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present invention is further described in detail below in conjunction with preferred embodiments to facilitate a full understanding of the present invention. The following description explains more details. However, the present invention can obviously be implemented in a variety of other ways different from the description. Those skilled in the art can make similar generalizations and deductions based on actual application situations without violating the connotation of the present invention. Therefore, the protection scope of the present invention should not be limited by the content of this specific embodiment.

[0039] The accompanying drawings are schematic diagrams of embodiments of the present invention. It should be noted that the drawings are only examples and are not drawn to scale, and should not be used to limit the actual protection scope of the present invention.

[0040] A system for capturing and identifying acupuncture operations, such as Figure 1 As shown, it includes a feature extraction unit, a feature conversion unit, a feature fusion unit and a prediction and evaluation unit. The feature extraction unit extracts the first-person view posture from the first-person view, and projects the first-person view posture to the latent space feature to obtain the first-person view posture feature. The feature conversion unit converts the first-person view posture feature into the third-person view posture feature. The feature fusion unit fuses the first-person view posture feature and the third-person view posture feature to obtain a posture fusion feature. The prediction and evaluation unit predicts the action quality evaluation score according to the posture fusion feature.

[0041] A method for capturing and identifying acupuncture operations, applicable to a system for capturing and identifying acupuncture operations, such as Figure 2 As shown, the specific steps include:

[0042] Step 1: A feature extraction unit extracts a first-person view posture from a first-person view, and projects the first-person view posture to a latent space feature to obtain a first-person view posture feature.

[0043] Step 2: The feature conversion unit converts the first-person view posture feature into a third-person view posture feature.

[0044] Step 3: The feature fusion unit fuses the first-person view posture feature and the third-person view posture feature to obtain a posture fusion feature.

[0045] Step 4: The prediction evaluation unit predicts the action quality evaluation score based on the posture fusion features.

[0046] More specifically, the specific operation of the feature extraction unit in step 1 is:

[0047] First, the human pose estimation model is used to extract the first-person pose representing the hand bone joints frame by frame from the recorded first-person video. , The first-person view posture is then encoded by a two-dimensional encoder and projected into the latent space feature to obtain the first-person view posture feature , .

[0048] Where T represents the duration of the input first-person view pose sequence, K is the number of hand skeleton joints, express The number of channels in Represents the 2D coordinates of the hand skeleton joints.

[0049] The human body posture estimation model described in the present invention includes but is not limited to Media Pipe, etc., as long as the first-person posture of some bone joints can be extracted.

[0050] More specifically, the specific operation steps of the feature conversion unit in step 2 are:

[0051] The feature conversion unit includes a spatial-temporal guided cross-view translation module (STGCT), which uses a space-time transformer to convert the first-person view posture features Convert to third person view pose feature .

[0052] In the first-person view, the actual finger movements are likely to be occluded by the back of the hand, which significantly reduces the information required for effective first-person view action quality assessment (AQA) in TCM physical therapy. To supplement these missing details occluded by the back of the hand, the spatiotemporal guided cross-view conversion module uses a space-time transformer to convert the first-person perspective posture features into third-person posture features. This effectively models the correlation between posture joints in the first-person view and captures the relationship between the first-person view and the third-person view.

[0053] The spatiotemporal guided cross-view transformation module consists of J space-time transformer blocks, where J is greater than or equal to 2 and less than or equal to 4. For each space-time transformer block j, the following is processed by the self-attention mechanism:

[0054]

[0055] Among them, Q represents the query matrix, K represents the key matrix, and V represents the value matrix. The Q, K and V matrices are input features from the first-person view pose feature The matrices are derived from Q, K, V using separate learnable weight matrices independent of each space-time transformer block j. Projected. represents the dimensions of K keys and Q queries, which helps normalization and keeps the gradient stable during training. T represents the duration of the input first-person view pose sequence.

[0056] The output of each spatio-temporal transformer block j is then passed through a first feed-forward neural network, which is applied to each position separately and consists of two linear transformations with a ReLU activation in between:

[0057]

[0058] Among them, x represents the feature vector of the first feedforward neural network; W1 represents the weight matrix for transforming the feature vector in the first layer of the first feedforward neural network; b1 represents the bias term added to the transformed feature vector of the first layer; W2 represents the weight matrix for transforming the feature vector in the second layer of the first feedforward neural network; b2 represents the bias term added to the transformed feature vector of the second layer.

[0059] By applying the first feedforward neural network to the output of the self-attention layer, the spatial-temporal transformer block is able to model complex relationships in the data. The output of the final spatial-temporal transformer block J is the third-person view pose feature. , , and then add the third person view pose feature Passed into the 2D convolution decoder to reconstruct the third-person view , ,Thereby enhancing the model’s ability to transform and enrich first-person view pose features into a more comprehensive third-person view.

[0060] More specifically, the specific operation steps of the feature fusion unit in step 3 are:

[0061] The feature fusion unit includes a cross-view spatial-temporal attention module (CSTA). The cross-view spatial-temporal attention module uses the mean square error to integrate the first-person view posture features. and third person view pose features Matching is performed to find the first person view pose features and third person view pose features Then the first person view pose feature and third person view pose features Perform feature fusion to obtain posture fusion features.

[0062] The present invention uses mean-square error (MSE) as an auxiliary loss to align first-person view pose features and third person view pose features , minimizing the first person view pose features and third person view pose features The difference between these two feature sets ensures a perfect match between them for subsequent processing.

[0063] The most relevant features are the features that provide the most informative details for action quality evaluation. This is achieved by calculating an attention matrix, which quantifies the relevance of each posture feature in the first-person view and the third-person view according to the relationship between each posture feature, that is, the first-person view posture feature and third person view pose features The attention mechanism highlights the most relevant features by focusing on the highest correlation values, thereby identifying the most relevant features.

[0064] The cross-view spatial-temporal attention module is composed of L cross-view spatial-temporal transformation blocks; the number of the cross-view spatial-temporal transformation blocks is greater than or equal to 2 and less than or equal to 4. It should be noted that L and J need to be set to the same value to keep the 2D encoder and 2D decoder have the same depth to ensure consistent processing and feature extraction of the entire network.

[0065] For each cross-view spatial-temporal transformation block l, it is processed through the cross-attention mechanism:

[0066]

[0067] Among them, Q represents the The query matrix derived from Derived bond matrix; V represents the bond matrix derived from derived value matrix; represents the dimensions of the key and query, the dot product is normalized to stabilize the training; T represents the duration of the input first-person view pose sequence.

[0068] Effectively capture first-person view pose features through the cross-attention mechanism and third person view pose features The interactions between the spaces highlight the most predictive features for accurate quality assessment in AQA.

[0069] Each output of the cross-attention mechanism is further processed by a second feedforward neural network in each cross-view spatial-temporal transformation block l, acting on the position of the output of each cross-view spatial-temporal transformation block l:

[0070]

[0071] Among them, x represents the feature vector processed by the second feedforward neural network; W1 represents the weight matrix for transforming the feature vector in the first layer of the second feedforward neural network; b1 represents the bias term added to the transformed feature vector of the first layer; W2 represents the weight matrix for transforming the feature vector in the second layer of the second feedforward neural network; b2 represents the bias term added to the transformed feature vector of the second layer.

[0072] The second feedforward neural network applies nonlinearity to integrate the information processed by the cross-attention mechanism to refine the posture fusion feature , , the posture fusion feature Used for subsequent action quality evaluation tasks.

[0073] The first feedforward neural network and the second feedforward neural network include but are not limited to a single-layer feedforward neural network, a multi-layer feedforward neural network, a recursive feedforward neural network, and a convolutional feedforward neural network. The first feedforward neural network and the second feedforward neural network may be the same or different, and there is no specific limitation here.

[0074] Obtained by temporal average pooling The global representation of , the 2D convolutional decoder will Processed into M skill subjects to generate the final AQA prediction label The sum of the two-dimensional convolution is the different scores of different subjects of MM. For MM, the present invention obtains the best range of less than or equal to 10 by testing the value not exceeding 10.

[0075] The temporal average pooling is a data processing technique, mainly used for processing time series data. It smoothes the data by averaging the values ​​in adjacent regions, reduces noise, and reveals trends and periodicities in the data. An average pooling layer is used along the time dimension to aggregate the features of each frame. This temporal average pooling calculates the feature mean of each time step, effectively summarizing temporal information and reducing the serial length. This operation helps to minimize the computational complexity of subsequent layers while preserving the basic temporal dynamics of the input features.

[0076] More specifically, the specific operation steps of the prediction evaluation unit in step 4 are:

[0077] The prediction and evaluation unit includes a cross-view enhanced egocentric action quality evaluation module (CE-EAQA). The cross-view enhanced egocentric action quality evaluation module is used to predict and score the posture fusion feature and obtain the action quality evaluation score of the input first-person view posture feature.

[0078] First, the cross-view enhanced ego-centric action quality evaluation module needs to be trained. The training loss of the training consists of two parts:

[0079]

[0080] in, Represents an estimated third-person view With true third person view The L1 norm between ; Represents the predicted label With the true label The cross entropy between Indicates that the The weight of Indicates that the The weight of .

[0081] It should be noted that there is no restriction on the specific values ​​of α and β, and α and β can be adjusted as long as the sum of the two is equal to 1.

[0082] The trained cross-view enhanced ego-centric action quality assessment module performs an estimated assessment on the input first-person view to obtain a predicted action quality assessment AQA of the first-person view.

[0083] like Figure 3 As shown, the following is a detailed description with reference to specific embodiments:

[0084] The original dataset was composed of 61 test data collected by ourselves, and a five-fold crossover test was conducted on the original dataset. The network hyperparameters were configured as J=4, L=4, α=0.5, and β=0.5 on an NVIDIA TITAN RTX 24G GPU. The batch size was set to 4 and the training lasted for 100 epochs. The Adam optimizer was used with a learning rate of , The AQA score is predicted by the system and method for capturing and identifying acupuncture operations described in the present invention.

[0085] The learning rate is a hyperparameter that determines the step size for updating the model weights during training. β1 controls the decay rate of the gradient moving average, β1=0 means no momentum is applied. β2 controls the decay rate of the squared gradient moving average, β2=0.99 gives more weight to recent gradients. β1 and β2 work together in the Adam optimizer to control different aspects: β1 affects momentum, while β2 affects the adaptive learning rate, and they are not directly related to each other.

[0086] The basic idea of ​​the five-fold crossover test is to divide the collected original dataset into five equal-sized subsets (or folds), four of which are used to train the model, while the remaining one is used to test the model. The whole process is repeated five times, each time a different subset is selected as the test set, and the rest are used as the training set. Finally, the five performance evaluation results are averaged to obtain the final evaluation result.

[0087] The present invention uses accuracy and F1 score to evaluate the accuracy of predicting AQA, providing clearer clinical significance. The evaluation performance accuracy of the system and method for capturing and identifying acupuncture operations described in the present invention is represented by the indicators in Table 1: accuracy / F1 score. The F1 score evaluates the accuracy of the model by considering precision and recall. The F1 score is the harmonic mean of precision and recall, providing a single score that balances the two.

[0088] Table 1 Performance of different methods in acupuncture tasks (accuracy / F1 score)

[0089]

[0090] The system and method for capturing and identifying acupuncture manipulations of the present invention are compared with a classic posture-based human action recognition method (STGCN) based on graph neural networks and a long short-term attention method (LSTA) for egocentric video analysis. As shown in the results in Table 1, the system and method for capturing and identifying acupuncture manipulations proposed by the present invention outperform the other two methods in various skill subjects of acupuncture tasks. Specifically, the system and method for capturing and identifying acupuncture manipulations show higher accuracy, better alignment with expert scores, and more accurate action quality assessment.

[0091] The present invention is a system and method for capturing and identifying acupuncture operations. By converting first-person view posture features into third-person view posture features, the occlusion of the back of the hand on the actual finger movements is effectively reduced, thereby displaying more information required for effective first-person view AQA in traditional Chinese medicine physical therapy. Then, by matching and fusing the first-person view posture features and the third-person view posture features, the ability to capture fine-grained details and reduce occlusion effects is enhanced, thereby obtaining the most relevant features to predict AQA.

[0092] The above content is an explanation of the preferred embodiments of the present invention, which can help those skilled in the art to more fully understand the technical solution of the present invention. However, these embodiments are merely illustrative, and it cannot be determined that the specific implementation methods of the present invention are limited to the description of these embodiments. For ordinary technicians in the technical field to which the present invention belongs, without departing from the concept of the present invention, several simple deductions and transformations can be made, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A system for capturing and identifying acupuncture operations, characterized in that: It includes a feature extraction unit, a feature conversion unit, a feature fusion unit and a prediction evaluation unit; the feature extraction unit extracts the first-person view posture from the first-person view, and projects the first-person view posture to the latent space feature to obtain the first-person view posture feature; The feature conversion unit converts the first-person view posture feature into a third-person view posture feature; The feature conversion unit includes a spatiotemporal guided cross-view conversion module; the spatiotemporal guided cross-view conversion module uses a space-time transformer to convert the first-person view posture features Convert to third person view pose feature ; The spatiotemporal guided cross-view transformation module consists of J space-time transformer blocks. For each space-time transformer block j, the following is processed by the self-attention mechanism: ; Among them, Q represents the query matrix, K represents the key matrix, and V represents the value matrix. The Q, K and V matrices are input features from the first-person view pose feature The matrices are derived from Q, K, V using a separate learnable weight matrix unique to each space-time transformer block j. Projected represents the dimension of K keys and Q queries, which helps normalization and keeps the gradient stable during training; T represents the duration of the input first-person view pose sequence; The output of each spatio-temporal transformer block j is passed through a first feed-forward neural network, which is applied to each position separately and consists of two linear transformations with a ReLU activation in between: ; Wherein, x represents the feature vector of the first feedforward neural network; W1 represents the weight matrix of transforming the feature vector in the first layer of the first feedforward neural network; b1 represents the bias term added to the transformed feature vector of the first layer; W2 represents the weight matrix of transforming the feature vector in the second layer of the first feedforward neural network; b2 represents the bias term added to the transformed feature vector of the second layer; The feature fusion unit fuses the first-person view posture feature and the third-person view posture feature to obtain a posture fusion feature; The prediction and evaluation unit predicts the action quality evaluation score according to the posture fusion feature; the prediction and evaluation unit includes a cross-view enhanced self-centered action quality evaluation module; firstly, the cross-view enhanced self-centered action quality evaluation module is trained: ; in, Represents an estimated third-person view With true third person view The L1 norm between ; Represents the predicted label With the true label The cross entropy between Indicates that the The weight of Indicates that the The weight of .

2. A system for capturing and identifying acupuncture operations according to claim 1, characterized in that: The feature extraction unit firstly uses the human posture estimation model to extract the first-person posture representing the hand bone joints frame by frame from the recorded first-person video. , ; Then the first-person view posture is encoded by a two-dimensional encoder and projected into the latent space feature to obtain the first-person view posture feature , ; Where T represents the duration of the input first-person view pose sequence, K is the number of hand skeleton joints, express The number of channels in Represents the 2D coordinates of the hand skeleton joints.

3. A system for capturing and identifying acupuncture operations according to claim 1, characterized in that: The feature fusion unit includes a cross-view spatial-temporal attention module; the cross-view spatial-temporal attention module uses the mean square error to integrate the first-person view posture features and third person view pose features Matching is performed to find the first person view pose features and third person view pose features The most relevant feature between the first person view pose features and third person view pose features Perform feature fusion to obtain posture fusion features.

4. A system for capturing and identifying acupuncture operations according to claim 3, characterized in that: Using mean squared error as auxiliary loss to align first-person view pose features and third person view pose features ; The most relevant features are identified by computing an attention matrix that quantifies the relevance of each pose feature in the first-person view and the third-person view according to its relationship, i.e., the first-person view pose feature and third person view pose features The correlation between The attention mechanism highlights the most relevant features by focusing on the highest correlation values, thereby identifying the most relevant features.

5. A system for capturing and identifying acupuncture operations according to claim 4, characterized in that: The cross-view spatial-temporal attention module consists of L cross-view spatial-temporal transformation blocks; for each cross-view spatial-temporal transformation block l, the cross-attention mechanism effectively captures the first-person view posture features. and third person view pose features The interactions between spaces highlight the most predictive features for accurate quality assessment in action quality evaluation; Each output of the cross-attention mechanism is further processed by a second feedforward neural network in each cross-view spatial-temporal transformation block l, acting on the position of the output of each cross-view spatial-temporal transformation block l: ; Wherein, x represents the feature vector processed by the second feedforward neural network; W1 represents the weight matrix for transforming the feature vector in the first layer of the second feedforward neural network; b1 represents the bias term added to the transformed feature vector of the first layer; W2 represents the weight matrix for transforming the feature vector in the second layer of the second feedforward neural network; b2 represents the bias term added to the transformed feature vector of the second layer; The second feedforward neural network applies nonlinearity to integrate the information processed by the cross-attention mechanism to refine the posture fusion feature , the posture fusion feature Used for subsequent action quality evaluation tasks.

6. A method for capturing and identifying acupuncture operations, applicable to the system for capturing and identifying acupuncture operations, characterized in that: The specific steps include: Step 1: The feature extraction unit extracts a first-person view posture from the first-person view, and projects the first-person view posture to a latent space feature to obtain a first-person view posture feature; Step 2, a feature conversion unit converts the first-person view posture feature into a third-person view posture feature; The feature conversion unit includes a spatiotemporal guided cross-view conversion module; the spatiotemporal guided cross-view conversion module uses a space-time transformer to convert the first-person view posture features Convert to third person view pose feature ; The spatiotemporal guided cross-view transformation module consists of J space-time transformer blocks. For each space-time transformer block j, the following is processed by the self-attention mechanism: ; Among them, Q represents the query matrix, K represents the key matrix, and V represents the value matrix. The Q, K and V matrices are input features from the first-person view pose feature The matrices are derived from Q, K, V using a separate learnable weight matrix unique to each space-time transformer block j. Projected represents the dimension of K keys and Q queries, which helps normalization and keeps the gradient stable during training; T represents the duration of the input first-person view pose sequence; The output of each spatio-temporal transformer block j is passed through a first feed-forward neural network, which is applied to each position separately and consists of two linear transformations with a ReLU activation in between: ; Wherein, x represents the feature vector of the first feedforward neural network; W1 represents the weight matrix of transforming the feature vector in the first layer of the first feedforward neural network; b1 represents the bias term added to the transformed feature vector of the first layer; W2 represents the weight matrix of transforming the feature vector in the second layer of the first feedforward neural network; b2 represents the bias term added to the transformed feature vector of the second layer; Step 3: The feature fusion unit fuses the first-person view posture feature and the third-person view posture feature to obtain a posture fusion feature; Step 4: The prediction and evaluation unit predicts the action quality evaluation score according to the posture fusion feature; the prediction and evaluation unit includes a cross-view enhanced self-centered action quality evaluation module; first, the cross-view enhanced self-centered action quality evaluation module is trained: ; in, Represents an estimated third-person view With true third person view The L1 norm between ; Represents the predicted label With the true label The cross entropy between Indicates that the The weight of Indicates that the The weight of .