A multimodal acupuncture technique recognition method and system integrating vision and touch

Abstract

This invention relates to a multimodal acupuncture technique recognition method and system that integrates visual and tactile senses. The recognition method includes: extracting features from tactile piezoelectric signals and hand spatial signals; then randomly outputting single-signal or multi-signal feature fusion based on the obtained piezoelectric signal features and hand spatial signal features; processing acupuncture image data through multiple independent convolutional channels, while adaptively calibrating the weights of each convolutional channel through squeezing and nonlinear activation operations to obtain image features; performing a tensor product operation on the random output and image features to obtain a tensor outer product; and performing tensor fusion based on the tensor outer product and weight matrices constructed for the image features and random output respectively to obtain a fused tensor, i.e., the fused recognition result. The multimodal fusion scheme provided by this invention can integrate these multiple modalities of information, achieving more comprehensive and accurate acupuncture technique monitoring and recognition.

Description

Technical Field

[0001] This invention relates to the field of data recognition technology, and in particular to a multimodal acupuncture technique recognition method and system that integrates visual and tactile senses. Background Technology

[0002] Acupuncture manipulation is a traditional Chinese medicine treatment method. Currently, the transmission and teaching of acupuncture manipulation is mainly carried out through oral instruction, lacking a unified scheme for quantifying and modeling acupuncture manipulation techniques. Due to the complex hand movements, small range of motion, and difficulty in accurately representing the relative movement information between the fingers during acupuncture manipulation, an effective quantification and identification method is still lacking.

[0003] Even with traditional acupuncture technique parameter instruments, which can quantify the tactile features of the fingers, it is impossible to quantify the movement patterns of the hand during acupuncture from an overall visual perspective. Furthermore, the more expensive method based on 3D motion tracking not only requires setting up multiple cameras simultaneously for gesture tracking, but also can only obtain the relative spatial information of the fingers from a visual perspective, and cannot obtain tactile information such as the pressure and speed between the fingers during acupuncture. Therefore, it is difficult to build a multimodal acupuncture technique recognition scheme that takes into account both visual and tactile features using existing technologies. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a multimodal acupuncture technique recognition method and system that integrates visual and tactile senses, which solves the technical problem that the prior art is unable to recognize acupuncture techniques that take into account both visual and tactile features.

[0006] (II) Technical Solution

[0007] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0008] In a first aspect, embodiments of the present invention provide a multimodal acupuncture technique recognition method that integrates visual and tactile sensations, comprising:

[0009] Feature extraction is performed on the tactile piezoelectric signal and hand spatial signal represented in waveform form, and then random output is performed based on the obtained piezoelectric signal features and hand spatial signal features, either by fusing single or multiple signal features.

[0010] The needle puncture image data, represented in the form of video frames, is processed by multiple independent convolutional channels. Attention mechanisms are applied through squeezing and nonlinear activation operations to adaptively calibrate the weights of each convolutional channel to obtain image features.

[0011] Tensor product operation is performed on the random output result and image features to obtain tensor outer product. Tensor fusion is then performed based on the tensor outer product and the weight matrices constructed for the image features and random output result respectively to obtain the fused tensor, which is the fused recognition result.

[0012] Optionally, before extracting features from the tactile piezoelectric signal and hand spatial signal represented in waveform form, and before performing random output by fusing single-signal or multi-signal features based on the obtained piezoelectric signal features and hand spatial signal features, the method further includes:

[0013] The raw data of the acquired tactile piezoelectric signal and the raw data of the hand spatial signal are preprocessed to obtain the tactile piezoelectric signal and the hand spatial signal.

[0014] The acquired acupuncture technique videos are segmented frame by frame to generate video frame datasets corresponding to different acupuncture techniques.

[0015] By packaging the tactile piezoelectric signal, hand spatial signal, and video frame dataset corresponding to each acupuncture technique, a synthetic array for each acupuncture technique is obtained for subsequent feature extraction.

[0016] Optionally, feature extraction is performed on the tactile piezoelectric signal and hand spatial signal represented in waveform form, and then a random output is obtained by fusing single-signal features or multiple-signal features based on the obtained piezoelectric signal features and hand spatial signal features, including:

[0017] We extracted features from the tactile piezoelectric signal and hand spatial signal, which were represented in waveform form, using CNN and sliding window respectively, to obtain the piezoelectric signal features and hand spatial signal features.

[0018] A random factor is used to output either the piezoelectric signal feature or the hand spatial signal feature, or a random output resulting from the fusion of multiple signal features, including both piezoelectric and hand spatial features.

[0019] The random factor satisfies the following random output condition:

[0020]

[0021] In equation (1), α is the piezoelectric signal feature, β is the hand spatial signal feature, and a is a random factor used to process the tactile piezoelectric signal and hand spatial signal represented in waveform form, and generate a random output result containing one or more of the piezoelectric signal features and hand spatial signal features.

[0022] Optionally, the acupuncture image data represented in video frames is processed by multiple independent convolutional channels, while an attention mechanism is applied through squeezing and nonlinear activation operations to adaptively calibrate the weights of each convolutional channel to obtain image features, including:

[0023] The needle puncture image data, represented in the form of video frames, is processed by convolutional channels that are independent of each other.

[0024] Information on convolutional channel features is generated by squeezing and nonlinear activation of needle puncture image data represented in video frames, and global spatial information is collected to perform adaptive calibration of the weights of each convolutional channel.

[0025] Based on adaptive weights, features are fused from data extracted from different convolutional channels using a concatenation approach.

[0026] Image features are obtained by repeatedly processing the data through independent convolutional channels and applying attention mechanisms.

[0027] When the data is processed by the convolution channel, the needle puncture image data is expanded outward by padding with zeros on the outer layer of the image so that the convolution kernel slides once each time. That is, the stride is set to 1 for each convolution calculation. The operation of the convolution kernel satisfies the following conditions:

[0028]

[0029]

[0030]

[0031] In equations (2)-(4), padding is the padding parameter, which completes the padding of the outer layer of the image with zeros, f is the length of the convolution kernel in each layer, s is the stride of the convolution kernel, h is the length of the input feature map, w is the width of the input feature map, and h out To determine the length of the output feature map, w out This is the width of the output feature map.

[0032] Optionally, information on convolutional channel features is generated by squeezing and nonlinearly activating the needle puncture image data represented in video frames, and global spatial information is collected to perform adaptive calibration of the weights for each convolutional channel, including:

[0033] The acupuncture image data, represented in the form of video frames, is compressed in the corresponding dimensional space using the first compression formula;

[0034] The acupuncture image data, which was compressed initially, is compressed again using a second compression formula;

[0035] The ReLU activation function is used to remove data from the recompressed acupuncture image data that does not meet the preset requirements.

[0036] The activation function stretches the dimensional space of the data after the culling operation back to the original compressed dimensional space, and the sigmoid activation function maps the stretched and restored data into a range of [0,1].

[0037] in,

[0038] The first extrusion formula is:

[0039]

[0040] In equation (5), F sq This refers to the output feature value u c This refers to the compression of the input feature map in the spatial dimension W×H through the compression of the average pooling layer, that is, the input of W×H×C is transformed into the output of 1×1×C;

[0041] The second extrusion formula is:

[0042] x=FC(F sq (u c (6)

[0043] In equation (6), x represents the output of the second compression process, and FC represents the compression process in the spatial dimension C completed by the second compression through the fully connected layer, that is, the input of 1×1×C is converted into the output of 1×1×C / r.

[0044] The ReLU activation function is:

[0045]

[0046] The sigmoid activation function is:

[0047]

[0048] In equation (8), e represents the natural logarithm;

[0049] The activation function is:

[0050] s = F ex (z,W)=Relu(g(z,W))=σ(W2δ(W1z)) (9)

[0051] In equation (9), F exThe mechanism of gates in a recurrent neural network is represented by the parameter W, which generates weights for each feature channel. The parameter W is used to explicitly model the correlation between feature channels. z represents the output of formula (5), g represents the gating mechanism in the form of sigmoid, δ and σ represent the ReLU and sigmoid activation functions, respectively, and w1 and w2 are the parameters obtained by vector decomposition of parameter W. is the number of channels in the image, and r is a hyperparameter for dimensionality reduction.

[0052] Optionally, a tensor product operation is performed on the random output result and image features to obtain a tensor outer product. Then, tensor fusion is performed based on the tensor outer product and the weight matrices constructed for the image features and the random output result, respectively, to obtain a fused tensor. The fused recognition result includes:

[0053] Obtain a one-dimensional tensor of the random output result and a one-dimensional tensor of the image features;

[0054] The one-dimensional tensor of the random output and the one-dimensional tensor of the image features are expanded to increase the tensor by one dimension, resulting in high-dimensional tensor features of the random output and the image features.

[0055] Tensor cross product is obtained by performing tensor product operation based on the high-dimensional tensor features of random output results and image features;

[0056] Construct corresponding weight matrices for the image features and the random output results respectively;

[0057] Based on the tensor outer product, the weight matrix of image features, and the weight matrix of random output results, tensor fusion is performed using the tensor fusion formula to obtain the fused recognition result;

[0058] The tensor fusion formula is as follows:

[0059]

[0060] In equation (10), h is the tensor obtained after final fusion, r is the number of reconstruction matrices, and w a w represents the weights of the tensor of the random output result. v Z represents the weights of the tensor of image features, and Z is the tensor outer product. a The high-dimensional tensor features obtained by augmenting the random output results, z v These are high-dimensional tensor features obtained by expanding image features.

[0061] Secondly, the present invention provides a multimodal acupuncture technique recognition model that integrates visual and tactile senses, including:

[0062] The waveform data feature extraction and processing module is used to extract features from tactile piezoelectric signals and hand spatial signals represented in waveform form, and to randomly output single-signal features or multi-signal feature fusion based on the obtained piezoelectric signal features and hand spatial signal features;

[0063] The visual data feature extraction and processing module is used to process the needle puncture image data represented in the form of video frames through multiple independent convolutional channels. At the same time, it applies an attention mechanism through squeezing and nonlinear activation operations to adaptively calibrate the weights of each convolutional channel to obtain image features.

[0064] The multimodal feature fusion module is used to perform tensor product operation on random output results and image features to obtain tensor outer product, and then perform tensor fusion based on tensor outer product and weight matrices constructed for image features and random output results to obtain fused tensor, i.e. fused recognition result.

[0065] Optionally, the waveform data feature extraction and processing module includes:

[0066] The waveform data preprocessing unit is used to preprocess the raw data of the acquired tactile piezoelectric signal and the raw data of the hand spatial signal to obtain the tactile piezoelectric signal and the hand spatial signal.

[0067] A CNN network is used to extract features from the acquired hand spatial signal, which is represented in waveform form, to obtain hand spatial signal features.

[0068] A sliding window feature extraction network is used to extract features from the acquired tactile piezoelectric signals represented in waveform form to obtain tactile piezoelectric signal features;

[0069] The random block is used to process tactile piezoelectric signals and hand spatial signals represented in waveform form, and generate random output results containing one or more of the characteristics of piezoelectric signals and hand spatial signals.

[0070] The random block uses a random factor to output either the piezoelectric signal feature or the hand spatial signal feature, or a random output that combines multiple signal features such as piezoelectric signal features and hand spatial signal features.

[0071] The random factor satisfies the following random output condition:

[0072]

[0073] In equation (1), α is the piezoelectric signal feature, β is the hand spatial signal feature, and a is a random factor used to process the tactile piezoelectric signal and hand spatial signal represented in waveform form, and generate a random output result containing one or more of the piezoelectric signal features and hand spatial signal features.

[0074] Optionally, the visual data feature extraction and processing module includes:

[0075] The video frame preprocessing unit is used to segment the acquired acupuncture technique video into frames and generate video frame datasets corresponding to different acupuncture techniques.

[0076] Multiple stacked visual feature extraction attention blocks, each of which includes a convolutional-like component and an attention mechanism component;

[0077] The convolutional component includes three independent and parallel first convolutional channels, second convolutional channels, and third convolutional channels, as well as a concatenation layer. The first convolutional channel includes a 1×1 convolutional layer and a 3×3 convolutional layer connected in sequence. The second convolutional channel includes a 1×1 convolutional layer and a max pooling layer connected in sequence. The third convolutional channel includes a 1×1 convolutional layer and a 5×5 convolutional layer connected in sequence. The concatenation layer connects the outputs of the first, second, and third convolutional channels to perform feature fusion of the data extracted from the different convolutional channels.

[0078] The attention mechanism components include an average pooling layer, a first fully connected layer, a ReLU layer, a second fully connected layer, and a sigmoid layer connected in sequence. The average pooling layer is used to compress the needle puncture image data represented in the form of video frames in the corresponding dimensional space using a first compression formula. The first fully connected layer is used to compress the needle puncture image data compressed for the first time again using a second compression formula. The ReLU layer is used to remove data in the needle puncture image data that does not meet the preset requirements using the ReLU activation function. The second fully connected layer is used to stretch and restore the dimensional space of the data after the removal operation to the spatial dimension of the first compression. The sigmoid layer is used to map the stretched and restored data to a range of [0,1] using the sigmoid activation function.

[0079] Optionally, the multimodal feature fusion module includes:

[0080] Tensor acquisition unit, used to acquire one-dimensional tensors of random output results and one-dimensional tensors of image features;

[0081] Tensor expansion unit is used to expand the one-dimensional tensor of random output results and one-dimensional tensor of image features, so that the tensor increases by one dimension, and obtains high-dimensional tensor features of random output results and image features.

[0082] The tensor product operation unit is used to perform tensor product operations based on the high-dimensional tensor features of random output results and image features to obtain tensor outer products;

[0083] The weight matrix construction unit is used to construct the corresponding weight matrices for image features and random output results, respectively.

[0084] The fusion recognition unit is used to perform tensor fusion based on the tensor outer product, the weight matrix of image features, and the weight matrix of random output results, and obtain the fusion recognition result by using the tensor fusion formula.

[0085] The tensor fusion formula is as follows:

[0086]

[0087] In equation (10), h is the tensor obtained after final fusion, r is the number of reconstruction matrices, and w a w represents the weights of the tensor of the random output result. v Z represents the weights of the tensor of image features, and Z is the tensor outer product. a The high-dimensional tensor features obtained by augmenting the random output results, z v These are high-dimensional tensor features obtained by expanding image features.

[0088] (III) Beneficial Effects

[0089] The beneficial effects of this invention are as follows: This invention proposes a multimodal acupuncture technique recognition scheme that integrates visual and tactile senses. For action recognition such as acupuncture techniques, which contain multiple modal information, corresponding feature extraction attention blocks and random combination output modes are designed for tactile piezoelectric signals and hand spatial signals represented in waveform form. This enhances the diversity of tactile features, enriches the combination forms of waveform data features, and improves the generalization ability of the model. Furthermore, a complementary approach of convolutional channels and attention mechanisms is designed for acupuncture image data represented in video frame form, improving the ability to extract important data features from the image data while increasing network depth and width while reducing the computational load. The subsequent multimodal fusion recognition scheme of this invention, based on tensor-based multimodal feature fusion, can fuse these two information forms together to achieve more comprehensive and accurate acupuncture technique monitoring and recognition. Attached Figure Description

[0090] Figure 1 This is a flowchart illustrating a multimodal acupuncture technique recognition method that integrates vision and touch, as proposed in an embodiment of the present invention.

[0091] Figure 2 This is an overall flowchart of a multimodal acupuncture technique recognition method that integrates vision and touch, as proposed in an embodiment of the present invention.

[0092] Figure 3-1 and Figure 3-2These are schematic diagrams of tactile piezoelectric signals before and after preprocessing in a multimodal acupuncture technique recognition method that integrates vision and touch, as proposed in an embodiment of the present invention.

[0093] Figure 4 This is a schematic diagram illustrating the processing of acupuncture image data in a multimodal acupuncture technique recognition method that integrates vision and touch, as proposed in an embodiment of the present invention.

[0094] Figure 5 This is a schematic diagram illustrating the data packaging and tagging process of a multimodal needlestick technique recognition method integrating vision and touch, as proposed in an embodiment of the present invention.

[0095] Figure 6 This is a schematic diagram of the tensor product operation process of a multimodal acupuncture technique recognition method that integrates vision and touch, as proposed in an embodiment of the present invention.

[0096] Figure 7 A schematic diagram of the tensor-based fusion calculation process of a multimodal acupuncture technique recognition method that integrates vision and touch, as proposed in an embodiment of the present invention;

[0097] Figure 8 This is a schematic diagram of the composition of a multimodal acupuncture technique recognition system that integrates vision and touch, as proposed in an embodiment of the present invention.

[0098] Figure 9 This is a schematic diagram of the composition of the visual feature extraction attention block of a multimodal acupuncture technique recognition system that integrates vision and touch, as proposed in an embodiment of the present invention.

[0099] Figure 10 This is a schematic diagram illustrating the implementation process of visual feature extraction attention blocks in a multimodal acupuncture technique recognition system that integrates vision and touch, as proposed in an embodiment of the present invention. Detailed Implementation

[0100] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0101] like Figure 1As shown in the embodiment of the present invention, a multimodal acupuncture technique recognition method integrating vision and touch is proposed, comprising: First, extracting features from tactile piezoelectric signals and hand spatial signals represented in waveform form, and then randomly outputting single-signal features or multi-signal feature fusion based on the obtained piezoelectric signal features and hand spatial signal features; Second, processing acupuncture image data represented in video frame form through multiple independent convolutional channels, and simultaneously applying an attention mechanism through compression and nonlinear activation operations to adaptively calibrate the weights of each convolutional channel to obtain image features; Third, performing tensor product operation on the random output result and image features to obtain tensor outer product, and performing tensor fusion based on the tensor outer product and the weight matrices constructed for the image features and random output result respectively to obtain a fused tensor, i.e., the fused recognition result.

[0102] This invention proposes a multimodal acupuncture technique recognition scheme that integrates visual and tactile feedback. For acupuncture techniques, which involve multiple modalities of information, corresponding feature extraction attention blocks and random combination output modes are designed for tactile piezoelectric signals and hand spatial signals represented in waveform form. This enhances the diversity of tactile features, enriches the combination forms of waveform data features, and improves the model's generalization ability. Furthermore, a complementary approach of convolutional channels and attention mechanisms is designed for acupuncture image data represented in video frame form, improving the extraction capability of important data features from the image data while increasing network depth and width while reducing network computation. The subsequent multimodal fusion recognition scheme based on tensor-based multimodal feature fusion can combine these two information forms to achieve more comprehensive and accurate acupuncture technique monitoring and recognition.

[0103] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0104] Specifically, the present invention provides a multimodal acupuncture technique recognition method that integrates visual and tactile senses, including:

[0105] S1. Extract features from the tactile piezoelectric signal and hand spatial signal represented in waveform form, and then randomly output single-signal features or multi-signal feature fusion based on the obtained piezoelectric signal features and hand spatial signal features.

[0106] Before step S1, the following is also included:

[0107] F11. Preprocess the raw data of the acquired tactile piezoelectric signal and the raw data of the hand spatial signal to obtain the tactile piezoelectric signal and the hand spatial signal.

[0108] F12. The acquired acupuncture technique video is segmented frame by frame to generate a video frame dataset corresponding to different acupuncture techniques.

[0109] F13. By packaging the tactile piezoelectric signal, hand spatial signal and video frame dataset corresponding to each acupuncture technique, a synthetic array for each acupuncture technique is obtained for subsequent feature extraction.

[0110] refer to Figure 2 It can be seen that before the feature extraction step, the tactile signals and hand spatial signals collected by the physician's finger cot and accelerometer during acupuncture, based on an array of dual-layer PVDF films, are preprocessed, including moving average filtering, to obtain the processed piezoelectric data and accelerometer angle data. Taking a physician as an example... Figure 3-1 and Figure 3-2 The images show raw dual-layer PVDF piezoelectric signal samples and processed piezoelectric signals obtained by a physician performing four typical techniques while wearing a flexible tactile finger cot. Figure 3-1 and Figure 3-2 It can be seen that the processed dual-PVDF piezoelectric data has more prominent features and more obvious characteristics, with less noise; while the original data before processing is more messy, with less obvious features, posing a great challenge to effective classification.

[0111] It's important to know that data collected by binocular cameras cannot be used directly and requires further processing, such as... Figure 4 As shown, this invention segments the video according to a set number of time frames. Video extraction automatically retrieves keyframes from the video and combines them to generate a video frame dataset. Specifically, this invention processes the acquired acupuncture technique videos frame by frame to generate video frame datasets corresponding to four typical techniques (twisting and supplementing, twisting and draining, lifting and inserting supplementing, and lifting and inserting draining). Furthermore, the image size of each frame is uniformly compressed to 224*224 pixels, and 70% of the data is used as the training set, 20% as the validation set, and 10% as the test set.

[0112] refer to Figure 5It is known that before feature extraction, the three modalities need to be "packaged" and "labeled." During model training, the data needs to be scrambled to improve the model's generalization ability and increase the accuracy of technique classification. If different types of data describing the same technique are not packaged, data corruption will occur; for example, voltage data describing technique A will be fed into the network simultaneously with image and angular velocity data describing technique B, leading to training errors. Here, data A, data B, and data C represent the three types of data to be processed, and X0, X1, and X2 represent the data to be input into the network after packaging and labeling.

[0113] like Figure 5 As shown, after inputting the three modalities of data, they are temporarily stored in an array storage area. Once all data has been input, the corresponding three sets of data are combined into an array, and so on, packaging all data from one method together in this way. After all four methods have been packaged, they are proceeded together to the subsequent feature extraction steps.

[0114] Further, step S1 includes:

[0115] S11. The tactile piezoelectric signal and hand spatial signal, represented in waveform form, are extracted using CNN and sliding window respectively to obtain piezoelectric signal features and hand spatial signal features.

[0116] S12. Using a random factor, output the piezoelectric signal feature or the hand spatial signal feature, or simultaneously generate a random output from the fusion of multiple signal features, including piezoelectric signal features and hand spatial signal features:

[0117] The random factor satisfies the following random output condition:

[0118]

[0119] In equation (1), α is the piezoelectric signal feature, β is the hand spatial signal feature, and a is a random factor used to process the tactile piezoelectric signal and hand spatial signal represented in waveform form, and generate a random output result containing one or more of the piezoelectric signal features and hand spatial signal features.

[0120] S2. The needle puncture image data, represented in the form of video frames, is processed by multiple independent convolutional channels. Attention mechanisms are applied through squeezing and nonlinear activation operations to adaptively calibrate the weights of each convolutional channel to obtain image features.

[0121] Further, step S2 includes:

[0122] S21. The needle puncture image data, represented in the form of video frames, is processed by convolutional channels that are independent of each other.

[0123] S22. Information on convolutional channel features is generated by squeezing and nonlinearly activating the needle puncture image data represented in video frames, and global spatial information is collected to perform adaptive calibration of the weights of each convolutional channel.

[0124] Considering that the output length and width must be equal to the output width and height when image samples pass through the convolution channel, to prevent data loss during image sample processing, the image is padded with zeros on the outer layer. This allows for better feature extraction of image edge data while preserving data integrity. The convolution kernel slides once per operation, with a stride of 1, ensuring that the kernel slides only one unit to the right each time, preserving more data features. Therefore, when processing the acupuncture image data through the convolution channel, the acupuncture image data is padded with zeros on the outer layer, and the convolution kernel operates under the following conditions:

[0125]

[0126]

[0127]

[0128] In equations (2)-(4), padding is the padding parameter, which completes the padding of the outer layer of the image with zeros, f is the length of the convolution kernel in each layer, s is the stride of the convolution kernel, h is the length of the input feature map, w is the width of the input feature map, and h out To determine the length of the output feature map, w out This is the width of the output feature map.

[0129] Furthermore, step S22 includes:

[0130] S211. The needle puncture image data represented in the form of video frames is compressed in the corresponding dimensional space using the first compression formula.

[0131] S212. The needle puncture image data that was compressed initially is compressed again using the second compression formula.

[0132] S213. The ReLU activation function is used to remove data that does not meet the preset requirements from the recompressed needle puncture image data.

[0133] S214. The activation function stretches the dimensional space of the data after the culling operation back to the original compressed dimensional space, and then uses the sigmoid activation function to stretch and map the restored data into a range of [0,1]. The activation operation can be seen as a process of increasing dimensionality. The previous compression operation can be seen as a reduction in the dimensionality of the data and a centralization of the information in the data. At this point, increasing the dimensionality and stretching it out is aimed at limiting the model complexity and assisting the model in generalization.

[0134] in,

[0135] The first extrusion formula is:

[0136]

[0137] In equation (5), F sq This refers to the output feature value u c This refers to the compression of the input feature map in the spatial dimension W×H through the compression of the average pooling layer, that is, the input of W×H×C is transformed into the output of 1×1×C;

[0138] The second extrusion formula is:

[0139] x=FC(F sq (u c (6)

[0140] In equation (6), x represents the output of the second compression process, and FC represents the compression process in the spatial dimension C completed by the second compression through the fully connected layer, that is, the input of 1×1×C is converted into the output of 1×1×C / r.

[0141] The ReLU activation function is:

[0142]

[0143] In equation (8), e represents the natural logarithm.

[0144] The sigmoid activation function is:

[0145]

[0146] The activation function is:

[0147] s = F ex (z,W)=Relu(g(z,W))=σ(W2δ(W1z)) (9)

[0148] In equation (9), S is the output after excitation, and F exThe mechanism of gates in a recurrent neural network is represented by the parameter W, which generates weights for each feature channel. The parameter W is used to explicitly model the correlation between feature channels. z represents the output of formula (5), g represents the gating mechanism in the form of sigmoid, δ and σ represent the ReLU and sigmoid activation functions, respectively, and w1 and w2 are the parameters obtained by vector decomposition of parameter W. denoted as the number of channels in the image, and r is a hyperparameter for dimensionality reduction.

[0149] S23. Based on adaptive weights, feature fusion is performed on the data extracted from different convolution channels using a concatenation method.

[0150] S24. After repeatedly processing the data through independent convolutional channels and applying attention mechanisms, image features are obtained.

[0151] S3. Perform tensor product operation on the random output result and image features to obtain tensor outer product, and perform tensor fusion based on the tensor outer product and the weight matrices constructed for the image features and random output result respectively to obtain the fused tensor, that is, the fused recognition result.

[0152] Further, step S3 includes:

[0153] S31. Obtain the one-dimensional tensor of the random output result and the one-dimensional tensor of the image features.

[0154] S32. Expand the one-dimensional tensor of the random output result and the one-dimensional tensor of the image feature to increase the tensor by one dimension, and obtain the high-dimensional tensor feature of the random output result and the image feature.

[0155] S33. Based on the high-dimensional tensor features of the random output results and image features, perform tensor product operation to obtain tensor outer product.

[0156] S34. Construct corresponding weight matrices for the image features and random output results, respectively.

[0157] S35. Based on the tensor cross product, the weight matrix of image features, and the weight matrix of random output results, tensor fusion is performed using the tensor fusion formula to obtain the fused recognition result.

[0158] The tensor fusion formula is as follows:

[0159]

[0160] In equation (10), h is the tensor obtained after final fusion, r is the number of reconstruction matrices, and w a w represents the weights of the tensor of the random output result. vZ represents the weights of the tensor of image features, and Z is the tensor outer product. a The high-dimensional tensor features obtained by augmenting the random output results, z v These are high-dimensional tensor features obtained by expanding image features.

[0161] refer to Figure 6 First, after obtaining the one-dimensional tensors of image features, tactile piezoelectric signal features, and hand spatial signal features, the tensors are expanded by adding one dimension to obtain high-dimensional tensor features. Then, tensor product operations are performed to obtain the tensor outer product. It can be seen that after expansion, the feature correlation between the two modes is calculated, while the information of the specific mode is preserved.

[0162] Subsequently, reference Figure 7 The process of fusing tensors into an h-dimensional feature tensor according to the tensor fusion formula is as follows: First, perform the outer product calculation of the tensors; then, construct R weight matrices for each modality. At this time, set R=4. After fusion according to the tensor fusion formula, perform matrix multiplication on the features of each modality to obtain an h-dimensional feature; then, perform pixel-level multiplication on the h-dimensional feature vectors obtained from each modality.

[0163] On the other hand, such as Figure 8 As shown, this invention provides a multimodal acupuncture technique recognition model that integrates visual and tactile senses, comprising:

[0164] The waveform data feature extraction and processing module is used to extract features from tactile piezoelectric signals and hand spatial signals represented in waveform form, and to randomly output single-signal features or multi-signal feature fusion based on the obtained piezoelectric signal features and hand spatial signal features.

[0165] The visual data feature extraction and processing module is used to process the needle puncture image data represented in the form of video frames through multiple independent convolutional channels. At the same time, it applies an attention mechanism through squeezing and nonlinear activation operations to adaptively calibrate the weights of each convolutional channel to obtain image features.

[0166] The multimodal feature fusion module is used to perform tensor product operation on random output results and image features to obtain tensor outer product, and then perform tensor fusion based on tensor outer product and weight matrices constructed for image features and random output results to obtain fused tensor, i.e. fused recognition result.

[0167] The waveform data feature extraction and processing module includes:

[0168] The waveform data preprocessing unit is used to preprocess the raw data of the acquired tactile piezoelectric signal and the raw data of the hand spatial signal to obtain the tactile piezoelectric signal and the hand spatial signal.

[0169] A CNN network is used to extract features from the acquired hand spatial signal, which is represented in waveform form, to obtain the hand spatial signal features.

[0170] A sliding window feature extraction network is used to extract features from the acquired tactile piezoelectric signals represented in waveform form, thereby obtaining tactile piezoelectric signal features.

[0171] The random block is used to process tactile piezoelectric signals and hand spatial signals represented in waveform form, and generate random output results that include one or more of the characteristics of piezoelectric signals and hand spatial signals.

[0172] Therefore, the present invention provides a random block for processing angular features and voltage feature values ​​and angular change waveform data feature maps expressed as waveforms. After the two types of data enter this unit, the voltage waveform features and angular waveform features can have richer combination forms.

[0173] This randomized block, designed for the diversity of tactile features, enhances the model's generalization ability, allowing for more feature combinations and strengthening the expressive power of the data features. This reduces the risk of overfitting during training and improves the model's robustness. The module includes a random factor that changes randomly in each training round, resulting in different outputs for the features.

[0174] The output has the following three possibilities

[0175] (1) Characteristics of individual tactile piezoelectric signals.

[0176] (2) Characteristics of the angle signal of the accelerometer alone.

[0177] (3) Fusion of voltage characteristic value and angle change waveform data feature map.

[0178] The random block uses a random factor to output either the piezoelectric signal feature or the hand spatial signal feature, or a random output that combines multiple signal features such as piezoelectric signal features and hand spatial signal features.

[0179] The random factor satisfies the following random output condition:

[0180]

[0181] In the formula, α is the feature signal of the tactile piezoelectric signal after passing through the sliding window, β is the feature vector of the hand spatial signal after passing through the CNN, and a is a random factor used to process the tactile piezoelectric signal and the hand spatial signal represented in waveform form, and generate a random output result containing one or more of the features of the piezoelectric signal and the hand spatial signal.

[0182] Next, the visual data feature extraction and processing module includes:

[0183] The video frame preprocessing unit is used to segment the acquired acupuncture technique video into frames and generate video frame datasets corresponding to different acupuncture techniques.

[0184] Multiple stacked visual feature extraction attention blocks, each of which includes a convolutional-like component and an attention mechanism component.

[0185] The convolutional-like component includes three independent and parallel convolutional channels: a first convolutional channel, a second convolutional channel, and a third convolutional channel, as well as a concatenation layer. The first convolutional channel includes a 1×1 convolutional layer and a 3×3 convolutional layer connected in sequence. The second convolutional channel includes a 1×1 convolutional layer and a max pooling layer connected in sequence. The third convolutional channel includes a 1×1 convolutional layer and a 5×5 convolutional layer connected in sequence. The concatenation layer connects the outputs of the first, second, and third convolutional channels to perform feature fusion of the data extracted from the different convolutional channels.

[0186] The attention mechanism components include an average pooling layer, a first fully connected layer, a ReLU layer, a second fully connected layer, and a sigmoid layer connected in sequence. The average pooling layer is used to compress the needle puncture image data represented in the form of video frames in the corresponding dimensional space using a first compression formula. The first fully connected layer is used to compress the needle puncture image data compressed for the first time again using a second compression formula. The ReLU layer is used to remove data in the needle puncture image data that does not meet the preset requirements using the ReLU activation function. The second fully connected layer is used to stretch and restore the dimensional space of the data after the removal operation to the spatial dimension of the first compression. The sigmoid layer is used to map the stretched and restored data to a range of [0,1] using the sigmoid activation function.

[0187] refer to Figure 9 The visual feature extraction attention block of this invention is mainly divided into two parts, left and right. The left-side convolutional component, i.e., the convolutional part of the network, extracts image features from video frames of the acupuncture technique using convolutional kernels with different structures. While increasing the network depth, it effectively controls the increase in computational cost and reduces the model size. The right-side attention mechanism component references the compression and activation operations of the attention mechanism in the SENet architecture, adding an attention mechanism in the channel dimension (i.e., the C dimension in H×W×C). It enhances the model's expressive power by utilizing the relationships between the channels of the convolutional features.

[0188] The following sections will provide a detailed introduction to the convolution-like components and the attention mechanism components:

[0189] (1) Convolutional-like components

[0190] In this invention, a highly parallel convolutional unit is designed to extract features from data from different network channels. This unit contains three paths that can be computed in parallel without any interaction. The output layers of the three independent paths concatenate the data extracted from different channels to fuse features, and this fusion serves as the output of the current visual feature extraction attention block, which is then connected to the input of the next visual feature extraction attention block.

[0191] Subsequently, the convolutional channels of the convolutional-like components include 3×3, 1×1, and 5×5 convolutional layers, max-pooling layers, and cascaded layers. The 1×1 convolutional layer added before the 3×3 and 5×5 convolutions helps reduce the dimensionality of the feature maps, and stacking more convolutional kernels on the same receptive field allows the model to learn richer features. This method is superior to the traditional convolutional layer operation that uses only a single-size convolutional kernel for input data. Furthermore, the model's dimensionality can be reduced, thus decreasing computational cost. When a convolutional layer receives a large number of feature maps as input, the computational cost of convolution operations becomes very high. If the feature maps are first reduced in dimensionality before performing convolution operations, the computational cost will be significantly reduced.

[0192] In convolutional components, 3×3 and 5×5 convolutional kernels help the model learn the spatial patterns of all dimensions of the input. Figure 9 As can be seen, the image learns more features and acquires more information after passing through the convolutional part. These features are then fused, and the resulting new features contain even more information, enhancing the network's learning ability. Max pooling layers were added to remove noise and eliminate redundant data. Figure 10 A schematic diagram of multi-scale visual feature extraction in "convolution-like components" is given. Figure 9 The left side shows a schematic diagram of the "convolutional component-like" branch.

[0193] It is worth noting that the amount of data in the "convolution-like component" is the sum of the three branches. For example, image data with a pixel size of 28×28×256 is input into the convolution part of the attention module.

[0194] Consider inputting an H×W×C (C=16) video frame image of acupuncture technique into a "convolutional unit". The main data processing flow of the convolutional component is as follows:

[0195] Step 1: First convolutional channel: Input image samples pass through a 1×1 convolution kernel one after another. At this time, the structure of the image will become 28×28×64. Then the data of this structure will be processed by a 3×3 convolution kernel, and the structure of the feature vector will be transformed into 28×28×192.

[0196] Step 2: Second Convolutional Channel: Input image samples are successively passed through a 1×1 convolutional layer. After this modification, the structure of its feature vector will become 28×28×64. Then, it passes through a pooling layer, resulting in an output feature vector with the same structure.

[0197] Step 3: Third convolution channel: After the input image samples pass through a 1×1 convolution kernel, the feature vector structure of the image samples will become 28×28×64. Then, after processing through a 5×5 convolution kernel, the feature vector of the image samples will become 28×28×96. Finally, after concatenation, the three feature vectors are fused together, and the overall computational cost is 28×28×(192+64+96), which is 275968.

[0198] (2) Attention mechanism components

[0199] refer to Figure 9 On the right side, the "attention mechanism component" of the attention block designed in this invention is dedicated to locating the positional information of the hand in acupuncture techniques, i.e., finding important data in the image while ignoring environmental and other factors. It generates channel feature information and collects global spatial information through squeezing, global average pooling layers, and activation functions to complement the convolutional part and integrate the features obtained from the left and right units of the overall attention module. Therefore, the attention mechanism is embedded into the left convolutional part to obtain more representative fused features. This involves adaptively recalibrating the weights of each channel to determine the target object of attention. The channel squeezing and activation operations proposed in the SENet network are used to construct a dependency model between channels, thereby improving the expressive power of the network model.

[0200] Consider a sample with a structure of W×H×C as input into the network. The network computation process corresponding to the "attention unit" on the right side of the attention block is as follows:

[0201] 1) Pooling operation (extrusion operation)

[0202] First, the image samples pass through an average pooling layer. This layer averages and optimizes the data in the resulting feature map. The attention unit operation then uses a global average pooling layer to compress the data from the entire feature map onto a single channel, effectively compressing the data structure into a 1×1×C format. This compression operation utilizes global average pooling, leveraging contextual information beyond the local receptive field to generate channel feature data. The calculation process is as follows:

[0203]

[0204] Among them, F sqThe output feature value refers to the shrinkage of the input feature map in the spatial dimension W×H obtained by compression through the average pooling layer, in order to achieve statistical calculation of channel features. This formula transforms the input into a 1×1×C output.

[0205] The compressed data will then enter a fully connected layer, where the dimensionality will be further reduced to C / r×C, where r = H×W. This parameter is used to reduce the number of parameters and thus reduce computation. Since the output dimension of the previous layer is 1×1×C, after passing through the fully connected layer, the feature structure becomes 1×1×C / r. Therefore:

[0206] x=FC(F sq (u c ))

[0207] 2) Nonlinear activation (excitation operation)

[0208] After the compression phase, the next step is activation. First, the feature map passes through a ReLU layer. The ReLU layer has a simple structure and effectively removes unsuitable data, providing excellent dilution. The output dimension of the feature map remains unchanged, but more information is retained. Then, the feature map passes through a fully connected layer. This fully connected layer has the opposite effect to the previous one; it expands the 1×1×C / r feature structure into 1×1×C. Finally, the feature map passes through a sigmoid activation function layer, mapping the final data to the [0,1] interval. The formula is as follows:

[0209]

[0210]

[0211] s = F ex (z,W)=Relu(g(z,W))=σ(W2δ(W1z))

[0212] Here, δ and σ represent the Rectified Linear Unit (ReLU) and the Sigmoid activation function, respectively. The activation operation can be viewed as a process of increasing dimensionality. The previous compression operation reduced the dimensionality of the data and centralized the information. Now, increasing the dimensionality expands the data to limit model complexity and help the model generalize.

[0213] Furthermore, the multimodal feature fusion module includes:

[0214] The tensor acquisition unit is used to acquire one-dimensional tensors of random output results and one-dimensional tensors of image features.

[0215] Tensor expansion units are used to expand the one-dimensional tensor of random output results and the one-dimensional tensor of image features, increasing the tensor by one dimension to obtain high-dimensional tensor features of random output results and image features.

[0216] The tensor product operation unit is used to perform tensor product operations based on the high-dimensional tensor features of random output results and image features to obtain tensor outer products.

[0217] The weight matrix construction unit is used to construct the corresponding weight matrices for image features and random output results, respectively.

[0218] The fusion recognition unit is used to perform tensor fusion based on the tensor outer product, the weight matrix of image features, and the weight matrix of random output results, and obtain the fusion recognition result by using the tensor fusion formula.

[0219] Furthermore, the present invention also provides a device for extracting device outline and pipeline information from flowcharts, comprising: at least one database; and a memory communicatively connected to the at least one database; wherein the memory stores instructions executable by the at least one database, the instructions being executed by the at least one database to enable the at least one database to perform the multimodal acupuncture technique recognition method that integrates vision and touch as described above.

[0220] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multimodal needle-puncture technique recognition method that integrates vision and touch as described above.

[0221] In one specific embodiment, to objectively reflect the recognition performance of different multimodal feature extraction methods and different multimodal fusion methods under the overall acupuncture technique recognition model, firstly, based on the multimodal feature fusion method used in this invention, the performance of technique recognition was verified under different submodal feature extraction methods. Secondly, unlike this invention, LSTM or CNN is used to extract features from waveform display data, and classical neural networks are used to extract features from image data. A total of 13 classical neural network models were used for comparison, while other parts remained unchanged.

[0222] Furthermore, to verify the impact of different multimodal feature fusion methods on gesture recognition performance, this invention references the validation set accuracy and test set accuracy of each model, as shown in Table 1. The test data includes the overall accuracy and the individual accuracy for each gesture. The waveform data feature extraction and processing module was replaced with CNN and LSTM models respectively, and the accuracy change curves were obtained after 70 training iterations.

[0223] Furthermore, some scholars have proposed two methods for multimodal feature fusion: Multi-modal Factorized Bilinear Pooling (MFB) and Multimodal Compact Bilinear Pooling (MCB). Table 1 demonstrates the performance of this invention in identifying acupuncture techniques within the proposed framework, comparing the performance of the aforementioned two multimodal fusion methods with that of the multimodal fusion method proposed in this invention.

[0224] Table 1. Accuracy of the proposed solution compared to the validation and test sets of classical neural networks.

[0225]

[0226] Based on the results in Table 1, the following conclusions can be drawn: First, the traditional back-end fusion method achieves a recognition accuracy of 68% to 81% on the validation set for the four classic acupuncture techniques. Due to the temporal nature of voltage and angle data, the recognition accuracy obtained by processing them using the LSTM model is better than that of CNN. The temporal influence of voltage and angle data generally makes LSTM more robust than CNN. Second, because the lifting, inserting, and purging movements are small in amplitude and have a high frequency, the recognition rate of neural networks is generally lower than that of the other three techniques. The method of this invention has the highest recognition accuracy, reaching 91.6% on the validation set and 83% on the test set, which is higher than other methods.

[0227] Therefore, it can be seen that the accuracy of training data using traditional neural networks and CNNs before 10 rounds of training is generally lower than that of traditional neural networks and LSTMs. This also proves that LSTM performs better than CNNs for time-series data. However, the traditional neural network plus LSTM method has greater volatility and converges slightly slower than the traditional neural network plus CNN method. Traditional neural networks mainly extract features from image contours or structures for the four classic techniques. However, the hand shape difference in the acupuncture technique is low, and the movements are subtle, with little change in contour and structure, resulting in limited recognition ability of traditional neural networks for the four techniques. The "visual feature extraction attention block" designed in this invention not only extracts the contour and structural features of the hand, but also incorporates an attention mechanism to focus on important information and ignore unimportant information, thereby improving the recognition ability of the four classic techniques.

[0228] It can also be found that the traditional Googlenet and Inception-v2 methods are not good at recognizing acupuncture techniques. In particular, the combination of Googlenet and LSTM has an accuracy of less than 50% for recognizing four classic acupuncture techniques, which proves that the improvement of the Inception module in this invention is necessary. After adding the attention mechanism, the recognition accuracy of the four acupuncture techniques exceeds 90%.

[0229] In summary, this invention provides a multimodal acupuncture technique recognition method and system that integrates visual and tactile senses, with the following improvements:

[0230] (1) In the recognition method and system of the present invention, based on the attention mechanism, multiple stacked visual feature extraction attention blocks are defined to integrate the visual features of acupuncture manipulation movements. These visual feature extraction attention blocks can explicitly establish a quantification model of the interdependencies between network channels. Subsequently, in the overall acupuncture manipulation recognition deep learning network, the visual feature extraction attention blocks are stacked to form a visual feature extraction network. This increases the network depth and width while reducing the computational load.

[0231] (2) To construct a variety of tactile features, a random block was designed to enhance the diversity of tactile features and strengthen the generalization ability of the overall technique recognition model.

[0232] (3) Based on the idea of ​​multimodal feature fusion of tensors, a mechanism for fusion of visual and tactile features of acupuncture techniques was constructed, and the construction of a deep learning network for overall acupuncture technique recognition was completed.

[0233] Since the systems / devices described in the above embodiments of the present invention are systems / devices used to implement the methods of the above embodiments of the present invention, those skilled in the art can understand the specific structure and modifications of the systems / devices based on the methods described in the above embodiments of the present invention, and therefore will not be repeated here. All systems / devices used in the methods of the above embodiments of the present invention fall within the scope of protection of the present invention.

[0234] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0235] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. A multimodal acupuncture technique recognition method integrating visual and tactile senses, characterized in that, include: Feature extraction is performed on the tactile piezoelectric signal and hand spatial signal represented in waveform form. Then, based on the obtained piezoelectric signal features and hand spatial signal features, a random output is generated by fusing single or multiple signal features. This includes: extracting features from the tactile piezoelectric signal and hand spatial signal represented in waveform form using CNN and sliding window respectively, to obtain piezoelectric signal features and hand spatial signal features; and using a random factor to output either the piezoelectric signal features, the hand spatial signal features, or a combination of both, resulting in a random output. The random factor satisfies the following random output condition: （1） In equation (1), α piezoelectric signal characteristics, β The spatial signal characteristics of the hand a It is a random factor used to process tactile piezoelectric signals and hand spatial signals represented in waveform form, and to generate random output results containing one or more of the characteristics of piezoelectric signals and hand spatial signals. The needle puncture image data, represented in the form of video frames, is processed by multiple independent convolutional channels. Attention mechanisms are applied through squeezing and nonlinear activation operations to adaptively calibrate the weights of each convolutional channel to obtain image features. Tensor product operation is performed on the random output result and image features to obtain tensor outer product. Tensor fusion is then performed based on the tensor outer product and the weight matrices constructed for the image features and random output result respectively to obtain the fused tensor, which is the fused recognition result.

2. The multimodal acupuncture technique recognition method integrating vision and touch as described in claim 1, characterized in that, Before extracting features from the tactile piezoelectric signal and hand spatial signal represented in waveform form, and then randomly outputting single-signal or multi-signal feature fusion based on the obtained piezoelectric signal features and hand spatial signal features, the process also includes: The raw data of the acquired tactile piezoelectric signal and the raw data of the hand spatial signal are preprocessed to obtain the tactile piezoelectric signal and the hand spatial signal. The acquired acupuncture technique videos are segmented frame by frame to generate video frame datasets corresponding to different acupuncture techniques. By packaging the tactile piezoelectric signal, hand spatial signal, and video frame dataset corresponding to each acupuncture technique, a synthetic array for each acupuncture technique is obtained for subsequent feature extraction.

3. The multimodal acupuncture technique recognition method integrating vision and touch as described in claim 1, characterized in that, The acupuncture image data, represented as video frames, is processed by multiple independent convolutional channels. Simultaneously, an attention mechanism is applied through squeezing and non-linear activation operations to adaptively calibrate the weights of each convolutional channel to obtain image features, including: The needle puncture image data, represented in the form of video frames, is processed by convolutional channels that are independent of each other. Information on convolutional channel features is generated by squeezing and nonlinear activation of needle puncture image data represented in video frames, and global spatial information is collected to perform adaptive calibration of the weights of each convolutional channel. Based on adaptive weights, features are fused from data extracted from different convolutional channels using a concatenation approach. Image features are obtained by repeatedly processing the data through independent convolutional channels and applying attention mechanisms. When the data is processed by the convolution channel, the needle puncture image data is expanded outward by padding with zeros on the outer layer of the image so that the convolution kernel slides once each time. That is, the stride is set to 1 for each convolution calculation. The operation of the convolution kernel satisfies the following conditions: （2） （3） （4） In equations (2)-(4), padding The padding parameter is used to pad the outer layer of the image with zeros. f The length of the convolutional kernel in each layer ，s The stride of the convolution kernel. h The length of the input feature map, w The width of the input feature map ， To output the length of the feature map, This is the width of the output feature map.

4. The multimodal acupuncture technique recognition method integrating vision and touch as described in claim 3, characterized in that, Information about convolutional channel features is generated by squeezing and nonlinearly activating acupuncture image data represented in video frames, and global spatial information is collected to perform adaptive calibration of the weights for each convolutional channel, including: The acupuncture image data, represented in the form of video frames, is compressed in the corresponding dimensional space using the first compression formula; The acupuncture image data, which was compressed initially, is compressed again using a second compression formula; The ReLU activation function is used to remove data from the recompressed acupuncture image data that does not meet the preset requirements. The activation function stretches the dimensional space of the data after the culling operation back to the original compressed dimensional space, and the sigmoid activation function maps the stretched and restored data into a range of [0,1]. in, The first extrusion formula is: （5） In equation (5), This refers to the output feature value This refers to the compression of the input feature map in the spatial dimension W×H through the compression of the average pooling layer, that is, the input of W×H×C is transformed into the output of 1×1×C; The second extrusion formula is: （6） In equation (6), x This indicates the output of the re-extrusion process. FC This means that the compression of the spatial dimension C is completed through the further compression of the fully connected layer, that is, the 1×1×C input is transformed into the 1×1×C / r output; The ReLU activation function is: （7） The sigmoid activation function is: （8） In equation (8), e Represents the natural logarithm; The activation function is: （9） In equation (9), F ex This represents the mechanism of gates in a recurrent neural network, that is, through parameters. W To generate weights for each feature channel, where the parameters W It is used to explicitly model the correlation between feature channels, z represents the output of formula (5), and g represents the gating mechanism in the form of Sigmoid. δ and σ These represent the ReLU and Sigmoid activation functions, respectively. w 1. w 2 refers to the parameters obtained by vector decomposition of parameter W. , denoted as the number of channels in the image, and r is a hyperparameter for dimensionality reduction.

5. The multimodal acupuncture technique recognition method integrating vision and touch as described in claim 1, characterized in that, Tensor product operations are performed on the random output and image features to obtain the tensor outer product. Then, tensor fusion is performed based on the tensor outer product and the weight matrices constructed for the image features and the random output, respectively, to obtain the fused tensor. The fused recognition result includes: Obtain a one-dimensional tensor of the random output result and a one-dimensional tensor of the image features; The one-dimensional tensor of the random output and the one-dimensional tensor of the image features are expanded to increase the tensor by one dimension, resulting in high-dimensional tensor features of the random output and the image features. Tensor cross product is obtained by performing tensor product operation based on the high-dimensional tensor features of random output results and image features; Construct corresponding weight matrices for the image features and the random output results respectively; Based on the tensor outer product, the weight matrix of image features, and the weight matrix of random output results, tensor fusion is performed using the tensor fusion formula to obtain the fused recognition result; The tensor fusion formula is as follows: （10） In equation (10), h is the tensor obtained after final fusion. r To the number of reconstructed matrices, w a The weights of the tensor representing the random output result. w v Z represents the weights of the tensor of image features, and Z is the tensor outer product. a The high-dimensional tensor features obtained by augmenting the random output results, z v These are high-dimensional tensor features obtained by expanding image features.

6. A multimodal acupuncture technique recognition system integrating vision and touch, characterized in that, include: The waveform data feature extraction and processing module is used to extract features from tactile piezoelectric signals and hand spatial signals represented in waveform form, and to randomly output single-signal features or multi-signal feature fusion based on the obtained piezoelectric signal features and hand spatial signal features; The visual data feature extraction and processing module is used to process the needle puncture image data represented in the form of video frames through multiple independent convolutional channels. At the same time, it applies an attention mechanism through squeezing and nonlinear activation operations to adaptively calibrate the weights of each convolutional channel to obtain image features. The multimodal feature fusion module is used to perform tensor product operation on random output results and image features to obtain tensor outer product, and then perform tensor fusion based on tensor outer product and weight matrices constructed for image features and random output results to obtain fused tensor, i.e. fused recognition result; The waveform data feature extraction and processing module includes: a waveform data preprocessing unit, used to preprocess the raw data of the acquired tactile piezoelectric signal and the raw data of the hand spatial signal to obtain the tactile piezoelectric signal and the hand spatial signal; a CNN network, used to extract features from the acquired hand spatial signal represented in waveform form to obtain hand spatial signal features; a sliding window feature extraction network, used to extract features from the acquired tactile piezoelectric signal represented in waveform form to obtain tactile piezoelectric signal features; and a random block, used to process the tactile piezoelectric signal and the hand spatial signal represented in waveform form and generate random output results containing one or more of the piezoelectric signal features and hand spatial signal features; the random block uses a random factor to output piezoelectric signal features, hand spatial signal features, or a random output that combines piezoelectric signal features and hand spatial signal features simultaneously: wherein the random factor satisfies the following random output conditions: （1） In equation (1), α piezoelectric signal characteristics, β The spatial signal characteristics of the hand a For random factors The processor is used to process tactile piezoelectric signals and hand spatial signals represented in waveform form, and generate random output results that include one or more of the characteristics of piezoelectric signals and hand spatial signals.

7. The multimodal acupuncture technique recognition system integrating vision and touch as described in claim 6, characterized in that, The visual data feature extraction and processing module includes: The video frame preprocessing unit is used to segment the acquired acupuncture technique video into frames and generate video frame datasets corresponding to different acupuncture techniques. Multiple stacked visual feature extraction attention blocks, each of which includes a convolutional-like component and an attention mechanism component; The convolutional component includes three independent and parallel first convolutional channels, second convolutional channels, and third convolutional channels, as well as a concatenation layer. The first convolutional channel includes a 1×1 convolutional layer and a 3×3 convolutional layer connected in sequence. The second convolutional channel includes a 1×1 convolutional layer and a max pooling layer connected in sequence. The third convolutional channel includes a 1×1 convolutional layer and a 5×5 convolutional layer connected in sequence. The concatenation layer connects the outputs of the first, second, and third convolutional channels to perform feature fusion of the data extracted from the different convolutional channels. The attention mechanism components include an average pooling layer, a first fully connected layer, a ReLU layer, a second fully connected layer, and a sigmoid layer connected in sequence. The average pooling layer is used to compress the needle puncture image data represented in the form of video frames in the corresponding dimensional space using a first compression formula. The first fully connected layer is used to compress the needle puncture image data compressed for the first time again using a second compression formula. The ReLU layer is used to remove data in the needle puncture image data that does not meet the preset requirements using the ReLU activation function. The second fully connected layer is used to stretch and restore the dimensional space of the data after the removal operation to the spatial dimension of the first compression. The sigmoid layer is used to map the stretched and restored data to a range of [0,1] using the sigmoid activation function.

8. The multimodal acupuncture technique recognition system integrating vision and touch as described in claim 6, characterized in that, The multimodal feature fusion module includes: Tensor acquisition unit, used to acquire one-dimensional tensors of random output results and one-dimensional tensors of image features; Tensor expansion unit is used to expand the one-dimensional tensor of random output results and one-dimensional tensor of image features, so that the tensor increases by one dimension, and obtains high-dimensional tensor features of random output results and image features. The tensor product operation unit is used to perform tensor product operations based on the high-dimensional tensor features of random output results and image features to obtain tensor outer products; The weight matrix construction unit is used to construct the corresponding weight matrices for image features and random output results, respectively. The fusion recognition unit is used to perform tensor fusion based on the tensor outer product, the weight matrix of image features, and the weight matrix of random output results, and obtain the fusion recognition result by using the tensor fusion formula. The tensor fusion formula is as follows: （10） In equation (10), h is the tensor obtained after final fusion. r To the number of reconstructed matrices, w a The weights of the tensor representing the random output result. w v Z represents the weights of the tensor of image features, and Z is the tensor outer product. a The high-dimensional tensor features obtained by augmenting the random output results, z v These are high-dimensional tensor features obtained by expanding image features.

Images