A Method and System for Designing Multimodal Neural Network Architectures for Neural Interface Chips

Abstract

This invention provides a method and system for designing a multimodal neural network architecture for neural interface chips, comprising: timestamping multimodal data samples via a host computer; uniformly converting data files from different modalities into CSV format and standardizing the data format through timestamp alignment and downsampling; preprocessing the multimodal data and extracting sample data within a time window through timestamp normalization and interpolation alignment; designing a lightweight multimodal neural network architecture based on MobileNetV2, with each channel fusing features extracted through an inverse residual module; constructing a dynamic loss function, dynamically adjusting the weights of joint angle regression, pressure regression, and classification tasks through exponential decay and the sigmoid function to achieve weighted loss fusion; adjusting network hyperparameters and training the model, outputting the prediction results of hand joint angles and fingertip pressure. This invention can accurately predict hand grasping actions with a limited number of network parameters and is suitable for lightweight neural interface chips.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of deep learning and neural interface technology, and more specifically, to a method and system for designing multimodal neural network architectures for neural interface chips. Background Technology

[0002] With the rapid development of neuroscience and microelectronics, neural interface chips, as a key technology connecting the nervous system to external devices, have gradually become a research hotspot in fields such as brain-computer interfaces, neurorehabilitation, and intelligent prostheses. Neural interface chips can acquire and process neural signals in real time, providing a foundation for precise control of external devices. However, traditional neural interface chips face significant challenges in processing multimodal neural signals (such as electromyography (EMG) signals, pressure, joint angles, and temperature). The sampling rates, data formats, and noise levels of different modalities vary considerably, leading to high complexity in data alignment and fusion. For example, EMG signals are typically sampled at kilohertz frequencies, while temperature signals may only be a few hertz; this heterogeneity results in low efficiency for time synchronization and feature extraction. Furthermore, the low power consumption and limited computing resources of neural interface chips restrict the direct deployment of traditional high-complexity neural networks, hindering the real-time processing capabilities of multimodal data.

[0003] In the field of multimodal data fusion, traditional methods such as BP neural networks and RBF neural networks, while possessing certain nonlinear modeling capabilities, suffer from high computational resource consumption due to their fixed structure and large number of parameters, and are also sensitive to noise. In recent years, lightweight networks (such as MobileNetV2) have demonstrated efficient feature extraction capabilities in image processing, but their direct transfer to multimodal neural signal processing still presents adaptation issues. For example, multimodal data exhibits significant spatiotemporal differences, making it difficult for a single network architecture to adequately account for the characteristics of each modality. Furthermore, static loss function design cannot balance the weight distribution during multi-task training, easily leading to overfitting or unstable model convergence. In addition, existing methods are inefficient in data preprocessing; the format conversion, timestamp alignment, and noise filtering processes for data from different sensors are cumbersome, further increasing system latency.

[0004] Multimodal fusion technology holds significant value in scenarios such as hand movement prediction. By integrating the dynamic characteristics of electromyographic signals, tactile feedback from pressure signals, and the motion trajectory of joint angles, the accuracy and real-time performance of movement recognition can be significantly improved, providing more refined feedback for neurorehabilitation training. However, current technologies still face bottlenecks such as weak model generalization ability and poor hardware adaptability. Therefore, there is an urgent need for a lightweight and robust multimodal neural network architecture to resolve the contradiction between data heterogeneity, resource constraints, and task diversity, and to promote the widespread application of neural interface chips in clinical and engineering fields. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for designing a multimodal neural network architecture for neural interface chips.

[0006] According to the present invention, a multimodal neural network architecture design method for a neural interface chip is provided, the method comprising the following steps:

[0007] Step S1: The multimodal data samples are timestamped by the host computer. The multimodal data includes fingertip pressure, palm pressure, arm electromyography signals, hand joint angles, and fingertip temperature.

[0008] Step S2: Convert data files of different modalities into CSV format and standardize the data format through timestamp alignment and downsampling.

[0009] Step S3: Preprocess the multimodal data using bandpass filtering, notch filtering, and low-pass filtering, and extract sample data within the time window by timestamp normalization and interpolation alignment;

[0010] Step S4: Design a lightweight multimodal neural network architecture based on MobileNetV2, which includes parallel processing of electromyography channels, joint angle channels, pressure channels and temperature channels. Each channel extracts features through an inverse residual module and then fuses them.

[0011] Step S5: Construct a dynamic loss function, and dynamically adjust the weights of joint angle regression, stress regression and classification task through exponential decay and sigmoid function to achieve weighted loss fusion;

[0012] Step S6: Adjust the network hyperparameters and train the model, outputting the prediction results of hand joint angles and fingertip pressure.

[0013] Preferably, the downsampling process in step S2 includes:

[0014] The downsampling interval k is determined based on the ratio of the target sampling rate to the original sampling rate. The calculation formula is as follows:

[0015]

[0016] The downsampled electromyographic data were extracted using a slicing operation:

[0017]

[0018] All other modal data are uniformly converted to a second-level timestamp format and stored using the naming rule "name_serial number.csv".

[0019] Preferably, the preprocessing in step S3 includes:

[0020] A bandpass filter is used to remove noise outside the 20Hz to 490Hz range. The transfer function is:

[0021]

[0022] in, and The filter order is ;

[0023] A notch filter with a center frequency of 50Hz is used to eliminate power frequency interference. The transfer function is:

[0024]

[0025] in, Q=30;

[0026] Timestamp alignment includes:

[0027] The original timestamps of each modality's data are normalized to their offset relative to the starting timestamp, using the following formula:

[0028]

[0029] And through linear interpolation or pruning operations, the index offset of the sample extraction time window is calculated as follows:

[0030]

[0031] in, For window size, The sampling frequency;

[0032] The dataset is divided into a training set of 70%, a validation set of 10%, and a test set of 20%, and the sample labels include specific feature values ​​of joint angle and pressure.

[0033] Preferably, the structure of the inverse residual module in step S4 includes: a 1×1 convolutional expansion layer, a 3×3 depth-separable convolutional layer, a ReLU6 activation function, a 1×1 convolutional compression layer, and residual connections;

[0034] The electromyography channel has 8 input channels, which output 32 channels of features after passing through 3 inverted residual modules; the joint angle channel has 15 input channels, which output 32 channels of features; the pressure channel has 8 input channels, which output 64 channels of features; and the temperature channel expands the 1-dimensional input to 32 dimensions through a fully connected layer.

[0035] The feature fusion method of the multimodal neural network architecture is as follows: the features output from the electromyography channel, joint angle channel and pressure channel are compressed into a 1×1 vector by global average pooling, and then concatenated with the fully connected output of the temperature channel. Multimodal feature fusion and classification are then achieved through a fully connected layer.

[0036] The depthwise separable convolutional module includes: a 3×3 depthwise convolutional layer, a batch normalization layer, a ReLU6 activation function, and a 1×1 pointwise convolutional layer, which are used to reduce the number of network parameters.

[0037] Preferably, the weight adjustment of the dynamic loss function in step S5 includes:

[0038] Weights of joint angle regression task Decline by exponential rate:

[0039]

[0040] Weight of the stress regression task Increase gradually:

[0041]

[0042] And the base weights of the stress task are dynamically adjusted using the sigmoid function:

[0043]

[0044] The total loss function is:

[0045] ;

[0046] The hyperparameter adjustment in step S6 includes learning rate, batch size, and dynamic loss function. The optimization aims to minimize the validation set loss and improve the accuracy of hand motion prediction.

[0047] The present invention also provides a multimodal neural network architecture design system for neural interface chips, the system comprising the following modules:

[0048] Module M1: Timestamps multimodal data samples via a host computer. The multimodal data includes fingertip pressure, palm pressure, arm electromyography signals, hand joint angles, and fingertip temperature.

[0049] Module M2: Converts data files of different modalities into CSV format and standardizes the data format through timestamp alignment and downsampling.

[0050] Module M3: Preprocesses multimodal data using bandpass filtering, notch filtering, and lowpass filtering, and extracts sample data within the time window through timestamp normalization and interpolation alignment;

[0051] Module M4: Based on MobileNetV2, a lightweight multimodal neural network architecture is designed, which includes parallel processing electromyography channels, joint angle channels, pressure channels and temperature channels. Each channel is fused after feature extraction through the inverse residual module.

[0052] Module M5: Constructs a dynamic loss function, which dynamically adjusts the weights of joint angle regression, stress regression, and classification tasks through exponential decay and sigmoid functions to achieve weighted loss fusion;

[0053] Module M6: Adjusts network hyperparameters and trains the model, outputting the prediction results of hand joint angles and fingertip pressure.

[0054] Preferably, the downsampling process in module M2 includes:

[0055] The downsampling interval k is determined based on the ratio of the target sampling rate to the original sampling rate. The calculation formula is as follows:

[0056]

[0057] The downsampled electromyographic data were extracted using a slicing operation:

[0058]

[0059] All other modal data are uniformly converted to a second-level timestamp format and stored using the naming rule "name_serial number.csv".

[0060] Preferably, the preprocessing in module M3 includes:

[0061] A bandpass filter is used to remove noise outside the 20Hz to 490Hz range. The transfer function is:

[0062]

[0063] in, and The filter order is ;

[0064] A notch filter with a center frequency of 50Hz is used to eliminate power frequency interference. The transfer function is:

[0065]

[0066] in, Q=30;

[0067] Timestamp alignment includes:

[0068] The original timestamps of each modality's data are normalized to their offset relative to the starting timestamp, using the following formula:

[0069]

[0070] And through linear interpolation or pruning operations, the index offset of the sample extraction time window is calculated as follows:

[0071]

[0072] in, For window size, The sampling frequency;

[0073] The dataset is divided into a training set of 70%, a validation set of 10%, and a test set of 20%, and the sample labels include specific feature values ​​of joint angle and pressure.

[0074] Preferably, the structure of the inverse residual module in module M4 includes: a 1×1 convolutional expansion layer, a 3×3 depth-separable convolutional layer, a ReLU6 activation function, a 1×1 convolutional compression layer, and residual connections;

[0075] The electromyography channel has 8 input channels, which output 32 channels of features after passing through 3 inverted residual modules; the joint angle channel has 15 input channels, which output 32 channels of features; the pressure channel has 8 input channels, which output 64 channels of features; and the temperature channel expands the 1-dimensional input to 32 dimensions through a fully connected layer.

[0076] The feature fusion method of the multimodal neural network architecture is as follows: the features output from the electromyography channel, joint angle channel and pressure channel are compressed into a 1×1 vector by global average pooling, and then concatenated with the fully connected output of the temperature channel. Multimodal feature fusion and classification are then achieved through a fully connected layer.

[0077] The depthwise separable convolutional module includes: a 3×3 depthwise convolutional layer, a batch normalization layer, a ReLU6 activation function, and a 1×1 pointwise convolutional layer, which are used to reduce the number of network parameters.

[0078] Preferably, the weight adjustment of the dynamic loss function in module M5 includes:

[0079] Weights of joint angle regression task Decline by exponential rate:

[0080]

[0081] Weight of the stress regression task Increase gradually:

[0082]

[0083] And the base weights of the stress task are dynamically adjusted using the sigmoid function:

[0084]

[0085] The total loss function is:

[0086] ;

[0087] The hyperparameter adjustment in module M6 includes learning rate, batch size, and dynamic loss function. The optimization aims to minimize the validation set loss and improve the accuracy of hand motion prediction.

[0088] Compared with the prior art, the present invention has the following beneficial effects:

[0089] 1. This invention effectively solves the problems of differences in sampling rate, format and noise level of heterogeneous data such as electromyography signals, pressure, joint angles and temperature by using unified format conversion, precise timestamp alignment and multimodal data fusion technology; the use of preprocessing methods such as bandpass filtering, notch filtering and low-pass filtering significantly improves signal quality, ensures data consistency and validity, and lays a high-quality data foundation for subsequent model training.

[0090] 2. This invention features a lightweight neural network architecture based on MobileNetV2, which combines depthwise separable convolution and inverse residual modules to significantly reduce the number of network parameters and computational complexity. The modular design is optimized for different modal data (such as electromyography channels, pressure channels, etc.), ensuring feature extraction efficiency while meeting the stringent requirements of neural interface chips for low power consumption and real-time performance.

[0091] 3. This invention innovatively proposes a phased dynamic loss function, which dynamically adjusts the weights of joint angle regression, stress regression, and classification tasks through exponential decay and the Sigmoid function. This design effectively balances the target conflict in multi-task learning, alleviates the overfitting problem, and significantly improves the generalization performance and prediction stability of the model in complex scenarios.

[0092] 4. Through multimodal feature fusion and efficient inference of lightweight networks, the system can accurately capture subtle features of hand grasping actions (such as pressure distribution and joint angle changes), and achieve low latency and high accuracy motion prediction with a limited number of parameters, providing reliable technical support for neurorehabilitation training and intelligent prosthetic control.

[0093] 5. This invention supports multiple sensor data formats (such as C3D and CSV) and different sampling rates (5 Hz to 1000 Hz), achieving cross-platform data compatibility through interpolation, cropping, and timestamp normalization techniques. This architecture can be flexibly extended to other multimodal application scenarios (such as brain-computer interfaces and motion analysis), possessing broad market application prospects. Attached Figure Description

[0094] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0095] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0096] Figure 2 This is a diagram of a lightweight multimodal neural network architecture in an embodiment of the present invention;

[0097] Figure 3 This is a schematic diagram of the convolution and residual modules in an embodiment of the present invention. Detailed Implementation

[0098] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0099] Example 1:

[0100] Reference Figure 1 and Figure 2 According to the present invention, a multimodal neural network architecture design method for a neural interface chip is provided, the method comprising the following steps:

[0101] Step S1: The multimodal data samples are timestamped by the host computer. The multimodal data includes fingertip pressure, palm pressure, arm electromyography signals, hand joint angles, and fingertip temperature.

[0102] Step S2: Convert data files of different modalities into CSV format and standardize the data format through timestamp alignment and downsampling.

[0103] Downsampling includes:

[0104] The downsampling interval k is determined based on the ratio of the target sampling rate to the original sampling rate. The calculation formula is as follows:

[0105]

[0106] The downsampled electromyographic data were extracted using a slicing operation:

[0107]

[0108] All other modal data are uniformly converted to a second-level timestamp format and stored using the naming rule "name_serial number.csv".

[0109] Step S3: Preprocess the multimodal data using bandpass filtering, notch filtering, and low-pass filtering, and extract sample data within the time window by timestamp normalization and interpolation alignment;

[0110] Preprocessing includes:

[0111] A bandpass filter is used to remove noise outside the 20Hz to 490Hz range. The transfer function is:

[0112]

[0113] in, and The filter order is ;

[0114] A notch filter with a center frequency of 50Hz is used to eliminate power frequency interference. The transfer function is:

[0115]

[0116] in, Q=30;

[0117] Timestamp alignment includes:

[0118] The original timestamps of each modality's data are normalized to their offset relative to the starting timestamp, using the following formula:

[0119]

[0120] And through linear interpolation or pruning operations, the index offset of the sample extraction time window is calculated as follows:

[0121]

[0122] in, For window size, The sampling frequency;

[0123] The dataset is divided into a training set of 70%, a validation set of 10%, and a test set of 20%, and the sample labels include specific feature values ​​of joint angle and pressure. The specific feature values ​​include statistics such as period, mean, standard deviation, maximum value, minimum value, and time distribution density of joint angle and pressure data within the set sliding time window. The statistics can be calculated from the data distributed according to the time series within the sliding window. The features of the data are represented by these data quantities such as period, mean, and maximum value.

[0124] Step S4: Design a lightweight multimodal neural network architecture based on MobileNetV2, which includes parallel processing of electromyography channels, joint angle channels, pressure channels, and temperature channels. Each channel is fused after feature extraction through the inverse residual module. The structure of the inverse residual module includes: a 1×1 convolutional expansion layer, a 3×3 depthwise separable convolutional layer, a ReLU6 activation function, a 1×1 convolutional compression layer, and residual connections.

[0125] The electromyography channel has 8 input channels, which output 32 channels of features after passing through 3 inverted residual modules; the joint angle channel has 15 input channels, which output 32 channels of features; the pressure channel has 8 input channels, which output 64 channels of features; and the temperature channel expands the 1-dimensional input to 32 dimensions through a fully connected layer.

[0126] The feature fusion method of the multimodal neural network architecture is as follows: the features output from the electromyography channel, joint angle channel and pressure channel are compressed into a 1×1 vector by global average pooling, and then concatenated with the fully connected output of the temperature channel. Multimodal feature fusion and classification are then achieved through a fully connected layer.

[0127] The depthwise separable convolutional module includes: a 3×3 depthwise convolutional layer, a batch normalization layer, a ReLU6 activation function, and a 1×1 pointwise convolutional layer, which are used to reduce the number of network parameters.

[0128] Step S5: Construct a dynamic loss function, dynamically adjusting the weights of joint angle regression, stress regression, and classification tasks through exponential decay and the sigmoid function to achieve weighted loss fusion; the weight adjustment of the dynamic loss function includes:

[0129] Weights of joint angle regression task Decline by exponential rate:

[0130]

[0131] Weight of the stress regression task Increase gradually:

[0132]

[0133] And the base weights of the stress task are dynamically adjusted using the sigmoid function:

[0134]

[0135] The total loss function is:

[0136] ;

[0137] Step S6: Adjust the network hyperparameters and train the model, outputting the predicted results of hand joint angles and fingertip pressure. Hyperparameter adjustment includes learning rate, batch size, and dynamic loss function. The optimization aims to minimize the validation set loss and improve the accuracy of hand motion prediction.

[0138] The present invention also provides a multimodal neural network architecture design system for neural interface chips. The multimodal neural network architecture design system for neural interface chips can be implemented by executing the process steps of the multimodal neural network architecture design method for neural interface chips. That is, those skilled in the art can understand the multimodal neural network architecture design method for neural interface chips as a preferred embodiment of the multimodal neural network architecture design system for neural interface chips.

[0139] Example 2:

[0140] The present invention also provides a multimodal neural network architecture design system for neural interface chips, the system comprising the following modules:

[0141] Module M1: Timestamps multimodal data samples via a host computer. The multimodal data includes fingertip pressure, palm pressure, arm electromyography signals, hand joint angles, and fingertip temperature.

[0142] Module M2: Converts data files of different modalities into CSV format and standardizes the data format through timestamp alignment and downsampling.

[0143] Downsampling includes:

[0144] The downsampling interval k is determined based on the ratio of the target sampling rate to the original sampling rate. The calculation formula is as follows:

[0145]

[0146] The downsampled electromyographic data were extracted using a slicing operation:

[0147]

[0148] All other modal data are uniformly converted to a second-level timestamp format and stored using the naming rule "name_serial number.csv".

[0149] Module M3: Preprocesses multimodal data using bandpass filtering, notch filtering, and lowpass filtering, and extracts sample data within the time window through timestamp normalization and interpolation alignment;

[0150] Preprocessing includes:

[0151] A bandpass filter is used to remove noise outside the 20Hz to 490Hz range. The transfer function is:

[0152]

[0153] in, and The filter order is ;

[0154] A notch filter with a center frequency of 50Hz is used to eliminate power frequency interference. The transfer function is:

[0155]

[0156] in, Q=30;

[0157] Timestamp alignment includes:

[0158] The original timestamps of each modality's data are normalized to their offset relative to the starting timestamp, using the following formula:

[0159]

[0160] And through linear interpolation or pruning operations, the index offset of the sample extraction time window is calculated as follows:

[0161]

[0162] in, For window size, The sampling frequency;

[0163] The dataset is divided into a training set of 70%, a validation set of 10%, and a test set of 20%, and the sample labels include specific feature values ​​of joint angle and pressure.

[0164] Module M4: Based on MobileNetV2, a lightweight multimodal neural network architecture is designed, which includes parallel processing of electromyography channels, joint angle channels, pressure channels and temperature channels. Each channel is fused after feature extraction through the inverse residual module. The structure of the inverse residual module includes: a 1×1 convolutional expansion layer, a 3×3 depthwise separable convolutional layer, a ReLU6 activation function, a 1×1 convolutional compression layer and residual connections.

[0165] The electromyography channel has 8 input channels, which output 32 channels of features after passing through 3 inverted residual modules; the joint angle channel has 15 input channels, which output 32 channels of features; the pressure channel has 8 input channels, which output 64 channels of features; and the temperature channel expands the 1-dimensional input to 32 dimensions through a fully connected layer.

[0166] The feature fusion method of the multimodal neural network architecture is as follows: the features output from the electromyography channel, joint angle channel and pressure channel are compressed into a 1×1 vector by global average pooling, and then concatenated with the fully connected output of the temperature channel. Multimodal feature fusion and classification are then achieved through a fully connected layer.

[0167] The depthwise separable convolutional module includes: a 3×3 depthwise convolutional layer, a batch normalization layer, a ReLU6 activation function, and a 1×1 pointwise convolutional layer, which are used to reduce the number of network parameters.

[0168] Module M5: Constructs a dynamic loss function, which dynamically adjusts the weights of joint angle regression, stress regression, and classification tasks through exponential decay and sigmoid functions to achieve weighted loss fusion;

[0169] The weight adjustment of the dynamic loss function includes:

[0170] Weights of joint angle regression task Decline by exponential rate:

[0171]

[0172] Weight of the stress regression task Increase gradually:

[0173]

[0174] And the base weights of the stress task are dynamically adjusted using the sigmoid function:

[0175]

[0176] The total loss function is:

[0177] ;

[0178] Module M6: Adjusts network hyperparameters and trains the model, outputting predictions of hand joint angles and fingertip pressure. Hyperparameter tuning includes adjusting the learning rate, batch size, and dynamic loss function. The optimization aims to minimize the validation set loss and improve the accuracy of hand motion prediction.

[0179] Example 3:

[0180] This invention provides a multimodal neural network architecture design for neural interface chips, comprising: Step S1: Labeling multimodal data (finger pressure, palm pressure, arm electromyography signals, hand joint angles, fingertip temperature) samples using a host computer; Step S2: Processing data samples, aligning and converting different modal data into a unified format; Step S3: Preprocessing different modal data using filtering techniques, performing sample segmentation and dataset partitioning to generate dataset files; Step S4: Designing a lightweight multimodal neural network architecture based on MobileNetV2, adapting to multimodal data, and ensuring that the number of network parameters meets the requirements of neural interface chips; Step S5: Designing a staged loss function based on the multimodal neural network architecture, using weighted summation to fuse the objective functions of three tasks: joint angle regression, fingertip pressure regression, and stage prediction; Step S6: Adjusting hyperparameters within the existing neural network training framework to obtain the optimal hand movement prediction result. This invention can accurately predict hand grasping movements with a limited number of network parameters and is suitable for lightweight neural interface chips.

[0181] This invention designs a lightweight multimodal neural network architecture that efficiently processes and fuses data from multiple modalities, including electromyography (EMG), joint angles, pressure, and fingertip temperature. This multimodal data fusion method more comprehensively reflects the characteristics of hand movements, providing more accurate feedback for neurorehabilitation training and significantly improving the accuracy and real-time performance of movement recognition. This invention proposes a dynamic loss function that dynamically adjusts the weights of different tasks during training. Dynamically adjusting weights at time steps effectively mitigates overfitting and improves the model's generalization ability. This dynamic adjustment mechanism allows the model to better balance the importance of different tasks during training, thereby improving prediction accuracy and stability. The multimodal neural network architecture of this invention is highly adaptable, capable of adapting to different data formats and sampling rates. Through timestamp alignment and data format conversion, it can process data from different sensors, exhibiting broad applicability. This invention is suitable for various application scenarios, including neurorehabilitation and intelligent prosthetics. By fusing multimodal data, it can provide more accurate predictions and feedback for these fields, demonstrating broad application prospects.

[0182] This invention provides a multimodal neural network architecture design for neural interface chips, addressing the challenges faced by neural interface chips in processing multimodal data and overcoming the shortcomings of traditional methods in data fusion, computational resource limitations, and model generalization capabilities. (Refer to...) Figure 1 As shown, the method specifically includes:

[0183] Step S1: Use the host computer to label the multimodal data samples;

[0184] Step S1 specifically includes:

[0185] Different modalities of data, such as fingertip pressure, palm pressure, arm electromyography signals, hand joint angles, and fingertip temperature, originate from different sensors. Different acquisition systems are used for data acquisition and export, resulting in timestamped data. A host computer is used to timestamp each data sample, providing a standard for time alignment and data segmentation of different modalities.

[0186] Step S2: Process the data samples, organize and align the data of different modalities, and convert them into a uniform format;

[0187] Step S2 specifically includes:

[0188] Data files exported by any modal acquisition software vary in format and data volume, so it is necessary to uniformly convert the data format and align the timestamps.

[0189] For electromyography (EMG) signal files, the C3D format file must first be converted to a CSV format file, retaining the valid information columns. Additionally, depending on the EMG signal settings, it's necessary to choose whether to downsample the data.

[0190] Let the original electromyography data be Its shape is [n, c], where n is the number of original sampling points and c is the number of channels. Original sampling rate The target sampling rate is .

[0191] First, calculate the downsampling interval k:

[0192]

[0193] Then, a sample is extracted from every k points in the original data to obtain the downsampled data. :

[0194]

[0195] in, Indicates from Take one row every k rows, keeping the columns unchanged.

[0196] Finally, the downsampled data The shape is [m,c], where This is the number of sampling points after downsampling.

[0197] For other modal data files, they are all converted to CSV format, while retaining valid data channels.

[0198] In any modal CSV file, the timestamp format needs to be adjusted to a format such as "2024-12-31 11:25:22.654294", with the seconds accurate to six decimal places. Furthermore, if the timestamp in the acquisition software cannot be accurately formatted uniformly, the time interval will be divided evenly according to the sampling frequency.

[0199] Finally, name all modal files uniformly as "name_number.csv" and copy them to the project directory.

[0200] Step S3: Use filtering techniques to preprocess data of different modalities, and perform sample segmentation and dataset partitioning to generate dataset files;

[0201] Step S3 specifically includes:

[0202] The acquired multimodal data is preprocessed and filtered to remove noise and extract the effective signal.

[0203] For electromyographic signals, bandpass filtering and notch filtering can be performed, as shown below:

[0204] Bandpass filtering removes low-frequency and high-frequency noise, with a frequency range of [frequency range missing]. and The filter order is The filter's transfer function is:

[0205]

[0206] in, The signal frequency is denoted by . The electromyographic signal processed by the bandpass filter can effectively remove power frequency interference and other noise.

[0207] Band-stop filtering (notch filtering) can further remove noise at specific frequencies (such as 50 Hz power frequency interference), with a center frequency of _____. The quality factor of the filter is Q=30. The transfer function of the filter is:

[0208]

[0209] For joint angle and pressure data, low-pass filtering can be used for smoothing. The cutoff frequency of the low-pass filter is... The filter order is The filter's transfer function is:

[0210]

[0211] In multimodal data processing, accurate timestamp alignment and sample extraction are crucial steps to ensure data consistency and validity. This invention implements timestamp processing and sample extraction through the following detailed steps:

[0212] The purpose of timestamp preprocessing is to unify the timestamps of different modalities onto a standard time base for subsequent sample extraction and alignment. First, timestamp normalization is performed. For each modality (e.g., electromyography, joint angle, pressure, and fingertip temperature data), its timestamp column is extracted. Assume the timestamp column is in seconds and increments from zero. The timestamps are normalized using the following formula:

[0213]

[0214] in, It is the value of the i-th timestamp. It is the start timestamp. It is the sampling frequency of the modality data (e.g., the sampling frequency of electromyography data is 1000 Hz, the sampling frequency of joint angle and pressure data is 100 Hz, and the sampling frequency of temperature data is 5 Hz).

[0215] Next, timestamp alignment is performed. To ensure accurate alignment of timestamps from different modalities, interpolation or cropping is used to unify the timestamps of all modalities onto a common time base. The specific method is as follows:

[0216] For cases where timestamps do not match, the timestamps are aligned to the closest common time point using linear interpolation or nearest neighbor interpolation methods.

[0217] If the timestamp ranges are inconsistent, the redundant parts are trimmed to ensure that the time ranges of all modal data are consistent.

[0218] Sample extraction is based on timestamps in the labeled file, extracting samples from the preprocessed multimodal data. The labeled file records the time points of key events, which indicate the start and end positions of the samples. First, timestamp processing is performed, i.e., reading the timestamps of key events from the labeled file to indicate the start and end positions of the samples. Assume the labeled file records time points... , representing the start, middle, and end time points of the sample, respectively.

[0219] Next, timestamp lookup is performed for each marked time point. The nearest timestamp index is found in the preprocessed multimodal data. The nearest timestamp index is calculated using the following formula:

[0220]

[0221] in, These are timestamps in the preprocessed multimodal data. It represents the time point in the marker file.

[0222] Next, sample extraction is performed. Based on the found timestamp index, samples are extracted from the multimodal data. For each labeled time point... Extract data within the corresponding time window. The window size is [size missing]. (e.g., 100 ms) The extracted sample data includes electromyography (EMG) data, joint angle data, pressure data, and fingertip temperature data. The specific extraction formula is as follows:

[0223]

[0224] in, It is the index offset corresponding to the window size, and the calculation formula is:

[0225]

[0226] Finally, the extracted sample data are concatenated into a complete sample, and corresponding labels are generated. The label data typically includes specific feature values ​​of joint angle and pressure data, which are used for subsequent model training and validation.

[0227] The extracted samples were divided into training, validation, and test sets, with proportions of 70%, 10%, and 20%, respectively. The resulting datasets were then saved as training, validation, and test set files for subsequent model training and validation.

[0228] Step S4: Design a lightweight multimodal neural network architecture based on MobileNetV2 to ensure that the number of network parameters meets the requirements of the neural interface chip;

[0229] Step S4 specifically includes:

[0230] This invention proposes a lightweight multimodal neural network architecture based on MobileNetV2 for efficiently processing and fusing data from multiple modalities, including electromyography (EMG) data, joint angle data, pressure data, and fingertip temperature data. This architecture, through modular design, ensures efficient feature extraction and data processing while adapting to limited computing resources. The specific architecture design is as follows:

[0231] Convolution module, see Figure 3 As shown, convolutional modules serve as the basic building blocks of the network, used to extract features from the input data. Each convolutional module includes:

[0232] (1) Convolutional layer: Features are extracted using two-dimensional convolution operation. The kernel size is 3×3, the stride is 1, and the padding is 1.

[0233] (2) Batch normalization: Normalize the output of the convolutional layer to stabilize the training process.

[0234] (3) ReLU6 activation function: Apply the ReLU6 activation function to the normalized data to limit the output range to [0,6].

[0235] Depthwise separable convolutional modules, see reference Figure 3 As shown, the convolution operation is separated, reducing computational cost and the number of parameters. This module includes:

[0236] (1) Depthwise convolution: Perform convolution operation on each input channel separately. The kernel size is 3×3, the stride is 1, and the padding is 1.

[0237] (2) Batch normalization: Normalize the output of depthwise convolution.

[0238] (3) ReLU6 activation function: Apply the ReLU6 activation function.

[0239] (4) Pointwise convolution: Use a 1×1 convolution kernel to expand or compress channels.

[0240] (5) Batch normalization: Normalize the output of pointwise convolution.

[0241] Inverted residual module, refer to Figure 3 As shown, combining expansion and compression operations improves the efficiency of feature extraction. This module includes:

[0242] (1) 1×1 convolution expansion: Expand the number of input channels using a 1×1 convolution kernel.

[0243] (2) Depth convolution: Perform depth convolution operation on the expanded features.

[0244] (3) ReLU6 activation function: Apply the ReLU6 activation function.

[0245] (4) 1×1 convolution compression: Use a 1×1 convolution kernel to compress the number of channels.

[0246] (5) Batch normalization: Normalize the compressed features.

[0247] (6) Residual connection: If the number of input and output channels is the same and the step size is 1, then add a residual connection.

[0248] Multimodal neural network architecture, refer to Figure 2 As shown, efficient feature extraction and classification are achieved by fusing data from different modalities. The architecture includes the following modules:

[0249] (1) Electromyography channels: Three inverted residual modules are used to process electromyography data, with 8 input channels and 32 output channels. The feature map is compressed to 1×1 by global average pooling.

[0250] (2) Joint angle channels: Three inverted residual modules are used to process joint angle data, with 15 input channels and 32 output channels. The feature map is compressed to 1×1 by global average pooling.

[0251] (3) Pressure Channel: Three inverted residual modules are used to process pressure data, with 8 input channels and 64 output channels. The feature map is compressed to 1×1 by global average pooling.

[0252] (4) Temperature channel: Three fully connected layers are used to process temperature data, with an input dimension of 1 and an output dimension of 32.

[0253] (5) Feature fusion: The processed electromyography, joint angle, pressure and temperature features are spliced ​​together.

[0254] (6) Fully connected layer: Feature fusion and classification are performed through multiple fully connected layers, and the target dimension is finally output.

[0255] Step S5: Design a staged loss function based on the multimodal neural network architecture, and use weighted summation to fuse the objective functions of the three tasks;

[0256] Step S5 specifically includes:

[0257] This invention proposes a dynamic loss function to dynamically adjust the weights of different tasks during training, thereby mitigating overfitting and improving the model's generalization ability. This loss function combines positional loss, stress loss, and classification loss, and dynamically adjusts the weights over time step t. The specific design is as follows:

[0258] Dynamic weight adjustment includes position loss weight and pressure loss weight, with the initial position loss weight being... And dynamically adjust using the following formula:

[0259]

[0260] in, It is a parameter that controls the dynamic adjustment rate. It is the time step of the current epoch or stage.

[0261] Regarding the pressure loss weight, the initial pressure loss weight is: And dynamically adjust using the following formula:

[0262]

[0263] Meanwhile, the base weight of the stress task is Further adjustments can be made using the following formula:

[0264]

[0265] in, It is the sigmoid function.

[0266] Position loss is calculated using mean squared error (MSE), which is the loss for the joint angle regression task. Stress loss is calculated using mean absolute error (MAE), which is the loss for the stress regression task, to prevent overfitting and is adjusted via dynamic weights.

[0267]

[0268] The classification loss is calculated using cross-entropy loss for the classification task.

[0269] The total loss is obtained by weighted summation of the three types of losses:

[0270]

[0271] By designing the dynamic loss function described above, the weights of different tasks can be effectively adjusted during training, thereby improving the model's generalization ability and prediction accuracy.

[0272] Step S6: Under the existing neural network training framework, adjust the hyperparameters to obtain the optimal hand movement prediction results.

[0273] Step S6 specifically includes:

[0274] Based on existing datasets, multimodal neural network architectures, and loss functions, parameters are adjusted to make the network converge, resulting in a prediction model. Given input, prediction results for hand joint angles and fingertip pressure are obtained.

[0275] This invention provides a multimodal neural network architecture design for neural interface chips. Addressing the challenges faced by neural interface chips in processing multimodal data (such as electromyography, joint angles, pressure, and temperature), the multimodal data fusion task is reformulated as a lightweight neural network architecture design problem. By introducing a dynamic loss function and a lightweight network structure, feature extraction and fusion of different modalities are incorporated into a unified framework. Position loss, pressure loss, and classification loss are expressed as a weighted summation objective function, where the weights are dynamically adjusted over time steps to achieve a balance between tasks. Simultaneously, the network architecture design considers the computational resource limitations of neural interface chips, ensuring efficient model operation under low power consumption and high accuracy requirements. This invention also develops a lightweight multimodal neural network architecture based on MobileNetV2, which can automatically optimize network parameters to achieve high efficiency in feature extraction and data processing. The solution obtained by this invention can accurately predict hand grasping actions with a limited number of network parameters, improving the model's generalization ability and prediction accuracy, and has high application value.

[0276] Those skilled in the art can understand this embodiment as a more specific description of Embodiment 1 and Embodiment 2.

[0277] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0278] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for designing a multimodal neural network architecture for a neural interface chip, characterized in that, The method includes the following steps: Step S1: The multimodal data samples are timestamped by the host computer. The multimodal data includes fingertip pressure, palm pressure, arm electromyography signals, hand joint angles, and fingertip temperature. Step S2: Convert data files of different modalities into CSV format and standardize the data format through timestamp alignment and downsampling. Step S3: Preprocess the multimodal data using bandpass filtering, notch filtering, and low-pass filtering, and extract sample data within the time window by timestamp normalization and interpolation alignment; Step S4: Design a lightweight multimodal neural network architecture based on MobileNetV2, which includes parallel processing of electromyography channels, joint angle channels, pressure channels and temperature channels. Each channel extracts features through an inverse residual module and then fuses them. Step S5: Construct a dynamic loss function, and dynamically adjust the weights of joint angle regression, stress regression and classification task through exponential decay and sigmoid function to achieve weighted loss fusion; Step S6: Adjust the network hyperparameters and train the model, outputting the prediction results of hand joint angles and fingertip pressure; The preprocessing in step S3 includes: For electromyographic signals, a bandpass filter is used to remove noise outside the 20Hz to 490Hz range. The transfer function is: in, and The filter order is , The signal frequency; A notch filter with a center frequency of 50Hz is used to eliminate power frequency interference. The transfer function is: in, , ; For joint angle and pressure data, a low-pass filter is applied for smoothing; the cutoff frequency of the low-pass filter is... The filter order is The filter's transfer function is: Timestamp alignment includes: The original timestamps of each modality's data are normalized to their offset relative to the starting timestamp, using the following formula: in, It is the value of the i-th timestamp. It is the start timestamp. This is the sampling frequency of the modal data; And through linear interpolation or pruning operations, the index offset of the sample extraction time window is calculated as follows: in, For window size, The sampling frequency; The dataset is divided into a training set of 70%, a validation set of 10%, and a test set of 20%, and the sample labels include specific feature values ​​of joint angle and pressure.

2. The multimodal neural network architecture design method for neural interface chips according to claim 1, characterized in that, The downsampling process in step S2 includes: The downsampling interval k is determined based on the ratio of the target sampling rate to the original sampling rate. The calculation formula is as follows: The downsampled electromyographic data were extracted using a slicing operation: All other modal data are uniformly converted to a second-level timestamp format and stored using the naming rule "name_serial number.csv".

3. The multimodal neural network architecture design method for neural interface chips according to claim 1, characterized in that, The structure of the inverse residual module in step S4 includes: a 1×1 convolutional expansion layer, a 3×3 depthwise separable convolutional layer, a ReLU6 activation function, a 1×1 convolutional compression layer, and residual connections. The electromyography channel has 8 input channels, which output 32 channels of features after passing through 3 inverted residual modules; the joint angle channel has 15 input channels, which output 32 channels of features; the pressure channel has 8 input channels, which output 64 channels of features; and the temperature channel expands the 1-dimensional input to 32 dimensions through a fully connected layer. The feature fusion method of the multimodal neural network architecture is as follows: the features output from the electromyography channel, joint angle channel and pressure channel are compressed into a 1×1 vector by global average pooling, and then concatenated with the fully connected output of the temperature channel. Multimodal feature fusion and classification are then achieved through a fully connected layer. The depthwise separable convolutional module includes: a 3×3 depthwise convolutional layer, a batch normalization layer, a ReLU6 activation function, and a 1×1 pointwise convolutional layer, which are used to reduce the number of network parameters.

4. The multimodal neural network architecture design method for neural interface chips according to claim 1, characterized in that, The weight adjustment of the dynamic loss function in step S5 includes: Weights of joint angle regression task With exponential decay, the initial position loss weight is... : in, It is a parameter that controls the dynamic adjustment rate. It is the time step of the current epoch or stage; Weight of the stress regression task The pressure loss is increased gradually, with the initial pressure loss weight being... : The base weights of the stress task are dynamically adjusted using the sigmoid function. The base weights of the stress task are... : in, It is the sigmoid function; The total loss function is: ; The hyperparameter adjustment in step S6 includes learning rate, batch size, and dynamic loss function. The optimization aims to minimize the validation set loss and improve the accuracy of hand motion prediction.

5. A multimodal neural network architecture design system for neural interface chips, characterized in that, The system includes the following modules: Module M1: Timestamps multimodal data samples via a host computer. The multimodal data includes fingertip pressure, palm pressure, arm electromyography signals, hand joint angles, and fingertip temperature. Module M2: Converts data files of different modalities into CSV format and standardizes the data format through timestamp alignment and downsampling. Module M3: Preprocesses multimodal data using bandpass filtering, notch filtering, and lowpass filtering, and extracts sample data within the time window through timestamp normalization and interpolation alignment; Module M4: Based on MobileNetV2, a lightweight multimodal neural network architecture is designed, which includes parallel processing electromyography channels, joint angle channels, pressure channels and temperature channels. Each channel is fused after feature extraction through the inverse residual module. Module M5: Constructs a dynamic loss function, which dynamically adjusts the weights of joint angle regression, stress regression, and classification tasks through exponential decay and sigmoid functions to achieve weighted loss fusion; Module M6: Adjusts network hyperparameters and trains the model, outputting the prediction results of hand joint angles and fingertip pressure; The preprocessing in module M3 includes: For electromyographic signals, a bandpass filter is used to remove noise outside the 20Hz to 490Hz range. The transfer function is: in, and The filter order is , The signal frequency; A notch filter with a center frequency of 50Hz is used to eliminate power frequency interference. The transfer function is: in, , ; For joint angle and pressure data, a low-pass filter is applied for smoothing; the cutoff frequency of the low-pass filter is... The filter order is The filter's transfer function is: Timestamp alignment includes: The original timestamps of each modality's data are normalized to their offset relative to the starting timestamp, using the following formula: in, It is the first The value of a timestamp, It is the start timestamp. This is the sampling frequency of the modal data; And through linear interpolation or pruning operations, the index offset of the sample extraction time window is calculated as follows: in, For window size, The sampling frequency; The dataset is divided into a training set of 70%, a validation set of 10%, and a test set of 20%, and the sample labels include specific feature values ​​of joint angle and pressure.

6. The multimodal neural network architecture design system for neural interface chips according to claim 5, characterized in that, The downsampling process in module M2 includes: The downsampling interval k is determined based on the ratio of the target sampling rate to the original sampling rate. The calculation formula is as follows: The downsampled electromyographic data were extracted using a slicing operation: All other modal data are uniformly converted to a second-level timestamp format and stored using the naming rule "name_serial number.csv".

7. The multimodal neural network architecture design system for neural interface chips according to claim 5, characterized in that, The structure of the inverse residual module in module M4 includes: a 1×1 convolutional expansion layer, a 3×3 depthwise separable convolutional layer, a ReLU6 activation function, a 1×1 convolutional compression layer, and residual connections; The electromyography channel has 8 input channels, which output 32 channels of features after passing through 3 inverted residual modules; the joint angle channel has 15 input channels, which output 32 channels of features; the pressure channel has 8 input channels, which output 64 channels of features; and the temperature channel expands the 1-dimensional input to 32 dimensions through a fully connected layer. The feature fusion method of the multimodal neural network architecture is as follows: the features output from the electromyography channel, joint angle channel and pressure channel are compressed into a 1×1 vector by global average pooling, and then concatenated with the fully connected output of the temperature channel. Multimodal feature fusion and classification are then achieved through a fully connected layer. The depthwise separable convolutional module includes: a 3×3 depthwise convolutional layer, a batch normalization layer, a ReLU6 activation function, and a 1×1 pointwise convolutional layer, which are used to reduce the number of network parameters.

8. The multimodal neural network architecture design system for neural interface chips according to claim 5, characterized in that, The weight adjustment of the dynamic loss function in module M5 includes: Weights of joint angle regression task With exponential decay, the initial position loss weight is... : in, It is a parameter that controls the dynamic adjustment rate. It is the time step of the current epoch or stage; Weight of the stress regression task The pressure loss is increased gradually, with the initial pressure loss weight being... : The base weights of the stress task are dynamically adjusted using the sigmoid function. The base weights of the stress task are... : in, It is the sigmoid function; The total loss function is: ; The hyperparameter adjustment in module M6 includes learning rate, batch size, and dynamic loss function. The optimization aims to minimize the validation set loss and improve the accuracy of hand motion prediction.

Images