Multi-modal neural network architecture design method and system for neural interface chip

Abstract

The invention provides a multi-modal neural network architecture design method and system for a neural interface chip. The method comprises the following steps: performing timestamp marking on a multi-modal data sample through an upper computer; uniformly converting data files of different modalities into a CSV format, and realizing data format standardization through timestamp alignment and downsampling processing; preprocessing the multi-modal data, and extracting sample data in a time window through timestamp normalization and interpolation alignment; a lightweight multi-modal neural network architecture is designed based on MobileNetV2, and features of all channels are extracted through an inverse residual module and then fused; a dynamic loss function is constructed, and weights of joint angle regression, pressure regression and classification tasks are dynamically adjusted through exponential decay and a sigmoid function, so that weighted loss fusion is realized; network hyper-parameters are adjusted, the model is trained, and prediction results of the hand joint angle and the fingertip pressure are output. According to the method, the hand grabbing action can be accurately predicted under the limited network parameter quantity, and the method is suitable for a lightweight neural interface chip.

Description

Technical Field

[0001] The present invention relates to the interdisciplinary field of deep learning and neural interface technology, and in particular to a multimodal neural network architecture design method and system for neural interface chips. Background Art

[0002] With the rapid development of neuroscience and microelectronics, neural interface chips, as key technologies for connecting the nervous system with external devices, have become a research hotspot in fields such as brain-computer interfaces, neurorehabilitation, and intelligent prosthetics. Neural interface chips can acquire and process neural signals in real time, providing the foundation for precise control of external devices. However, traditional neural interface chips face significant challenges in processing multimodal neural signals (such as electromyography, pressure, joint angle, and temperature). The sampling rates, data formats, and noise levels of different modal data vary significantly, leading to complex data alignment and fusion. For example, electromyography signals are typically sampled at high frequencies in the kilohertz range, while temperature signals may only be sampled at a few hertz. This heterogeneity makes time synchronization and feature extraction inefficient. Furthermore, neural interface chips are limited by low-power and low-computing hardware, making it difficult to directly deploy traditional high-complexity neural networks, limiting their ability to process multimodal data in real time.

[0003] In the field of multimodal data fusion, traditional methods such as BP neural networks and RBF neural networks have certain nonlinear modeling capabilities, but their fixed structures and large number of parameters lead to high consumption of computing resources and sensitivity to noise. In recent years, lightweight networks (such as MobileNetV2) have demonstrated efficient feature extraction capabilities in image processing, but there are still adaptability issues when directly migrating to multimodal neural signal processing. For example, the temporal and spatial characteristics of multimodal data vary significantly, and a single network architecture cannot take into account the characteristics of each modality. At the same time, the design of static loss functions cannot balance the weight distribution of multi-task training, which can easily lead to overfitting or unstable model convergence. In addition, existing methods are inefficient in the data preprocessing stage. The format conversion, timestamp alignment, and noise filtering processes of different sensor data are cumbersome, further increasing system latency.

[0004] Multimodal fusion technology is of great value in scenarios such as hand movement prediction. By integrating the dynamic characteristics of electromyographic signals, tactile feedback of pressure signals, and the motion trajectory of joint angles, the accuracy and real-time performance of movement recognition can be significantly improved, providing more refined feedback for neurorehabilitation training. However, existing technologies still face bottlenecks such as weak model generalization capabilities and poor hardware adaptability. Therefore, there is an urgent need for a lightweight, highly robust multimodal neural network architecture to resolve the contradiction between data heterogeneity, resource constraints, and task diversity, and to promote the widespread application of neural interface chips in clinical and engineering fields. Summary of the Invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a multimodal neural network architecture design method and system for neural interface chips.

[0006] According to the present invention, a multimodal neural network architecture design method for a neural interface chip is provided, the method comprising the following steps: Step S1: Timestamping the multimodal data samples by the host computer, wherein the multimodal data includes fingertip pressure, palm pressure, arm electromyographic signal, hand joint angle and fingertip temperature; Step S2: Convert the data files of different modalities into CSV format and standardize the data format through timestamp alignment and downsampling. Step S3: pre-process the multimodal data using band-pass filtering, notch filtering, and low-pass filtering, and extract sample data within the time window through timestamp normalization and interpolation alignment; Step S4: Design a lightweight multimodal neural network architecture based on MobileNetV2, which includes parallel processing of electromyographic channels, joint angle channels, pressure channels, and temperature channels. Each channel is fused after feature extraction through the inverted residual module; Step S5: Construct a dynamic loss function, dynamically adjust the weights of joint angle regression, pressure regression, and classification tasks through exponential decay and sigmoid functions, and achieve weighted loss fusion; Step S6: Adjust the network hyperparameters and train the model to output the prediction results of hand joint angles and fingertip pressure.

[0007] Preferably, the downsampling process in step S2 includes: The downsampling interval k is determined based on the ratio of the target sampling rate to the original sampling rate. The calculation formula is:

[0008] And extract the downsampled EMG data through slicing operation:

[0009] The remaining modal data are uniformly converted into second-level timestamp format and stored using the "name_serial number.csv" naming rule.

[0010] Preferably, the preprocessing in step S3 includes: A bandpass filter is used to remove noise outside the range of 20 Hz to 490 Hz. The transfer function is:

[0011] in, and , the filter order is ; A notch filter with a center frequency of 50Hz is used to eliminate power frequency interference. The transfer function is:

[0012] in, , Q=30; Timestamp alignment includes: Normalize the original timestamp of each modal data to the offset relative to the start timestamp. The formula is:

[0013] And unified to a common time base through linear interpolation or clipping operations, the index offset of the sample extraction time window is calculated as:

[0014] in, is the window size, is the sampling frequency; The dataset is divided into 70% training set, 10% validation set, and 20% test set, and the sample labels contain specific feature values ​​of joint angle and pressure.

[0015] Preferably, the structure of the inverted residual module in step S4 includes: a 1×1 convolution expansion layer, a 3×3 depth-separable convolution layer, a ReLU6 activation function, a 1×1 convolution compression layer and a residual connection; The myoelectric channel input is 8 channels, and it outputs 32-channel features through three inverted residual modules; the joint angle channel input is 15 channels, and it outputs 32-channel features; the pressure channel input is 8 channels, and it outputs 64-channel features; the temperature channel expands the 1-dimensional input to 32 dimensions through a fully connected layer; The feature fusion method of the multimodal neural network architecture is as follows: the features output by the electromyography channel, joint angle channel, and pressure channel are compressed into a 1×1 vector through global average pooling, and then spliced ​​with the fully connected output of the temperature channel. After that, multimodal feature fusion and classification are achieved through a fully connected layer. The depthwise separable convolution module includes: 3×3 depthwise convolution layer, batch normalization layer, ReLU6 activation function and 1×1 point-by-point convolution layer to reduce the number of network parameters.

[0016] Preferably, the weight adjustment of the dynamic loss function in step S5 includes: Weights for the joint angle regression task Exponential decay:

[0017] Weight of stress regression task Incremental increase:

[0018] And dynamically correct the basic weight of the stress task through the sigmoid function:

[0019] The total loss function is: ; The hyperparameter adjustment in step S6 includes the learning rate, batch size and dynamic loss function. to minimize the validation set loss and improve the hand motion prediction accuracy.

[0020] The present invention also provides a multimodal neural network architecture design system for a neural interface chip, the system comprising the following modules: Module M1: Timestamps multimodal data samples through the host computer. The multimodal data includes fingertip pressure, palm pressure, arm electromyography, hand joint angles, and fingertip temperature. Module M2: Convert data files of different modalities into CSV format and standardize the data format through timestamp alignment and downsampling. Module M3: Preprocesses multimodal data using bandpass filtering, notch filtering, and low-pass filtering, and extracts sample data within the time window through timestamp normalization and interpolation alignment; Module M4: A lightweight multimodal neural network architecture is designed based on MobileNetV2, which includes parallel processing of myoelectric channels, joint angle channels, pressure channels, and temperature channels. Each channel is fused after feature extraction through the inverted residual module; Module M5: Constructs a dynamic loss function, dynamically adjusts the weights of joint angle regression, pressure regression, and classification tasks through exponential decay and sigmoid functions, and realizes weighted loss fusion; Module M6: Adjust network hyperparameters and train the model to output the prediction results of hand joint angles and fingertip pressure.

[0021] Preferably, the downsampling process in the module M2 includes: The downsampling interval k is determined based on the ratio of the target sampling rate to the original sampling rate. The calculation formula is:

[0022] And extract the downsampled EMG data through slicing operation:

[0023] The remaining modal data are uniformly converted into second-level timestamp format and stored using the "name_serial number.csv" naming rule.

[0024] Preferably, the preprocessing in the module M3 includes: A bandpass filter is used to remove noise outside the range of 20 Hz to 490 Hz. The transfer function is:

[0025] in, and , the filter order is ; A notch filter with a center frequency of 50Hz is used to eliminate power frequency interference. The transfer function is:

[0026] in, , Q=30; Timestamp alignment includes: Normalize the original timestamp of each modal data to the offset relative to the start timestamp. The formula is:

[0027] And unified to a common time base through linear interpolation or clipping operations, the index offset of the sample extraction time window is calculated as:

[0028] in, is the window size, is the sampling frequency; The dataset is divided into 70% training set, 10% validation set, and 20% test set, and the sample labels contain specific feature values ​​of joint angle and pressure.

[0029] Preferably, the structure of the inverted residual module in the module M4 includes: a 1×1 convolution expansion layer, a 3×3 depth-separable convolution layer, a ReLU6 activation function, a 1×1 convolution compression layer and a residual connection; The myoelectric channel input is 8 channels, and it outputs 32-channel features through three inverted residual modules; the joint angle channel input is 15 channels, and it outputs 32-channel features; the pressure channel input is 8 channels, and it outputs 64-channel features; the temperature channel expands the 1-dimensional input to 32 dimensions through a fully connected layer; The feature fusion method of the multimodal neural network architecture is as follows: the features output by the electromyography channel, joint angle channel, and pressure channel are compressed into a 1×1 vector through global average pooling, and then spliced ​​with the fully connected output of the temperature channel. After that, multimodal feature fusion and classification are achieved through a fully connected layer. The depthwise separable convolution module includes: 3×3 depthwise convolution layer, batch normalization layer, ReLU6 activation function and 1×1 point-by-point convolution layer to reduce the number of network parameters.

[0030] Preferably, the weight adjustment of the dynamic loss function in the module M5 includes: Weights for the joint angle regression task Exponential decay:

[0031] Weight of stress regression task Incremental increase:

[0032] And dynamically correct the basic weight of the stress task through the sigmoid function:

[0033] The total loss function is: ; The hyperparameter adjustments in module M6 include learning rate, batch size and dynamic loss function. to minimize the validation set loss and improve the hand motion prediction accuracy.

[0034] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention effectively solves the problem of differences in sampling rate, format, and noise level among heterogeneous data such as electromyographic signals, pressure, joint angle, and temperature through unified format conversion, precise timestamp alignment, and multimodal data fusion technology. It also adopts preprocessing methods such as bandpass filtering, notch filtering, and low-pass filtering to significantly improve signal quality, ensure data consistency and validity, and lay a high-quality data foundation for subsequent model training. 2. This invention utilizes a lightweight neural network architecture based on MobileNetV2, combining depthwise separable convolution with an inverted residual module, significantly reducing the number of network parameters and computational complexity. Its modular design optimizes data from different modalities (such as myoelectric channels and pressure channels), ensuring feature extraction efficiency while meeting the stringent low power consumption and real-time requirements of neural interface chips. 3. This paper innovatively proposes a staged dynamic loss function that dynamically adjusts the weights of joint angle regression, pressure regression, and classification tasks through exponential decay and sigmoid functions. This design effectively balances the conflicting objectives in multi-task learning, alleviates overfitting, and significantly improves the model's generalization performance and prediction stability in complex scenarios. 4. Through the fusion of multimodal features and efficient reasoning of lightweight networks, the system can accurately capture the subtle features of hand grasping movements (such as pressure distribution and joint angle changes), achieve low-latency and high-accuracy movement prediction under a limited number of parameters, and provide reliable technical support for neurorehabilitation training and intelligent prosthetic control.

[0035] 5. This invention supports multiple sensor data formats (e.g., C3D, CSV) and different sampling rates (5 Hz to 1000 Hz), achieving cross-platform data compatibility through interpolation, cropping, and timestamp normalization. This architecture can be flexibly expanded to other multimodal application scenarios (e.g., brain-computer interfaces and motion analysis), and has broad market application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings: Figure 1 Schematic diagram of a method flow in an embodiment of the present invention; Figure 2 This is a diagram of a lightweight multimodal neural network architecture in an embodiment of the present invention; Figure 3 Schematic diagram of the convolution and residual module in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, without departing from the scope of the present invention, a number of variations and improvements may be made by those skilled in the art. These all fall within the scope of protection of the present invention.

[0038] Example 1: Reference Figure 1 and Figure 2 According to the present invention, a multimodal neural network architecture design method for a neural interface chip is provided, the method comprising the following steps: Step S1: Timestamp the multimodal data samples by the host computer, wherein the multimodal data includes fingertip pressure, palm pressure, arm electromyographic signal, hand joint angle and fingertip temperature; Step S2: Convert the data files of different modalities into CSV format and standardize the data format through timestamp alignment and downsampling. Downsampling processing includes: The downsampling interval k is determined based on the ratio of the target sampling rate to the original sampling rate. The calculation formula is:

[0039] And extract the downsampled EMG data through slicing operation:

[0040] The remaining modal data are uniformly converted into second-level timestamp format and stored using the "name_serial number.csv" naming rule.

[0041] Step S3: pre-process the multimodal data using band-pass filtering, notch filtering, and low-pass filtering, and extract sample data within the time window through timestamp normalization and interpolation alignment; Preprocessing includes: A bandpass filter is used to remove noise outside the range of 20 Hz to 490 Hz. The transfer function is:

[0042] in, and , the filter order is ; A notch filter with a center frequency of 50Hz is used to eliminate power frequency interference. The transfer function is:

[0043] in, , Q=30; Timestamp alignment includes: Normalize the original timestamp of each modal data to the offset relative to the start timestamp. The formula is:

[0044] And unified to a common time base through linear interpolation or clipping operations, the index offset of the sample extraction time window is calculated as:

[0045] in, is the window size, is the sampling frequency; The data set is divided into a training set ratio of 70%, a validation set ratio of 10%, and a test set ratio of 20%, and the sample labels contain specific characteristic values ​​of joint angle and pressure; the specific characteristic values ​​include the period, mean, standard deviation, maximum value, minimum value, time distribution density and other statistics corresponding to the joint angle data and pressure data within the set sliding time window. The statistics can be calculated through the data distributed in time series within the sliding window; the characteristics of the data are represented by these data quantities such as period, mean, and maximum value.

[0046] Step S4: Design a lightweight multimodal neural network architecture based on MobileNetV2, which includes parallel processing of myoelectric channels, joint angle channels, pressure channels, and temperature channels. Each channel is fused after feature extraction through an inverted residual module. The structure of the inverted residual module includes: a 1×1 convolution expansion layer, a 3×3 depthwise separable convolution layer, a ReLU6 activation function, a 1×1 convolution compression layer, and a residual connection. The myoelectric channel input is 8 channels, and it outputs 32-channel features through three inverted residual modules; the joint angle channel input is 15 channels, and it outputs 32-channel features; the pressure channel input is 8 channels, and it outputs 64-channel features; the temperature channel expands the 1-dimensional input to 32 dimensions through a fully connected layer; The feature fusion method of the multimodal neural network architecture is as follows: the features output by the electromyography channel, joint angle channel, and pressure channel are compressed into a 1×1 vector through global average pooling, and then spliced ​​with the fully connected output of the temperature channel. After that, multimodal feature fusion and classification are achieved through a fully connected layer. The depthwise separable convolution module includes: 3×3 depthwise convolution layer, batch normalization layer, ReLU6 activation function and 1×1 point-by-point convolution layer to reduce the number of network parameters.

[0047] Step S5: Construct a dynamic loss function and dynamically adjust the weights of joint angle regression, pressure regression, and classification tasks through exponential decay and sigmoid functions to achieve weighted loss fusion. The weight adjustment of the dynamic loss function includes: Weights for the joint angle regression task Exponential decay:

[0048] Weight of stress regression task Incremental increase:

[0049] And dynamically correct the basic weight of the stress task through the sigmoid function:

[0050] The total loss function is: ; Step S6: Adjust the network hyperparameters and train the model to output the prediction results of hand joint angle and fingertip pressure. Hyperparameter adjustment includes learning rate, batch size and dynamic loss function. to minimize the validation set loss and improve the hand motion prediction accuracy.

[0051] The present invention also provides a multimodal neural network architecture design system for a neural interface chip. The multimodal neural network architecture design system for a neural interface chip can be implemented by executing the process steps of the multimodal neural network architecture design method for a neural interface chip, that is, those skilled in the art can understand the multimodal neural network architecture design method for a neural interface chip as a preferred implementation of the multimodal neural network architecture design system for a neural interface chip.

[0052] Example 2: The present invention also provides a multimodal neural network architecture design system for a neural interface chip, the system comprising the following modules: Module M1: Timestamps multimodal data samples through the host computer. The multimodal data includes fingertip pressure, palm pressure, arm electromyography, hand joint angles, and fingertip temperature. Module M2: Convert data files of different modalities into CSV format and standardize the data format through timestamp alignment and downsampling. Downsampling processing includes: The downsampling interval k is determined based on the ratio of the target sampling rate to the original sampling rate. The calculation formula is:

[0053] And extract the downsampled EMG data through slicing operation:

[0054] The remaining modal data are uniformly converted into second-level timestamp format and stored using the "name_serial number.csv" naming rule.

[0055] Module M3: Preprocesses multimodal data using bandpass filtering, notch filtering, and low-pass filtering, and extracts sample data within the time window through timestamp normalization and interpolation alignment; Preprocessing includes: A bandpass filter is used to remove noise outside the range of 20 Hz to 490 Hz. The transfer function is:

[0056] in, and , the filter order is ; A notch filter with a center frequency of 50Hz is used to eliminate power frequency interference. The transfer function is:

[0057] in, , Q=30; Timestamp alignment includes: Normalize the original timestamp of each modal data to the offset relative to the start timestamp. The formula is:

[0058] And unified to a common time base through linear interpolation or clipping operations, the index offset of the sample extraction time window is calculated as:

[0059] in, is the window size, is the sampling frequency; The dataset is divided into 70% training set, 10% validation set, and 20% test set, and the sample labels contain specific feature values ​​of joint angle and pressure.

[0060] Module M4: A lightweight multimodal neural network architecture based on MobileNetV2, including parallel processing of myoelectric channels, joint angle channels, pressure channels, and temperature channels. Each channel is fused after feature extraction through an inverted residual module. The inverted residual module consists of a 1×1 convolution expansion layer, a 3×3 depthwise separable convolution layer, a ReLU6 activation function, a 1×1 convolution compression layer, and a residual connection. The myoelectric channel input is 8 channels, and it outputs 32-channel features through three inverted residual modules; the joint angle channel input is 15 channels, and it outputs 32-channel features; the pressure channel input is 8 channels, and it outputs 64-channel features; the temperature channel expands the 1-dimensional input to 32 dimensions through a fully connected layer; The feature fusion method of the multimodal neural network architecture is as follows: the features output by the electromyography channel, joint angle channel, and pressure channel are compressed into a 1×1 vector through global average pooling, and then spliced ​​with the fully connected output of the temperature channel. After that, multimodal feature fusion and classification are achieved through a fully connected layer. The depthwise separable convolution module includes: 3×3 depthwise convolution layer, batch normalization layer, ReLU6 activation function and 1×1 point-by-point convolution layer to reduce the number of network parameters.

[0061] Module M5: Constructs a dynamic loss function, dynamically adjusts the weights of joint angle regression, pressure regression, and classification tasks through exponential decay and sigmoid functions, and realizes weighted loss fusion; The weight adjustment of the dynamic loss function includes: Weights for the joint angle regression task Exponential decay:

[0062] Weight of stress regression task Incremental increase:

[0063] And dynamically correct the basic weight of the stress task through the sigmoid function:

[0064] The total loss function is: ; Module M6: Adjust network hyperparameters and train the model to output the predicted results of hand joint angle and fingertip pressure. Hyperparameter adjustment includes learning rate, batch size and dynamic loss function. to minimize the validation set loss and improve the hand motion prediction accuracy.

[0065] Example 3: The present invention provides a multimodal neural network architecture design for a neural interface chip, comprising: Step S1: labeling multimodal data samples (fingertip pressure, palm pressure, arm electromyographic signals, hand joint angles, and fingertip temperature) using a host computer; Step S2: processing the data samples, aligning the different modal data, and converting them into a unified format; Step S3: preprocessing the different modal data using filtering techniques, performing sample segmentation and data set partitioning, and generating data set files; Step S4: designing a lightweight multimodal neural network architecture based on MobileNetV2 to adapt the multimodal data and ensure that the network parameters meet the requirements of the neural interface chip; Step S5: designing a staged loss function based on the multimodal neural network architecture, and using weighted summation to fuse the objective functions of the three tasks: joint angle regression, fingertip pressure regression, and stage prediction; and Step S6: adjusting hyperparameters within an existing neural network training framework to obtain optimal hand motion prediction results. This invention can accurately predict hand grasping motions with limited network parameters and is suitable for lightweight neural interface chips.

[0066] By designing a lightweight multimodal neural network architecture, the present invention can efficiently process and fuse data from multiple modalities, including electromyographic data, joint angle data, pressure data, and fingertip temperature data. This multimodal data fusion approach can more comprehensively reflect the characteristics of hand movements, provide more accurate feedback for neurorehabilitation training, and significantly improve the accuracy and real-time performance of movement recognition. The present invention proposes a dynamic loss function that dynamically adjusts the weights of different tasks during training. Dynamically adjusting weights by time steps effectively mitigates overfitting and improves the model's generalization ability. This dynamic adjustment mechanism enables the model to better balance the importance of different tasks during training, thereby improving prediction accuracy and stability. The multimodal neural network architecture design of the present invention is highly adaptable and can accommodate different data formats and sampling rates. Through timestamp alignment and data format conversion, it can process data from different sensors, making it widely applicable. The present invention is suitable for a variety of application scenarios, including neurorehabilitation and intelligent prosthetics. By fusing data from multiple modalities, it can provide more accurate predictions and feedback in these fields, and has broad application prospects.

[0067] The embodiments of the present invention provide a multimodal neural network architecture design for a neural interface chip, which solves the challenges faced by neural interface chips when processing multimodal data and overcomes the shortcomings of traditional methods in data fusion, computing resource limitations, and model generalization capabilities. Figure 1 As shown, the method specifically includes: Step S1: Use the host computer to mark the multimodal data samples; The step S1 specifically includes: Different modal data, such as fingertip pressure, palm pressure, arm electromyography, hand joint angles, and fingertip temperature, originate from different sensors. Different acquisition systems are used for data collection and export, resulting in timestamped data. The host computer timestamps each data sample, providing a standard for time alignment and data partitioning of data samples from different modalities.

[0068] Step S2: Process the data samples, align the data of different modalities, and convert them into a unified format; The step S2 specifically includes: When exporting data files using any modality acquisition software, the file formats and the amount of data contained in each modality differ. Therefore, it is necessary to uniformly convert the data formats and align the timestamps.

[0069] For EMG signal files, you first need to convert the C3D format file to a CSV format file, retaining the valid information columns. In addition, you can choose whether to downsample the data based on the EMG signal settings.

[0070] Assume that the original EMG data is , whose shape is [n,c], where n is the number of original sampling points and c is the number of channels. , the target sampling rate is .

[0071] First, calculate the downsampling interval k:

[0072] Then extract a sample every k points from the original data to get the downsampled data :

[0073] in, Indicates from Take every k rows in the , and keep the columns unchanged.

[0074] Finally, the downsampled data The shape of is [m,c], where is the number of sampling points after downsampling.

[0075] For other modal data files, they are all converted to CSV format and valid data channels are retained.

[0076] In CSV files of any modality, the timestamp format must be adjusted to a format such as "2024-12-31 11:25:22.654294", with seconds accurate to six decimal places. Additionally, if the acquisition software's timestamps cannot be accurately formatted, the time interval will be evenly divided based on the sampling frequency.

[0077] Finally, name all modal files as "name_serial number.csv" and copy them to the project directory.

[0078] Step S3: Preprocess the data of different modalities using filtering technology, perform sample segmentation and data set division, and generate data set files; The step S3 specifically includes: The collected multimodal data are preprocessed and filtered to remove noise and extract effective signals.

[0079] For electromyographic signals, bandpass filtering and notch processing can be performed, as shown below: Bandpass filtering removes low-frequency and high-frequency noise, and its frequency range is and , the filter order is The filter transfer function is:

[0080] in, is the signal frequency. The EMG signal processed by the bandpass filter can effectively remove power frequency interference and other noise.

[0081] Band-stop filtering (notch filtering) can further remove noise of a specific frequency (such as 50 Hz power frequency interference), the center frequency of which is , the quality factor of the filter is Q=30. The transfer function of the filter is:

[0082] For joint angle and pressure data, low-pass filtering can be used for smoothing. The cutoff frequency of the low-pass filter is , the filter order is The filter transfer function is:

[0083] In multimodal data processing, accurate timestamp alignment and sample extraction are key steps to ensure data consistency and validity. This invention implements timestamp processing and sample extraction through the following detailed steps: The purpose of timestamp preprocessing is to unify the timestamps of different modal data onto a standard time base for subsequent sample extraction and alignment. First, timestamp normalization is performed. For each modal data type (such as electromyography data, joint angle data, pressure data, and fingertip temperature data), its timestamp column is extracted. Assume that the timestamp column is in seconds and increments from zero. The timestamps are normalized using the following formula:

[0084] in, It is The value of the timestamp, is the starting timestamp, is the sampling frequency of the modal data (e.g., the sampling frequency of myoelectric data is 1000 Hz, the sampling frequency of joint angle and pressure data is 100 Hz, and the sampling frequency of temperature data is 5 Hz).

[0085] Next, timestamp alignment is performed. To ensure that the timestamps of data from different modalities are accurately aligned, the timestamps of all modal data are unified to a common time base through interpolation or clipping. The specific method is as follows: In the case of timestamp mismatch, the timestamps are aligned to the closest common time point through linear interpolation or nearest neighbor interpolation method; If the timestamp ranges are inconsistent, ensure that the time ranges of all modal data are consistent by trimming the redundant parts.

[0086] Sample extraction is based on the timestamps in the tag file, and samples are extracted from the preprocessed multimodal data. The tag file records the time points of key events, which are used to indicate the start and end positions of the sample. First, the tag timestamp processing is performed, that is, the timestamps of key events are read from the tag file, which are used to indicate the start and end positions of the sample. Assuming that the time points are recorded in the tag file , respectively represent the starting, middle and ending time points of the sample.

[0087] Next, perform a timestamp lookup. For each marked time point, , find the closest timestamp index in the preprocessed multimodal data. The closest timestamp index is calculated using the following formula:

[0088] in, is the timestamp in the preprocessed multimodal data, Marks a point in time in the file.

[0089] Next, sample extraction is performed, and samples are extracted from the multimodal data according to the timestamp index found. , extract the data within the corresponding time window. The window size is (e.g. 100 ms), the extracted sample data includes electromyographic data, joint angle data, pressure data and fingertip temperature data. The specific extraction formula is:

[0090]

[0091]

[0092]

[0093] in, is the index offset corresponding to the window size, and the calculation formula is:

[0094] Finally, the extracted sample data is concatenated into a complete sample and the corresponding labels are generated. The label data usually includes specific feature values ​​of joint angles and pressure data, which are used for subsequent model training and validation.

[0095] The extracted samples are divided into training, validation, and test sets with a ratio of 70%, 10%, and 20%, respectively. The divided datasets are saved as training, validation, and test set files for subsequent model training and verification.

[0096] Step S4: Design a lightweight multimodal neural network architecture based on MobileNetV2 to ensure that the network parameters meet the requirements of the neural interface chip; The step S4 specifically includes: This paper proposes a lightweight multimodal neural network architecture based on MobileNetV2 for efficient processing and integration of data from multiple modalities, including electromyography, joint angles, pressure, and fingertip temperature. This architecture, through its modular design, ensures efficient feature extraction and data processing while adapting to limited computing resources. The specific architecture design is as follows: Convolution module, see Figure 3 As shown in Figure 1, it is the basic building block of the network and is used to extract the features of the input data. Each convolution module includes: (1) Convolutional layer: Use two-dimensional convolution operation to extract features, with a convolution kernel size of 3×3, a stride of 1, and a padding of 1.

[0097] (2) Batch normalization: Normalize the output of the convolutional layer to stabilize the training process.

[0098] (3) ReLU6 activation function: Apply the ReLU6 activation function to the normalized data and limit the output range to [0,6].

[0099] Depthwise separable convolution module, see Figure 3 As shown in Figure 2, the convolution operation is separated to reduce the amount of calculation and the number of parameters. This module includes: (1) Depthwise convolution: A convolution operation is performed on each input channel separately, with a convolution kernel size of 3×3, a stride of 1, and a padding of 1.

[0100] (2) Batch normalization: Normalize the output of depthwise convolution.

[0101] (3) ReLU6 activation function: Apply the ReLU6 activation function.

[0102] (4) Point-by-point convolution: Use a 1×1 convolution kernel for channel expansion or compression.

[0103] (5) Batch normalization: Normalize the output of point-by-point convolution.

[0104] Inverted residual module, refer to Figure 3 As shown in Figure 1, the expansion and compression operations are combined to improve the efficiency of feature extraction. This module includes: (1) 1×1 convolution expansion: Use a 1×1 convolution kernel to expand the number of input channels.

[0105] (2) Depth convolution: Perform depth convolution operation on the expanded features.

[0106] (3) ReLU6 activation function: Apply the ReLU6 activation function.

[0107] (4) 1×1 convolution compression: Use 1×1 convolution kernel to compress the number of channels.

[0108] (5) Batch normalization: Normalize the compressed features.

[0109] (6) Residual connection: If the number of input and output channels is the same and the stride is 1, a residual connection is added.

[0110] Multimodal neural network architecture, refer to Figure 2 As shown in Figure 1, efficient feature extraction and classification are achieved by fusing data from different modalities. The architecture includes the following modules: (1) EMG channel: Three inverted residual modules are used to process EMG data, with 8 input channels and 32 output channels. The feature map is compressed to 1×1 through global average pooling.

[0111] (2) Joint angle channel: Three inverted residual modules are used to process joint angle data, with 15 input channels and 32 output channels. The feature map is compressed to 1×1 through global average pooling.

[0112] (3) Pressure channel: Three inverted residual modules are used to process the pressure data, with 8 input channels and 64 output channels. The feature map is compressed to 1×1 through global average pooling.

[0113] (4) Temperature channel: Use three fully connected layers to process temperature data, with an input dimension of 1 and an output dimension of 32.

[0114] (5) Feature fusion: The processed electromyography, joint angle, pressure and temperature features are spliced ​​together.

[0115] (6) Fully connected layer: Feature fusion and classification are performed through multiple fully connected layers, and the target dimension is finally output.

[0116] Step S5: Design a staged loss function based on the multimodal neural network architecture and use weighted summation to fuse the objective functions of the three tasks; The step S5 specifically includes: This paper proposes a dynamic loss function to dynamically adjust the weights of different tasks during training to alleviate overfitting and improve the generalization ability of the model. The loss function combines position loss, pressure loss and classification loss, and Dynamically adjust weights. The specific design is as follows: Dynamic weight adjustment includes position loss weight and pressure loss weight. The initial position loss weight is and dynamically adjusted by the following formula:

[0117] in, is a parameter that controls the dynamic adjustment rate. is the time step of the current epoch or stage.

[0118] For the pressure loss weight, the initial pressure loss weight is and dynamically adjusted by the following formula:

[0119] At the same time, the basic weight of the stress task is , further adjusted by the following formula:

[0120] in, is the sigmoid function.

[0121] The position loss uses the mean squared error (MSE) to calculate the position loss, that is, the loss of the joint angle regression task: The pressure loss uses the mean absolute error (MAE) to calculate the pressure loss, that is, the loss of the pressure regression task to prevent overfitting and through dynamic weight adjustment:

[0122] Classification loss uses cross entropy loss to calculate the loss for classification tasks.

[0123] The weighted sum of the above three losses gives the total loss:

[0124] Through the above-mentioned dynamic loss function design, the weights of different tasks can be effectively adjusted during the training process, thereby improving the generalization ability and prediction accuracy of the model.

[0125] Step S6: Under the existing neural network training framework, adjust the hyperparameters to obtain the optimal hand movement prediction result.

[0126] The step S6 specifically includes: Based on the existing dataset, multimodal neural network architecture and loss function, the parameters are adjusted to make the network converge and obtain the prediction model. Given the input, the prediction results of hand joint angle and fingertip pressure are obtained.

[0127] Embodiments of the present invention provide a multimodal neural network architecture design for a neural interface chip. This design addresses the challenges faced by neural interface chips when processing multimodal data (such as electromyography, joint angle, pressure, and temperature) by reformulating the multimodal data fusion task as a lightweight neural network architecture design problem. By introducing a dynamic loss function and a lightweight network structure, feature extraction and fusion of data from different modalities are integrated into a unified framework. Position loss, pressure loss, and classification loss are formulated as weighted summation objective functions, where the weights are dynamically adjusted over time to achieve a balance between tasks. Furthermore, the network architecture design takes into account the computational resource constraints of the neural interface chip, ensuring efficient model operation while maintaining low power consumption and high accuracy. The present invention also develops a lightweight multimodal neural network architecture based on MobileNetV2 that automatically optimizes network parameters to achieve efficient feature extraction and data processing. The resulting solution can accurately predict hand grasping movements with limited network parameters, improving the model's generalization and prediction accuracy, and possesses high application value.

[0128] Those skilled in the art may understand this embodiment as a more specific description of Embodiment 1 and Embodiment 2.

[0129] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0130] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A multimodal neural network architecture design method for a neural interface chip, characterized in that: The method comprises the following steps: Step S1: Timestamp the multimodal data samples by the host computer, wherein the multimodal data includes fingertip pressure, palm pressure, arm electromyographic signal, hand joint angle and fingertip temperature; Step S2: Convert the data files of different modalities into CSV format and standardize the data format through timestamp alignment and downsampling. Step S3: pre-process the multimodal data using band-pass filtering, notch filtering, and low-pass filtering, and extract sample data within the time window through timestamp normalization and interpolation alignment; Step S4: Design a lightweight multimodal neural network architecture based on MobileNetV2, which includes parallel processing of electromyographic channels, joint angle channels, pressure channels, and temperature channels. Each channel is fused after feature extraction through the inverted residual module; Step S5: Construct a dynamic loss function, dynamically adjust the weights of joint angle regression, pressure regression, and classification tasks through exponential decay and sigmoid functions, and achieve weighted loss fusion; Step S6: Adjust the network hyperparameters and train the model to output the prediction results of hand joint angles and fingertip pressure.

2. The multimodal neural network architecture design method for a neural interface chip according to claim 1, characterized in that: The downsampling process in step S2 includes: The downsampling interval k is determined based on the ratio of the target sampling rate to the original sampling rate. The calculation formula is: And extract the downsampled EMG data through slicing operation: The remaining modal data are uniformly converted into second-level timestamp format and stored using the "name_serial number.csv" naming rule.

3. The multimodal neural network architecture design method for a neural interface chip according to claim 1, characterized in that: The pre-processing in step S3 includes: A bandpass filter is used to remove noise outside the range of 20 Hz to 490 Hz. The transfer function is: in, and , the filter order is ; A notch filter with a center frequency of 50Hz is used to eliminate power frequency interference. The transfer function is: in, , Q=30; Timestamp alignment includes: Normalize the original timestamp of each modal data to the offset relative to the start timestamp. The formula is: And unified to a common time base through linear interpolation or clipping operations, the index offset of the sample extraction time window is calculated as: in, is the window size, is the sampling frequency; The dataset is divided into 70% training set, 10% validation set, and 20% test set, and the sample labels contain specific feature values ​​of joint angle and pressure.

4. The multimodal neural network architecture design method for a neural interface chip according to claim 1, characterized in that: The structure of the inverted residual module in step S4 includes: a 1×1 convolution expansion layer, a 3×3 depth-separable convolution layer, a ReLU6 activation function, a 1×1 convolution compression layer and a residual connection; The myoelectric channel input is 8 channels, and it outputs 32-channel features through three inverted residual modules; the joint angle channel input is 15 channels, and it outputs 32-channel features; the pressure channel input is 8 channels, and it outputs 64-channel features; the temperature channel expands the 1-dimensional input to 32 dimensions through a fully connected layer; The feature fusion method of the multimodal neural network architecture is as follows: the features output by the electromyography channel, joint angle channel, and pressure channel are compressed into a 1×1 vector through global average pooling, and then spliced ​​with the fully connected output of the temperature channel. After that, multimodal feature fusion and classification are achieved through a fully connected layer. The depthwise separable convolution module includes: 3×3 depthwise convolution layer, batch normalization layer, ReLU6 activation function and 1×1 point-by-point convolution layer to reduce the number of network parameters.

5. The multimodal neural network architecture design method for a neural interface chip according to claim 1, characterized in that: The weight adjustment of the dynamic loss function in step S5 includes: Weights for the joint angle regression task Exponential decay: Weight of stress regression task Incremental increase: And dynamically correct the basic weight of the stress task through the sigmoid function: The total loss function is: ; The hyperparameter adjustment in step S6 includes the learning rate, batch size and dynamic loss function. to minimize the validation set loss and improve the hand motion prediction accuracy.

6. A multimodal neural network architecture design system for a neural interface chip, characterized in that: The system includes the following modules: Module M1: Timestamps multimodal data samples through the host computer. The multimodal data includes fingertip pressure, palm pressure, arm electromyography, hand joint angles, and fingertip temperature. Module M2: Convert data files of different modalities into CSV format and standardize the data format through timestamp alignment and downsampling. Module M3: Preprocesses multimodal data using bandpass filtering, notch filtering, and low-pass filtering, and extracts sample data within the time window through timestamp normalization and interpolation alignment; Module M4: A lightweight multimodal neural network architecture is designed based on MobileNetV2, which includes parallel processing of myoelectric channels, joint angle channels, pressure channels, and temperature channels. Each channel is fused after feature extraction through the inverted residual module; Module M5: Constructs a dynamic loss function, dynamically adjusts the weights of joint angle regression, pressure regression, and classification tasks through exponential decay and sigmoid functions, and realizes weighted loss fusion; Module M6: Adjust network hyperparameters and train the model to output the prediction results of hand joint angles and fingertip pressure.

7. The multimodal neural network architecture design system for a neural interface chip according to claim 6, characterized in that: The downsampling process in the module M2 includes: The downsampling interval k is determined based on the ratio of the target sampling rate to the original sampling rate. The calculation formula is: And extract the downsampled EMG data through slicing operation: The remaining modal data are uniformly converted into second-level timestamp format and stored using the "name_serial number.csv" naming rule.

8. The multimodal neural network architecture design system for a neural interface chip according to claim 6, characterized in that: The preprocessing in the module M3 includes: A bandpass filter is used to remove noise outside the range of 20 Hz to 490 Hz. The transfer function is: in, and , the filter order is ; A notch filter with a center frequency of 50Hz is used to eliminate power frequency interference. The transfer function is: in, , Q=30; Timestamp alignment includes: Normalize the original timestamp of each modal data to the offset relative to the start timestamp. The formula is: And unified to a common time base through linear interpolation or clipping operations, the index offset of the sample extraction time window is calculated as: in, is the window size, is the sampling frequency; The dataset is divided into 70% training set, 10% validation set, and 20% test set, and the sample labels contain specific feature values ​​of joint angle and pressure.

9. The multimodal neural network architecture design system for a neural interface chip according to claim 6, characterized in that: The structure of the inverted residual module in the module M4 includes: a 1×1 convolution expansion layer, a 3×3 depth-separable convolution layer, a ReLU6 activation function, a 1×1 convolution compression layer and a residual connection; The myoelectric channel input is 8 channels, and it outputs 32-channel features through three inverted residual modules; the joint angle channel input is 15 channels, and it outputs 32-channel features; the pressure channel input is 8 channels, and it outputs 64-channel features; the temperature channel expands the 1-dimensional input to 32 dimensions through a fully connected layer; The feature fusion method of the multimodal neural network architecture is as follows: the features output by the electromyography channel, joint angle channel, and pressure channel are compressed into a 1×1 vector through global average pooling, and then spliced ​​with the fully connected output of the temperature channel. After that, multimodal feature fusion and classification are achieved through a fully connected layer. The depthwise separable convolution module includes: 3×3 depthwise convolution layer, batch normalization layer, ReLU6 activation function and 1×1 point-by-point convolution layer to reduce the number of network parameters.

10. The multimodal neural network architecture design system for a neural interface chip according to claim 6, characterized in that: The weight adjustment of the dynamic loss function in the module M5 includes: Weights for the joint angle regression task Exponential decay: Weight of stress regression task Incremental increase: And dynamically correct the basic weight of the stress task through the sigmoid function: The total loss function is: ; The hyperparameter adjustments in module M6 include learning rate, batch size and dynamic loss function. to minimize the validation set loss and improve the hand motion prediction accuracy.

Images