Motion artifact removal method based on convolutional neural network and penalty network

Abstract

The application discloses a method for removing motion artifacts in functional near-infrared spectroscopy signals based on a convolutional neural network and a penalty network, and belongs to the technical field of functional near-infrared spectroscopy. According to the characteristics of the motion artifacts, a de-artifact model is designed and trained to identify and remove the motion artifacts. In order to realize real-time processing of the functional near-infrared spectroscopy signals, effectively eliminate the motion artifacts and restore or enhance the signal-to-noise ratio of the static signals, a specific model training mode, training data acquisition and use mode are designed. In the training mode, only a small amount of prior experimental data is required for training the model, and the model can seamlessly adapt to different experimental paradigms, so that the brain activity measurement is more accurate, and the method has a wide application prospect in the fields of medical diagnosis and brain function research.

Description

Technical Field

[0001] This invention relates to the field of functional near-infrared spectroscopy, and more particularly to a method for motion artifact removal based on convolutional neural networks and penalty networks. Background Technology

[0002] Motion artifacts are a significant factor in functional near-infrared spectroscopy (fNIRS). Motion artifacts refer to the additional signal introduced during measurement due to subject movement or body part movement. In fNIRS, because the device is typically fixed to the subject's head, head movement can generate severe artifacts. These artifacts can increase data noise, thus affecting the accurate measurement of brain activity. Motion artifacts are a common problem in fNIRS data processing, negatively impacting the signal-to-noise ratio (SNR) and center-noise ratio (CNR). Therefore, effectively removing motion artifacts has been a key research focus. Summary of the Invention

[0003] To address the issue of motion artifacts in functional near-infrared spectral signals caused by signal distortion and device performance degradation due to motion in subjects or users, this paper proposes a motion artifact removal method based on convolutional neural networks and penalty networks. The method utilizes a neural network-based artifact removal model to eliminate outliers in the original optical response signal, truncate and reconnect the signal, and then perform correction and quantization analysis. This effectively eliminates motion artifacts and restores or enhances the signal-to-noise ratio of stationary signals. This novel univariate fNIRS signal processing method requires minimal prior experimental data and can seamlessly adapt to different experimental paradigms, showing broad application prospects in the fNIRS field.

[0004] To achieve the above objectives, the technical solution of this case is as follows.

[0005] Firstly, this case proposes a motion artifact removal method based on convolutional neural networks and penalty networks, the method comprising the following steps:

[0006] The functional near-infrared spectral signal containing motion artifacts is acquired. A portion of the data without motion artifacts is extracted from the functional near-infrared spectral signal. This data without motion artifacts is then simulated and expanded to generate semi-simulated data without motion artifacts. Finally, by superimposing the simulated motion artifact signal, semi-simulated data with motion artifacts is generated.

[0007] The semi-simulated data without motion artifacts and the semi-simulated data with motion artifacts are input into the basic trained artifact removal model for parameter fine-tuning to obtain a trained artifact removal model.

[0008] The above functional near-infrared spectral signal containing motion artifacts is input into the trained artifact removal model to obtain the output vector after removing motion artifacts.

[0009] Among them: the artifact removal model after basic training was obtained by training using simulated data without motion artifacts and simulated data with motion artifacts;

[0010] The artifact removal model includes a parallel spatiotemporal feature extraction module and a correction module, both constructed using neural networks. The spatiotemporal feature extraction module first acquires temporal features from the functional near-infrared spectral signal, then acquires spatial features from the acquired temporal features to obtain spatiotemporal features. Further feature extraction and classification are then performed on these spatiotemporal features. Finally, the correction module adjusts the classification of the spatiotemporal feature extraction module based on the current functional near-infrared spectral signal to obtain the output vector after removing motion artifacts.

[0011] In the above technical solution, one implementation of the spatiotemporal feature extraction module is to use a convolutional neural network. The convolutional neural network includes multiple max-pooling convolutional blocks, multiple upsampling convolutional blocks, and a first fully connected module. The multiple max-pooling convolutional blocks are configured to perform multi-scale temporal feature acquisition of motion information from motion data. Each max-pooling convolutional layer is composed of a combination of several convolutional layers and max-pooling layers. The multiple upsampling convolutional blocks are configured to perform multi-scale spatial feature acquisition on the temporal features of the acquired motion information. Each upsampling convolutional block is composed of a combination of several convolutional layers and upsampling layers. The output feature of the last upsampling convolutional block is used as the spatiotemporal feature of motion, and after passing through a first flattening layer, it is input into the first fully connected module. The first fully connected module consists of two fully connected layers. The first layer consists of 2N nodes, and the second layer consists of W nodes. Wherein, W is the sampling frequency F of the functional near-infrared spectroscopy measurement system. s And the product of the moving sampling window T, where N is the largest power of 2 less than W.

[0012] In the above technical solution, one implementation of the correction module is to use a penalty network. The penalty network includes a second flattening layer and a second fully connected module. The second fully connected module consists of two fully connected layers: the first layer consists of N nodes, and the second layer consists of W nodes; where W is the sampling frequency F of the functional near-infrared spectroscopy measurement system. s And the product of the moving sampling window T, where N is the largest power of 2 less than W.

[0013] In one implementation of the basic training of the artifact removal model in the above technical solution, the steps include:

[0014] Using a balloon model and a modified Beer-Lambert law, functional near-infrared spectral signals are generated as simulation data without motion artifacts.

[0015] Simulated motion artifact signals are superimposed on simulated motion artifact-free data to generate simulated motion artifact-containing data.

[0016] The simulation data without motion artifacts and the simulation data with motion artifacts were used as training samples to train the artifact removal model in order to determine the basic parameters of the model.

[0017] In the above technical solution, the loss function used in the artifact removal model is a weighted sum of the mean square error of the output vector and the mean square error of the last element of the output vector:

[0018]

[0019] In the formula: Loss(n) is the loss function at time n, k refers to the k-th element in the output vector Seq, the output vector consists of W elements, Seq0 refers to the target vector without motion artifact contamination, and w1 and w2 are the weighted values ​​of the two terms respectively.

[0020] In the above technical solution, the functional near-infrared spectral signal input to the artifact removal model undergoes preprocessing. One implementation of the preprocessing includes the following steps:

[0021] The functional near-infrared spectral signal is denoted as s, and a fixed sampling window T is used to capture the data point sequence vector Seq(n) corresponding to the target time n:

[0022] seq(n) = [s(n-T+1)...s(n)] T

[0023] K moving average filters are used to filter the data point sequence vector Seq(n) to obtain K smoothed signals sa(n,k), k = 1, 2, ..., K, where K is a set value;

[0024] The filtered signal sa(n, k) is combined with the original digital point sequence vector Seq(n) to form the augmented input matrix M. i (n), M i (n)=[Seq(n) sa(n, 1) ... sa(n, K)].

[0025] Secondly, this case proposes a motion artifact removal system based on convolutional neural networks and penalty networks. The system includes an artifact removal model, the construction steps of which include:

[0026] Combining several convolutional layers and max pooling layers forms a max pooling convolutional block, and combining several convolutional layers and upsampling layers forms an upsampling convolutional block.

[0027] A convolutional neural network is constructed by sequentially connecting several max-pooling convolutional blocks, several upsampling convolutional blocks, a first flattening layer, and a first fully connected module.

[0028] The second flattened layer and the second fully connected module are connected in series to form a penalty network;

[0029] The convolutional neural network and the penalty network are connected in parallel and then connected to the third fully connected module to form an artifact removal model. The trained artifact removal model can be used to remove artifacts in functional near-infrared spectral signals.

[0030] in:

[0031] The first fully connected module consists of two fully connected layers. The first layer consists of 2N nodes, and the second layer consists of W nodes; where W is the sampling frequency F of the functional near-infrared spectroscopy measurement system. s And the product of the moving sampling window T, where N is the largest power of 2 less than W;

[0032] The second fully connected module consists of two fully connected layers: the first layer consists of N nodes and the second layer consists of W nodes.

[0033] The third fully connected module consists of a fully connected layer of W nodes.

[0034] In the above technical solution, one training implementation of the artifact removal model includes basic training and fine-tuning training; wherein:

[0035] Basic training involves training the artifact removal model using simulated data without motion artifacts and simulated data with motion artifacts to determine the model's basic parameters.

[0036] Fine-tuning training involves generating semi-simulated motion-free data and semi-simulated motion-artifact-containing data based on real functional near-infrared spectral signals containing motion artifacts before using the artifact removal model to remove artifacts from functional near-infrared spectral signals containing motion artifacts. This allows for fine-tuning of the parameters of the basic-trained artifact removal model, resulting in a well-trained artifact removal model.

[0037] The loss function used in training is a weighted sum of the mean squared error of the output vector and the mean squared error of the last element of the output vector:

[0038]

[0039] In the formula: Loss(n) is the loss function at time n, k refers to the k-th element in the output vector Seq, the output vector consists of W elements, Seq0 refers to the target vector without motion artifact contamination, and w1 and w2 are the weighted values ​​of the two terms respectively.

[0040] In the above technical solution, one method of obtaining training data for the basic training of the artifact removal model includes the following steps:

[0041] Using a balloon model and a modified Beer-Lambert law, functional near-infrared spectral signals are generated as simulation data without motion artifacts.

[0042] The simulated motion artifact signal is superimposed on the simulated motion artifact-free data to generate the functional near-infrared spectral signal data containing motion artifacts, which is the simulated motion artifact data.

[0043] In the above technical solution, one method for obtaining training data for fine-tuning the artifact removal model includes the following steps:

[0044] A portion of real motion-free artifact data is extracted from real functional near-infrared spectral signals, and this motion-free artifact data is simulated and expanded to generate semi-simulated motion-free artifact data.

[0045] By superimposing simulated motion artifact signals onto semi-simulated data without motion artifacts, semi-simulated data with motion artifacts is generated.

[0046] The beneficial technical effects of this case are:

[0047] (1) The artifact removal model is constructed using a neural network. By introducing signals at different time scales, the network's ability to capture diverse patterns in the data is enhanced, especially for capturing motion artifacts of different durations. Furthermore, by enlarging the spatial size of the temporal feature map, it can better capture motion information over long distances.

[0048] (2) The input of the artifact removal model is a signal sequence, which provides a more comprehensive temporal context representation for potential signal delays and fluctuations.

[0049] (3) The artifact removal model adopts basic training and fine-tuning training. Through fine-tuning training, specific artifact features are captured, thereby suppressing specific motion artifacts in a targeted manner and clearly obtaining the intrinsic signal pattern. It can effectively eliminate motion artifacts and restore or enhance the signal-to-noise ratio of stationary signals, especially showing significant advantages in suppressing maximum motion artifacts.

[0050] (3) The model established in this scheme smooths the training data during training, making the network more robust to high-frequency noise and instantaneous fluctuations in the original data. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1A schematic diagram illustrating an implementation method for motion artifact removal based on convolutional neural networks and penalty networks;

[0053] Figure 2 , one Model structure diagram in one implementation method;

[0054] Figure 3 , one A schematic diagram of the simulation test verification results in one implementation method;

[0055] Figure 4 , one A distribution diagram of the laser source and optical probe of the functional near-infrared brain imaging system in one embodiment, and a schematic diagram of their corresponding positions on the brain.

[0056] Figure 5 The experimental paradigm used when obtaining experimental samples in experimental testing;

[0057] Figure 6 Schematic diagram of experimental test verification results;

[0058] Figure 7 A schematic diagram comparing one implementation of this scheme with other methods based on the d2 evaluation index during experimental testing. Detailed Implementation

[0059] Neural network-based functional near-infrared spectroscopy (fNIRS) motion artifact removal methods face several inherent challenges in their implementation in biomedical signal processing. These limitations include the availability and quality of training data, the potential for model overfitting to training data leading to poor generalization to new datasets, the opacity of the model's decision-making process, the computational demands of training and inference, the model's sensitivity to changes in experimental conditions, and the model's robustness to variations in the signal-to-noise ratio.

[0060] Based on this, this application designs an artifact removal model, which includes a parallel spatiotemporal feature extraction module and a correction module, both of which are constructed using neural networks. Specifically, the spatiotemporal feature extraction module first acquires the temporal features of the functional near-infrared spectral signal, then acquires the spatial features of the acquired temporal features to obtain spatiotemporal features, and then further extracts and classifies the features based on the spatiotemporal features. Finally, the correction module uses the current functional near-infrared spectral signal to correct and adjust the classification of the spatiotemporal feature extraction module to obtain the output vector after removing motion artifacts.

[0061] During the training of the artifact removal model, by constructing data with specific motion artifacts (such as spike-shaped and step-shaped motion artifacts) and designing specific training methods, the model can acquire the characteristics of specific motion artifact data, thereby effectively suppressing specific motion artifacts and clearly displaying the intrinsic signal pattern. For the trained model, when a functional near-infrared spectral signal containing motion artifacts is input, motion data after removing motion artifacts can be obtained.

[0062] The following description, in conjunction with the accompanying drawings, clearly and completely describes how the technical solution of this case is implemented. Obviously, the described embodiments are only a part of the embodiments of this case, and not all of them. Based on the embodiments in this case, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0063] (I) Constructing an artifact removal model

[0064] The spatiotemporal feature extraction module is implemented using a convolutional neural network, and the correction module is implemented using a penalty network.

[0065] The convolutional neural network (CNN) comprises multiple max-pooling convolutional blocks, multiple upsampling convolutional blocks, and a first fully connected module. The max-pooling convolutional blocks are configured to acquire multi-scale temporal features of motion data. Each max-pooling convolutional layer consists of a combination of several convolutional layers and max-pooling layers, corresponding to different time scales and capturing motion information at various time scales. Max-pooling layers reduce the spatial dimensionality of the feature map while retaining the most important feature information. The multiple upsampling convolutional blocks are configured to acquire multi-scale spatial features on the acquired temporal features of the motion information. Each upsampling convolutional block consists of a combination of several convolutional layers and upsampling layers. These upsampling layers increase the spatial size of the feature map, thereby better capturing motion information at greater distances. The output features of the last upsampling convolutional block are used as the spatiotemporal motion features, passed through a first flattening layer, and then input into the first fully connected module. The first fully connected module consists of two fully connected layers: the first layer has 2N nodes, and the second layer has W nodes. The first fully connected module performs feature extraction and classification on the spatiotemporal motion features.

[0066] The penalty network runs parallel to the convolutional neural network (CNN) module, receiving the same input data and allowing for parallel data processing. It consists of a series of fully connected layers. Specifically, the penalty network includes a second flattening layer and a second fully connected module. The second fully connected module comprises two fully connected layers: the first layer has N nodes, and the second layer has W nodes. The penalty network first flattens the input data using the second flattening layer and then processes it through the second fully connected module. The purpose of the penalty network is to introduce a regularization mechanism into the CNN, correcting and adjusting the CNN's classification to improve the model's generalization ability and robustness.

[0067] In the above, W represents the sampling frequency F of the functional near-infrared spectroscopy measurement system. s And the product of the moving sampling window T, where N is the largest power of 2 less than W. Specific values ​​can be adjusted according to the actual application.

[0068] A motion artifact removal model is constructed by connecting a convolutional neural network and a penalty network in parallel, followed by a third fully connected module. This network structure is designed to effectively suppress various types of motion artifacts while enhancing the clarity of the intrinsic signal patterns.

[0069] exist Figure 1 In the artifact removal model, the convolutional neural network consists of four max-pooling convolutional blocks, three upsampling convolutional blocks, a flattening layer, and a first fully connected module connected in series. The penalty network consists of one flattening layer, one fully connected layer with N input nodes, and one fully connected layer with W input nodes. The third fully connected module consists of one fully connected layer with W nodes, producing the final network output.

[0070] The terms "first," "second," and "third" used above are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," or "third" may explicitly or implicitly include one or more of that feature.

[0071] (II) Model Training and Application

[0072] The training data consists of data containing motion artifacts (s cor ) and motion artifact-free data (s ml To effectively eliminate motion artifacts and restore or enhance the signal-to-noise ratio of stationary signals, this study designs a specific model training method and a method for acquiring and using training data. In this training method, the model training only requires a small amount of prior experimental data and can seamlessly adapt to different experimental paradigms, making brain activity measurements more accurate and having broad application prospects in fields such as medical diagnosis and brain function research.

[0073] Specifically, the training of the artifact removal model includes basic training and fine-tuning training; among which:

[0074] Basic training involves training the artifact removal model using simulated data without motion artifacts and simulated data with motion artifacts to determine the model's basic parameters.

[0075] Fine-tuning training involves generating semi-simulated motion-free data and semi-simulated motion-artifact-containing data based on real functional near-infrared spectral signals containing motion artifacts before using the artifact removal model to remove artifacts from functional near-infrared spectral signals containing motion artifacts. This allows for fine-tuning of the parameters of the basic-trained artifact removal model, resulting in a well-trained artifact removal model.

[0076] The loss function used in training is a weighted sum of the mean squared error of the output vector and the mean squared error of the last element of the output vector (see the following formula for details). The model parameters are optimized by minimizing the loss function.

[0077]

[0078] In the formula: Loss(n) is the loss function at time n, k refers to the k-th element in the output vector Seq, the output vector consists of W elements, Seq0 refers to the target vector without motion artifact contamination, and w1 and w2 are the weighted values ​​of the two terms respectively.

[0079] Regarding the acquisition of training data:

[0080] (I) The steps for obtaining training data for the basic training of the artifact removal model include:

[0081] Using a balloon model and a modified Beer-Lambert law, functional near-infrared spectral signals are generated as simulation data without motion artifacts.

[0082] The simulated motion artifact signal is superimposed on the simulated motion artifact-free data to generate the functional near-infrared spectral signal data containing motion artifacts, which is the simulated motion artifact data.

[0083] (II) The steps for obtaining training data for fine-tuning the artifact removal model include:

[0084] A portion of real motion-free artifact data is extracted from real functional near-infrared spectral signals, and this motion-free artifact data is simulated and expanded to generate semi-simulated motion-free artifact data.

[0085] By superimposing simulated motion artifact signals onto semi-simulated data without motion artifacts, semi-simulated data with motion artifacts is generated.

[0086] During the training described above, a 5-fold cross-validation method was used on the training set (s). ml and scor Divide the model into validation and test sets to ensure its generalization ability. Both sets should include data with and without motion artifacts for model validation and performance evaluation.

[0087] During training, the model can be periodically evaluated using a validation set to prevent overfitting. Based on the performance on the validation set, the model's structure and hyperparameters can be adjusted to achieve better performance.

[0088] Set hyperparameters during training, such as 120 epochs and 64 batches. To improve the model's convergence speed and stability, the learning rate will be halved every 20 epochs.

[0089] For the trained model, the real functional near-infrared spectral signal containing motion artifacts is input into the trained artifact removal model to obtain the output vector after removing motion artifacts.

[0090] In summary, the working principle of the above artifact removal model is as follows: A functional near-infrared spectral signal containing motion artifacts is acquired; a portion of motion artifact-free data is extracted from the functional near-infrared spectral signal; this motion artifact-free data is then simulated and expanded to generate semi-simulated motion artifact-free data; subsequently, by superimposing simulated motion artifact signals, semi-simulated data containing motion artifacts is generated; the semi-simulated motion artifact-free data and the semi-simulated motion artifact-containing data are input into a pre-trained artifact removal model for parameter fine-tuning to obtain a trained artifact removal model; finally, the aforementioned functional near-infrared spectral signal containing motion artifacts is input into the trained artifact removal model to obtain the output vector after removing motion artifacts.

[0091] To provide a more comprehensive temporal context representation of potential signal delays and fluctuations, the functional near-infrared spectral signal input to the artifact removal model is preprocessed into a signal sequence. The preprocessing steps include:

[0092] The functional near-infrared spectral signal is denoted as s, and a fixed sampling window T is used to capture the data point sequence vector Seq(n) corresponding to the target time n:

[0093] Seq(n) = [s(n-W+1)...s(n)] T

[0094] Using window sizes ω1, ω2, ..., ω K K moving average filters are used to filter the data point sequence vector Seq(n) to obtain K smoothed signals sa(n,k), k = 1, 2, ..., K, where K is a set value;

[0095] The filtered signal sa(n, k) is combined with the original digital point sequence vector Seq(n) to form the augmented input matrix M. i(n), M i (n)=[Seq(n) sa(n, 1) ... sa(n, K)].

[0096] (III) Experiment

[0097] (3.1) Experiment 1

[0098] Generating simulated contaminated and uncontaminated data: Simulating functional near-infrared spectral signals to model contaminated data, including real motion artifacts. Simultaneously, simulating ideal physiological response signals free from any external noise or interference, serving as uncontaminated data.

[0099] After preprocessing, the simulation data is divided into training, validation, and test sets for model training, tuning, and evaluation. The training set contains data free from motion artifacts. ml As labeled data during model training, its corresponding data s containing motion artifacts is used. cor As input data during model training, the training set (s) is used. A 5-fold cross-validation method is employed to test the training set (s). m1 and s cor The dataset should be divided into two sets to ensure the model's generalization ability. The validation and test sets should also include data with and without motion artifacts for model validation and performance evaluation.

[0100] Set hyperparameters during training, such as 120 epochs and 64 batches. To improve the model's convergence speed and stability, the learning rate will be halved every 20 epochs.

[0101] The neural network model is trained using a training set, and the model parameters are optimized by minimizing the loss function. Training can be stopped and the trained model saved when the model's performance on the validation set meets expectations. During training, the model can be periodically evaluated using the validation set to prevent overfitting. Based on the performance on the validation set, the model's structure, hyperparameters, etc., can be adjusted to achieve better performance.

[0102] The data with motion artifacts in the test set are input into the trained model in real time / pseudo-real time to obtain the output vector Seq(n) after removing motion artifacts.

[0103] Take the last element of the output vector Seq(n) as the filter value at time n. And obtain the real-time filtered signal along the timeline. The results are as follows Figure 3 As shown, the gray bar-shaped areas indicate the simulation task time period of the design.

[0104]

[0105] The complete signal to be filtered is downsampled (frequency approximately 5-10Hz). Using the spline interpolation method (movement artifact removal algorithm, MARA) proposed by Scholkmann et al., the time periods in which motion artifacts exist are first identified. Then, the identified vector is upsampled to the original signal frequency. Finally, the signal correction part of MARA is used for correction to obtain the filtering result of the MARA offline algorithm.

[0106] The complete signal to be filtered is input into the wavelet filtering algorithm proposed by Molavi et al., and the filtering result of the offline wavelet filtering algorithm is obtained.

[0107] The complete signal to be filtered is sequentially input into the MARA offline algorithm and the offline wavelet filtering algorithm to obtain the filtering result of the offline combined filtering.

[0108] The filtering results of the training model of this invention Filtering results compared with other methods (such as MARA, offline wavelet filtering algorithms, and combinations thereof) and A comparison was made. The ΔSNR metric was used to evaluate the motion artifact suppression rate, i.e., to compare the signal-to-noise ratio difference of the signal before and after filtering using different methods:

[0109]

[0110] Where s0 represents data without motion artifacts, and s represents data with motion artifacts. The filtered data (available) and (Alternative). The larger the ΔSNR, the greater the improvement in the signal-to-noise ratio of the algorithm.

[0111] The ΔCC metric is used to evaluate signal distortion, which involves comparing the difference in correlation coefficients between signals before and after filtering using different methods. A larger ΔCC indicates that the filtered signal has less distortion compared to the real signal.

[0112] The comparison results are as follows Figure 3 As shown in the figure, it can be seen that the motion artifact and signal distortion evaluation under real-time filtering conditions are better than or equivalent to the three reference offline filtering algorithms.

[0113] (3.2) Experiment 2

[0114] See Figure 4-7 To better illustrate the method of this invention, the implementation of this solution will be described in detail using actual measured data as an example:

[0115] S10. Collect brain activity data of subjects using a functional near-infrared spectroscopy instrument.

[0116] The experiment was conducted as follows: Eight healthy volunteers without psychological, visual, or other related disabilities were recruited (2 females and 6 males, with an average age of 26.9 ± 2.75 years). Prior to the experiment, all volunteers voluntarily signed an informed consent form and fully understood the experiment's purpose, content, risks, benefits, and protective measures. All participants completed a pre-experiment warm-up exercise (24 three-digit addition and subtraction problems) to familiarize themselves with the experimental tasks. All experimental procedures strictly adhered to the latest Heisenberg Declaration.

[0117] During the experiment, volunteers sat in a dark room in front of a monitor with a black background throughout the experiment. Beforehand, volunteers were instructed to minimize limb and head movements. At the start of the experiment, volunteers were instructed to remain still for 120 seconds (using the monitor's signal as a baseline), then perform 3-digit addition and subtraction problems displayed on the monitor, each with a 10-second calculation period. The monitor would then display a single "+" sign in the center for 20 seconds to help volunteers focus on the screen and quickly return to a resting state. This process was repeated 13 times. During this time, the monitor would randomly change its background color to red for 0.5 seconds. Volunteers were instructed to quickly shake their heads upon seeing the color change, intentionally creating motion artifacts. The specific experimental paradigm is as follows: Figure 4 As shown.

[0118] A functional near-infrared system (ISS, Imagent) was used to record data. This system has 128 channels in a dual-band frequency domain (including 8 photosensitive sensors and 16 lasers), and requires a custom-designed photoelectric cap. After careful design, 3 photosensitive sensors and 8 lasers were selected, totaling 12 channels, arranged as follows: Figure 5 As shown, the laser and photosensitive sensor were symmetrically arranged on the volunteer's prefrontal cortex to achieve symmetry with the Fpz position in the EEG 10-20 system. This arrangement helps improve signal quality and accuracy.

[0119] S20. For each channel and each wavelength signal, extract a segment of the detection signal s during its baseline period (e.g., 30 seconds to 90 seconds). b According to s b Based on the frequency domain characteristics of the signal, the signal is simulated and reproduced. A blood oxygen signal generated by superimposing a balloon model is then applied to obtain a motion artifact-free semi-simulated signal (S). ml ), and convolution superposition of several step-type and spike-type motion artifact signals to obtain a semi-simulated signal containing motion artifacts (S cor ).

[0120]

[0121] MA sp =TR*Paradigm sp ,

[0122]

[0123]

[0124] MA st =Ramp*Paradigm st

[0125]

[0126] Where TR(t) is the trigonometric impulse function, MA sp Paradigm is a temporal signal for spike-type motion artifacts. sp The starting time label for the spike-type motion artifact is Ramp(t), which is the step function, and MA is the starting time label for the spike-type motion artifact. st Paradigm is the time-series signal for step motion artifacts. st This is the starting time label for step motion artifacts.

[0127] S30. Input the data obtained in S20 into the model trained in Experiment (I) and train it again. By retraining based on the original trained model, a specialized training model is formed, which effectively improves the model accuracy. During training, an early stopping function, Earlystop, is added as a callback to determine whether the current verification prediction loss function value is in a decreasing cycle after each cycle is completed. If it is in a decreasing cycle, training continues; otherwise, training stops. The obtained model is used for filtering experimental data.

[0128] S40. Input the experimental dataset collected in S10 into the model trained in step S30 in real time / pseudo-real time to obtain the output vector after removing motion artifacts.

[0129] S50. Take the last element of the output vector Seq(n) as the filter value at time n. And obtain the real-time filtered signal along the timeline.

[0130]

[0131] The complete signal to be filtered is downsampled (frequency approximately 5-10Hz). Using the spline interpolation method (movement artifact removal algorithm, MARA) proposed by Scholkmann et al., the time periods in which motion artifacts exist are first identified. Then, the identified vector is upsampled to the original signal frequency. Finally, the signal correction part of MARA is used for correction to obtain the filtering result of the MARA offline algorithm. The complete signal to be filtered is input into the wavelet filtering algorithm proposed by Molavi et al., and the filtering result of the offline wavelet filtering algorithm is obtained. The complete signal to be filtered is sequentially input into the MARA offline algorithm and the offline wavelet filtering algorithm to obtain the filtering result of the offline combined filtering. The filtering result is as follows Figure 6 As shown, the first row is the signal before filtering, and the second row is the signal after filtering. Motion artifacts are marked with gray bars.

[0132] The four filtered signals were compared with the original signals using the d2 evaluation index. The better the algorithm's suppression of motion artifacts, the larger the d2 value. The results are as follows... Figure 7 As shown, the filtered signal of the method of the present invention is mostly (>50%) filtered by the three existing methods used for reference (data points are distributed above the line).

[0133]

[0134] R = max(s) - min(s)

[0135] Analysis of the one-sided t-test shows that the method of the present invention outperforms MARA (t = 4.04, p < 0.01), the offline wavelet filtering method (t = 18.02, p < 0.01), and the combination of the two (t = 3.94, p < 0.01) in terms of filtering performance on experimental data.

[0136] The implementation process of the two experiments above shows that this scheme has the following advantages:

[0137] (1) High efficiency: The average signal processing time is only 0.59 milliseconds / sample, demonstrating strong potential for processing real-time functional near-infrared spectroscopy (fNIRS) brain-computer interface data.

[0138] (2) High adaptability: The novel fNIRS signal processing method proposed in this scheme requires less prior experimental data and can seamlessly adapt to different experimental paradigms.

[0139] (3) Effective removal of motion artifacts: Suppressing motion artifacts by using a one-dimensional convolutional neural network model helps improve data decoding and device performance.

[0140] The above implementation process enables a motion artifact removal system based on convolutional neural networks and penalty networks. The system includes an artifact removal model, the construction of which includes:

[0141] Combining several convolutional layers and max pooling layers forms a max pooling convolutional block, and combining several convolutional layers and upsampling layers forms an upsampling convolutional block.

[0142] A convolutional neural network is constructed by sequentially connecting several max-pooling convolutional blocks, several upsampling convolutional blocks, a first flattening layer, and a first fully connected module.

[0143] The second flattened layer and the second fully connected module are connected in series to form a penalty network;

[0144] The convolutional neural network and the penalty network are connected in parallel and then connected to the third fully connected module to form an artifact removal model. The trained artifact removal model can be used to remove artifacts in functional near-infrared spectral signals.

[0145] in:

[0146] The first fully connected module consists of two fully connected layers. The first layer consists of 2N nodes, and the second layer consists of W nodes; where W is the sampling frequency F of the functional near-infrared spectroscopy measurement system. s And the product of the moving sampling window T, where N is the largest power of 2 less than W;

[0147] The second fully connected module consists of two fully connected layers: the first layer consists of N nodes and the second layer consists of W nodes.

[0148] The third fully connected module consists of a fully connected layer of W nodes.

[0149] The artifact removal model can be used to identify and remove motion artifacts. With specific training data and methods, it can effectively eliminate motion artifacts and restore or enhance the signal-to-noise ratio of stationary signals in real time using functional near-infrared spectral signals with only a small amount of prior experimental data. It can seamlessly adapt to different experimental paradigms and help make brain activity measurements more accurate.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods, systems, or individual models of this disclosure can be implemented using software plus necessary general-purpose hardware, or they can be implemented using dedicated hardware, including dedicated integrated circuits, dedicated CPUs, dedicated memory, dedicated components, etc. Generally, any function performed by a computer program can be easily implemented using corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the purposes of this disclosure, software implementation is more often preferred for ease of migration, cost reduction, and application implementation.

[0151] Although embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art can make many other forms based on the guidance of this specification and without departing from the scope of protection of the claims of the present invention, and all of these are within the scope of protection of the present invention.

Claims

1. A method for motion artifact removal based on convolutional neural networks and penalty networks, characterized in that, The method includes the following steps: The functional near-infrared spectral signal containing motion artifacts is acquired. A portion of the data without motion artifacts is extracted from the functional near-infrared spectral signal. This data without motion artifacts is then simulated and expanded to generate semi-simulated data without motion artifacts. Finally, by superimposing the simulated motion artifact signal, semi-simulated data with motion artifacts is generated. The semi-simulated data without motion artifacts and the semi-simulated data with motion artifacts are input into the basic trained artifact removal model for parameter fine-tuning to obtain a trained artifact removal model. The above functional near-infrared spectral signal containing motion artifacts is input into the trained artifact removal model to obtain the output vector after removing motion artifacts. Among them: the artifact removal model after basic training was obtained by training using simulated data without motion artifacts and simulated data with motion artifacts; The artifact removal model includes a parallel spatiotemporal feature extraction module and a correction module, both constructed using neural networks. Specifically: the spatiotemporal feature extraction module first acquires temporal features from the functional near-infrared spectral signal, then acquires spatial features from the acquired temporal features to obtain spatiotemporal features, and further extracts and classifies these features. Finally, the correction module adjusts the classification of the spatiotemporal feature extraction module based on the current functional near-infrared spectral signal to obtain the output vector after removing motion artifacts. The spatiotemporal feature extraction module is implemented using a convolutional neural network (CNN). The CNN includes multiple max-pooling convolutional blocks, multiple upsampling convolutional blocks, and a first fully connected module. Specifically: the multiple max-pooling convolutional blocks are configured to acquire multi-scale temporal features of motion information from the motion data; each max-pooling convolutional layer consists of a combination of several convolutional layers and max-pooling layers. The multiple upsampling convolutional blocks are configured to acquire multi-scale spatial features on the acquired temporal features of the motion information; each upsampling convolutional block consists of a combination of several convolutional layers and upsampling layers. The output features of the last upsampling convolutional block are used as the spatiotemporal motion features, passed through a first flattening layer, and then input into the first fully connected module. The first fully connected module consists of two fully connected layers, with the first layer consisting of 2... N The second layer consists of nodes, and the second layer is composed of... W There are 10 nodes; among them... W Sampling frequency of a functional near-infrared spectroscopy measurement system F s and moving sampling window T The product of N Less than W The largest power of 2; The correction module is implemented using a penalty network, which includes a second flattening layer and a second fully connected module. The second fully connected module consists of two fully connected layers, the first of which is composed of… N The second layer consists of nodes, and the second layer is composed of... W There are 10 nodes; among them... W Sampling frequency of a functional near-infrared spectroscopy measurement system F s and moving sampling window T The product of N Less than W The largest power of 2.

2. The method according to claim 1, characterized in that, The basic training steps for artifact removal models include: Using a balloon model and a modified Beer-Lambert law, functional near-infrared spectral signals are generated as simulation data without motion artifacts. Simulated motion artifact signals are superimposed on simulated motion artifact-free data to generate simulated motion artifact-containing data. The simulation data without motion artifacts and the simulation data with motion artifacts were used as training samples to train the artifact removal model in order to determine the basic parameters of the model.

3. The method according to claim 1, characterized in that, The loss function of the artifact removal model is a weighted sum of the mean squared error of the output vector and the mean squared error of the last element of the output vector: In the formula: Loss ( n ) for in n The loss function at time step 1. k Refers to the first in the output vector Seq k The output vector has n elements and consists of n elements. W Composed of several elements, Seq0 refers to the target vector free from motion artifacts. w 1 and w 2. The weighted values ​​corresponding to the two items respectively.

4. The method according to claim 1, characterized in that, The functional near-infrared spectral signal input to the artifact removal model is preprocessed. The preprocessing steps include: The functional near-infrared spectral signal is denoted as s Using a fixed sampling window T To capture the target time n The corresponding data point sequence vector Seq( n ): use K A moving average filter is applied to the data point sequence vector Seq( n ) Perform filtering to obtain K A smoothed signal sa( n , k ), k = 1, 2, …, K , K Set value; The filtered signal sa( n , k ) and the original digital point sequence vector Seq( n Combined into an augmented input matrix M i ( n ), .

5. A motion artifact removal system based on a convolutional neural network and a penalty network, implemented according to claim 1, characterized in that, The system includes an artifact removal model, the construction steps of which include: Combining several convolutional layers and max pooling layers forms a max pooling convolutional block, and combining several convolutional layers and upsampling layers forms an upsampling convolutional block. A convolutional neural network is constructed by sequentially connecting several max-pooling convolutional blocks, several upsampling convolutional blocks, a first flattening layer, and a first fully connected module. The second flattened layer and the second fully connected module are connected in series to form a penalty network; The convolutional neural network and the penalty network are connected in parallel and then connected to the third fully connected module to form an artifact removal model. The trained artifact removal model can be used to remove artifacts in functional near-infrared spectral signals. in: The first fully connected module consists of two fully connected layers. The first layer consists of 2... N The second layer consists of nodes, and the second layer is composed of... W There are 10 nodes; among them... W Sampling frequency of a functional near-infrared spectroscopy measurement system F s and moving sampling window T The product of N Less than W The largest power of 2; The second fully connected module consists of two fully connected layers. The first layer is composed of… N The second layer consists of nodes, and the second layer is composed of... W One node; The third fully connected module consists of W The nodes are composed of a fully connected layer; The training of the artifact removal model includes basic training and fine-tuning training. Basic training involves training the artifact removal model using simulated data without motion artifacts and simulated data with motion artifacts to determine the model's basic parameters. Fine-tuning training involves generating semi-simulated data without motion artifacts and semi-simulated data with motion artifacts based on real functional near-infrared spectral signals with motion artifacts before using the artifact removal model to remove artifacts from functional near-infrared spectral signals with motion artifacts. This allows for fine-tuning of the parameters of the artifact removal model after basic training, resulting in a well-trained artifact removal model.

6. The system according to claim 5, characterized in that: The loss function used in training is a weighted sum of the mean squared error of the output vector and the mean squared error of the last element of the output vector: In the formula: Loss ( n ) for in n The loss function at time step 1. k Refers to the first in the output vector Seq k The output vector has n elements and consists of n elements. W Composed of several elements, Seq0 refers to the target vector free from motion artifacts. w 1 and w 2. The weighted values ​​corresponding to the two items respectively.

7. The system according to claim 6, characterized in that, The steps for obtaining training data for the basic training of the artifact removal model include: Using a balloon model and a modified Beer-Lambert law, functional near-infrared spectral signals are generated as simulation data without motion artifacts. The simulated motion artifact signal is superimposed on the simulated motion artifact-free data to generate the functional near-infrared spectral signal data containing motion artifacts, which is the simulated motion artifact data.

8. The system according to claim 7, characterized in that, The steps for obtaining training data for fine-tuning the artifact removal model include: A portion of real motion-free artifact data is extracted from real functional near-infrared spectral signals, and this motion-free artifact data is simulated and expanded to generate semi-simulated motion-free artifact data. By superimposing simulated motion artifact signals onto semi-simulated data without motion artifacts, semi-simulated data with motion artifacts is generated.

Images