An epileptic seizure warning device based on multimodal physiological data acquisition and synchronization

Abstract

The present invention relates to the field of epilepsy warning, and discloses an epilepsy attack warning device based on multimodal physiological data acquisition and synchronization. The device comprises a data acquisition module, a data processing module and a data warning module, and the device is suitable for being integrated in a wearable device; wherein the data acquisition module is suitable for acquiring data of multiple channels, and the data of each channel is acquired by acquiring data of a user according to a sensor in the wearable device; the data processing module is suitable for determining a reference signal from the data of multiple channels, and performing data synchronization on the data of each channel according to the reference signal, so as to generate displacement data of multiple channels; the data warning module is suitable for generating a data matrix for the displacement data of multiple channels, generating a feature matrix of multiple scales according to the data matrix, and generating an epilepsy probability value according to the feature matrix of multiple scales, and if it is judged that the epilepsy probability value is greater than a preset alarm threshold, an epilepsy warning is performed.

Description

Technical Field

[0001] The present invention relates to the field of epilepsy warning, and in particular to an epilepsy attack warning device based on multi-modal physiological data acquisition and synchronization. Background Art

[0002] At present, epilepsy has become a disease that troubles many patients. The uncertainty of epileptic seizures is very prominent, and it is impossible to predict whether the patient will have an epileptic seizure in the next second. Therefore, it is particularly important to implement early warning for epilepsy. In the existing technology, data such as body temperature and skin electrical signals are usually used to warn of epileptic seizures, but the accuracy is not high.

[0003] Therefore, a new epileptic seizure warning device based on multimodal physiological data acquisition and synchronization is needed. Summary of the invention

[0004] To this end, the present invention provides an epileptic seizure warning device based on multi-modal physiological data acquisition and synchronization, in an effort to solve the above problems.

[0005] According to one aspect of the present invention, there is provided an epileptic seizure warning device based on multimodal physiological data acquisition and synchronization, the device comprising a data acquisition module, a data processing module and a data warning module, the device being suitable for integration in a wearable device; wherein the data acquisition module is suitable for collecting data from multiple channels, the data of each channel being obtained by collecting data on the user according to a sensor in the wearable device; the data processing module is suitable for determining a reference signal from the data of multiple channels, and performing data synchronization on the data of each channel according to the reference signal to generate displacement data of multiple channels; the data warning module is suitable for generating a data matrix for the displacement data of multiple channels, generating a feature matrix of multiple scales according to the data matrix, and generating an epilepsy probability value according to the feature matrix of the multiple scales, and if it is judged that the epilepsy probability value is greater than a preset alarm threshold, an epilepsy warning is performed.

[0006] Optionally, in the device according to the present invention, the data warning module includes a feature extraction module, a feature processing module, a convolution block attention module and a feature warning module; the feature extraction module is suitable for generating a data matrix for the displacement data of multiple channels according to the number of channels and the data length, and generating a first feature matrix at multiple scales based on the data matrix; the feature processing module is suitable for performing a convolution operation on each first feature matrix to generate a second feature matrix; the convolution block attention module is suitable for determining the channel weight coefficient and the spatial weight coefficient according to each second feature matrix, and performing element-level calculation on each second feature matrix according to the channel weight coefficient and the spatial weight coefficient to generate a third feature matrix; the feature warning module is suitable for determining the corresponding feature probability value for each third feature matrix, obtaining the feature probability values ​​at multiple scales, and calculating the epilepsy probability value based on the feature probability values ​​at multiple scales. If it is judged that the epilepsy probability value is greater than the preset alarm threshold, an epilepsy warning is performed.

[0007] Optionally, in the device according to the present invention, the data of the multiple channels include multiple items of acceleration data, body temperature data, electromyography data, skin electricity data, pulse data, blood oxygen data, electrocardiogram data, and electroencephalogram data.

[0008] Optionally, in the device according to the present invention, the data processing module is also suitable for: generating multiple different test delays according to the step number, adding different test delays to the data of a channel to obtain test data of a channel data, obtaining a test data set including multiple test data, and the number of test data in the test data set is the same as the number of the test delays; calculating the correlation coefficient for each test data in the test data set according to the reference signal; taking the test delay corresponding to the test data with the largest correlation coefficient as the synchronization delay, and adding the synchronization delay to the data of the channel to obtain the displacement data corresponding to the channel data.

[0009] Optionally, in the device according to the present invention, the feature extraction module is also suitable for: setting the number of convolutions of each convolution kernel, the number of convolutions including multiple different convolution number values; generating a first feature matrix of corresponding scale according to the number of convolutions for the data matrix according to one or more convolution kernels, and obtaining multiple first feature matrices of different scales corresponding to different convolution number values.

[0010] Optionally, in the device according to the present invention, the feature processing module includes a first convolution block, a second convolution block and a third convolution block, and the feature processing module is suitable for performing convolution processing on the first feature matrix at each scale according to the first convolution block, the second convolution block and the third convolution block to generate a second feature matrix, so as to obtain multiple second feature matrices.

[0011] Optionally, in the device according to the present invention, the convolution block attention module is suitable for determining the channel weight coefficient according to each second feature matrix; performing element-level calculations based on the channel weight coefficients and the second feature matrix to generate an intermediate feature matrix; determining the spatial weight coefficient according to the intermediate feature matrix; and performing element-level calculations based on the spatial weight coefficients and the intermediate feature matrix to generate a third feature matrix.

[0012] Optionally, the device according to the present invention also includes: a feature warning module including a first fully connected layer, a second fully connected layer and a third fully connected layer, and the feature warning module is suitable for processing each third feature matrix according to the first fully connected layer, the second fully connected layer and the third fully connected layer to generate a feature probability value at each scale; calculating an epilepsy probability value according to the feature probability values ​​at multiple scales; judging whether the epilepsy probability value is greater than a preset alarm threshold, and if so, issuing an epilepsy warning.

[0013] According to another aspect of the present invention, a computing device is provided, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing the steps performed by the epileptic seizure warning device based on multimodal physiological data acquisition and synchronization according to the present invention.

[0014] According to another aspect of the present invention, there is provided a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions which, when executed by a computing device, cause the computing device to execute the steps performed by the epileptic seizure warning device based on multimodal physiological data acquisition and synchronization according to the present invention.

[0015] The present invention proposes an epilepsy attack warning device based on multimodal physiological data acquisition and synchronization, including a data acquisition module, a data processing module and a data warning module; the data acquisition module collects data from multiple channels for epilepsy warning, which can improve the accuracy and success rate of epilepsy warning. The data processing module synchronizes the data of multiple channels, so that the data of multiple channels can collaboratively generate a feature matrix and jointly perform epilepsy warning; the data warning module collects features of multiple scales on the data matrix to obtain multiple scale feature matrices, thereby extracting features from multiple dimensions, collecting information in the data as much as possible, and further improving the accuracy of epilepsy warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] To achieve the above and related purposes, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings, which indicate various ways in which the principles disclosed herein can be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. The above and other purposes, features and advantages of the present disclosure will become more apparent by reading the following detailed description in conjunction with the accompanying drawings. Throughout the present disclosure, the same reference numerals generally refer to the same parts or elements.

[0017] Figure 1a A schematic diagram of an epileptic seizure warning device 100 based on multi-modal physiological data acquisition and synchronization according to an exemplary embodiment of the present invention is shown;

[0018] Figure 1b A schematic diagram showing data synchronization by a data processing module according to an embodiment of the present invention is shown.

[0019] Figure 1c A schematic diagram of a data early warning module according to an exemplary embodiment of the present invention is shown;

[0020] Figure 2 A schematic diagram of performing convolution processing on a data matrix according to an exemplary embodiment of the present invention is shown;

[0021] Figure 3a A schematic diagram of a feature processing module according to an exemplary embodiment of the present invention is shown;

[0022] Figure 3b shows a schematic diagram of a convolution block according to an exemplary embodiment of the present invention;

[0023] Figure 4a A schematic diagram showing processing of a second feature matrix according to a channel attention module according to an exemplary embodiment of the present invention is shown;

[0024] Figure 4b A schematic diagram showing processing of an intermediate feature matrix according to a spatial attention module according to an exemplary embodiment of the present invention is shown;

[0025] Figure 5 A schematic diagram of a feature warning module according to an exemplary embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to enable the scope of the present disclosure to be fully conveyed to those skilled in the art. The same reference numerals generally refer to the same components or elements.

[0027] In the prior art, acceleration signals, body temperature signals, electromyography signals, skin electrical signals and electrocardiogram signals are usually collected, and the above signals are analyzed according to the self-developed attention mechanism Attention-LSTM to warn of epilepsy. The success rate and accuracy of the warning are low.

[0028] While exploring the pathogenesis of epilepsy, the present invention found that EEG is a more effective indicator for judging epileptic seizures. Therefore, the present invention collects EEG signals and synchronizes data of multiple signals. The onset of epilepsy is predicted based on the signals after data synchronization, thereby improving the success rate and accuracy of epilepsy onset prediction.

[0029] According to one embodiment of the present invention, an epileptic seizure warning device based on multimodal physiological data acquisition and synchronization includes a data acquisition module, a data processing module and a data warning module. Figure 1a A schematic diagram of an epileptic seizure warning device 100 based on multimodal physiological data acquisition and synchronization according to an exemplary embodiment of the present invention is shown. As shown in FIG1 , the epileptic seizure warning device based on multimodal physiological data acquisition and synchronization includes a data acquisition module 110, a data processing module 120 and a data warning module 130.

[0030] According to one embodiment of the present invention, an epileptic seizure warning device based on multimodal physiological data acquisition and synchronization can be integrated into a wearable device, which is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices may specifically include bracelets, watches, glasses, gloves, watches, clothing and shoes. One or more sensors are provided in the wearable device to collect different data, and different types can be collected by different sensors to obtain data from multiple channels.

[0031] The data acquisition module 110 is suitable for collecting data from multiple channels, and the data of each channel is obtained by collecting data from the user according to the sensor in the wearable device. The data processing module 120 is suitable for determining the reference signal from the data of multiple channels, and synchronizing the data of each channel according to the reference signal to generate displacement data of multiple channels. The data warning module 130 is suitable for generating a data matrix for the displacement data of multiple channels, generating a feature matrix of multiple scales according to the data matrix, and generating an epilepsy probability value according to the feature matrix of multiple scales. If it is judged that the epilepsy probability value is greater than the preset alarm threshold, an epilepsy warning is performed.

[0032] According to one embodiment of the present invention, the data acquisition module is suitable for collecting multiple data, and each type of data collected is the data of one channel. The data can be specifically implemented as physiological data, including acceleration data, body temperature data, electromyography data, skin electricity data, pulse data, blood oxygen data, electrocardiography data, electroencephalography data, etc. Among them, the collected electromyography data, electrocardiography data and electroencephalography data can be specifically implemented as graph data, and the horizontal coordinate of the graph data is time, and the vertical coordinate is the amplitude of the waveform. Acceleration data, body temperature data, skin electricity data, pulse data and blood oxygen data can be specifically implemented as a data sequence about time. The present invention does not limit the specific data form of each physiological data.

[0033] According to an embodiment of the present invention, when the data acquisition module acquires multiple data, the data may be acquired through a connected bracelet, electrocardiogram instrument, electroencephalogram instrument, etc.; for example, acceleration data, pulse data, body temperature data, etc. may be acquired through a bracelet, electrocardiogram data may be acquired through an electrocardiogram instrument, and electroencephalogram data may be acquired through an electroencephalogram instrument. The present invention does not limit the specific acquisition method of each type of data.

[0034] After collecting the data, the device constructs a data matrix for each type of data with the collection time as the horizontal coordinate and the amplitude or value as the collected data value as the vertical coordinate, and uses an algorithm to process and analyze the data matrix to extract features, thereby achieving the prediction of epilepsy.

[0035] According to one embodiment of the present invention, when the data acquisition module acquires data of each channel, the sampling frequency during acquisition is greater than 500 Hz, which facilitates data synchronization and improves the sensitivity and accuracy of epilepsy early warning.

[0036] According to one embodiment of the present invention, after the data acquisition module acquires data and obtains data of multiple channels, the data processing module performs data synchronization on the data of multiple channels. The data of each channel is a type of data acquired, including multiple data information acquired within a continuous period of time, and each data information includes an acquired data value and an acquisition time. The present invention does not limit the specific time width of the continuous time. The length of the data of each channel is the number of data information included in the data. If the channel data includes 500 data information, the length of the data is 500.

[0037] Figure 1b FIG. 2 shows a schematic diagram of data synchronization performed by a data processing module according to an embodiment of the present invention. Figure 1b As shown, first, step 121 is executed to obtain data of multiple channels. Then, step 122 is executed to unify the frequency of the data of each channel, such as uniformly reducing the frequency to 500 Hz. The data of multiple channels can be obtained using S1, S2, S3...S n Indicates, n indicates the channel. S1 indicates the data of the first channel, S2 indicates the data of the second channel, S3 indicates the data of the third channel, S n Represents the data of the nth channel.

[0038] Subsequently, step 123 is executed, the data processing module generates a plurality of different test delays according to the step number, adds different test delays to the data of one channel to obtain test data of one channel data, and obtains a test data set including a plurality of test data for each channel, wherein the number of test data in the test data set is the same as the number of test delays;

[0039] Then, step 124 is executed to calculate the correlation coefficient for each test data in the test data set according to the reference signal. Then, step 125 is executed to use the test delay corresponding to the test data with the largest correlation coefficient as the synchronization delay. Finally, step 126 is executed to add the synchronization delay to the data of the channel to obtain the displacement data corresponding to the channel data.

[0040] When the data processing module determines the reference signal, it can select data from one of the multiple channels of data collected as the reference signal, such as using the first channel data S1 as the reference signal. The present invention does not limit the specific method of selecting the reference signal.

[0041] When synchronizing the data of other channels according to the reference signal, first determine the synchronization delay τ of each channel data according to the reference signal. n , and then set each channel data according to the synchronization delay to generate displacement data. nIt represents the synchronization delay of the nth channel. For example, when the first channel data S1 is used as the reference signal, τ2 represents the synchronization delay of the second channel, and τ3 represents the synchronization delay of the third channel.

[0042] According to an embodiment of the present invention, when determining other channel data, such as the synchronization delay τ2 of the second channel data, according to the reference signal, a plurality of test delays τ are first generated according to the step number. j Where j is the number of steps to move to the positive or negative time axis, τ j is the test delay when the step number is j. j and τ j When j is positive (such as j = 1, τ j =0.02s), the acquisition time in the data information is added to the test delay and increases, and the acquisition time moves toward the positive time axis by step number j; j and τ j When it is negative, the acquisition time in the data information is reduced by adding the test delay, and the acquisition time moves toward the negative time axis by the step number j (such as j = -1, τ -1 =-0.02s).

[0043] The step is calculated based on a uniform frequency. For example, when the frequency is 500Hz, the step is 0.02s. Multiple test delays are generated based on the step. For example, the value range of the step number j is determined to be 1-50 based on 1 second and the step. The present invention does not limit the specific method and delay range for generating the test delay. For example, all possible delays within 1 second can be set according to the uniform frequency. The test delay is determined based on the step and the number of steps. For example, when the step is 0.02s and the step number j is 1, the test delay τ1=0.02s, when the step number j is 2, the test delay τ2=0.04s, when the step number j is 3, the test delay τ3=0.06s, ..., when the step number j is 50, the test delay τ 50 =1s.

[0044] Then, a test data set with test delay is generated according to each test delay and other channel data, such as the second channel data. The test data set is the test data set of other channel data, such as the second channel data, including 50 test data with different test delays. When generating corresponding test data according to each test delay, the acquisition time of each data information in other channel data, such as the second channel data, is added with the test delay to obtain the test data with test delay, which is expressed as Meaning is the nth channel data, with the jth test delay τ added j The test data obtained. 2，1 Indicates the data of the second channel, the first test delay τ1 is added (τ1 = 0.02s), and the acquisition time of each test data in the channel data is added with the jth test delay τ j, such as the first test delay τ1 = 0.02s, so that the acquisition time moves 1 step toward the positive time axis; if j = -1, τ -1 =0.02s, which can move the test data acquisition time one step toward the negative axis.

[0045] According to j test delays τ1-τ j Each other channel data (such as the second channel data) is processed to obtain each other channel data, such as j test data of the second channel data, and the test data of each other channel data is combined to obtain a test data set of each other channel data.

[0046] Then, the correlation coefficient r of each test data in the test data set with the reference signal is calculated. j , r j According to the j-th test delay τ j Calculate the test data When calculating the correlation coefficient, it can be calculated according to the following formula:

[0047]

[0048] Among them, E is the expected value operation, yes The conjugate matrix of .

[0049] According to an embodiment of the present invention, when calculating the correlation coefficient, the correlation coefficient may be cyclically calculated according to the xcorr function in MATLAB.

[0050] After obtaining j correlation coefficients, the test delay corresponding to the largest correlation coefficient is used as the synchronization delay τ of the data under this channel s The data of this channel are set according to the synchronization delay, and the displacement data with synchronization delay is obtained, which is recorded as S ns , which means the displacement data of the nth channel with added synchronization delay; specifically, the acquisition time of each data information of the channel data can be added with the synchronization delay τ s , and get the displacement data.

[0051] According to an embodiment of the present invention, when determining the synchronization delay, a test delay is first added to the data of the channel to generate 50 test data, and the correlation coefficients of the 50 test data are calculated with S1 respectively to obtain 50 correlation coefficients r1-r 50 , if the 14th correlation coefficient r 14 Maximum, then the corresponding test delay τ 14 As a synchronization delay, add S2 to obtain the second displacement signal S 2，14 , which is the displacement data after synchronizing the second channel data.

[0052] Subsequently, the synchronization delay is determined for the data of all channels according to the above method, and displacement data is generated to obtain displacement data with synchronization delay for n channels.

[0053] According to one embodiment of the present invention, different data can be collected by different sensors, such as acceleration data, body temperature data, electromyography data, skin electricity data, pulse data, blood oxygen data, etc. Different sensors may have different data collection frequencies and different start collection times. Therefore, the present invention synchronizes the data collected by each sensor. The present invention can solve the problem of insufficient accuracy of single data information in epilepsy prediction and data synchronization when multiple sensors are jointly collected.

[0054] Figure 1c FIG. 4 is a schematic diagram of a data early warning module according to an exemplary embodiment of the present invention. Figure 1c As shown, the data warning module 130 includes a feature extraction module 131, a feature processing module 132, a convolution block attention module 133 and a feature warning module 134. The feature extraction module 131 generates a data matrix for the displacement data of multiple channels according to the number of channels and the data length, and generates a first feature matrix at multiple scales according to the data matrix;

[0055] The feature processing module 132 performs a convolution operation on each first feature matrix to generate a second feature matrix; the convolution block attention module 133 determines a channel weight coefficient and a spatial weight coefficient according to each second feature matrix, and performs element-level calculation on each second feature matrix according to the channel weight coefficient and the spatial weight coefficient to generate a third feature matrix;

[0056] The feature warning module 134 determines the corresponding feature probability value for each third feature matrix, obtains the feature probability values ​​at multiple scales, and calculates the epilepsy probability value based on the feature probability values ​​at multiple scales. If it is determined that the epilepsy probability value is greater than the preset alarm threshold, an epilepsy warning is issued.

[0057] According to one embodiment of the present invention, after the data processing module generates displacement data, the feature extraction module constructs a data matrix for the displacement data of all channels with the acquisition time as the horizontal coordinate, which can be specifically realized as the acquisition data value of the amplitude or value as the vertical coordinate, so as to use the algorithm to process and analyze the data matrix, extract features, and further realize the prediction of epilepsy.

[0058] According to one embodiment of the present invention, the feature extraction module constructs a data matrix for the data of all channels, and the matrix size obtained is (n, 1, T), where n is the number of channels and T is the length of the displacement data in the channel.

[0059] Subsequently, the data matrix is ​​processed to extract feature information, including the steps of: setting the number of convolutions for each convolution kernel, where the number of convolutions includes a plurality of different convolution times; generating a first feature matrix of corresponding scale according to the number of convolutions for the data matrix based on one or more convolution kernels, and obtaining a plurality of first feature matrices of different scales corresponding to different convolution times.

[0060] According to one embodiment of the present invention, the convolution kernels set by the feature extraction module include a (1*1) convolution kernel and a (1*2) convolution kernel. The present invention does not limit the number of convolution kernels set by the feature extraction module and the size of the convolution kernels. The convolution times of the set convolution times include 1, 2, 3, and 4 times. The present invention does not limit the setting method of the convolution times of each convolution kernel.

[0061] According to one embodiment of the present invention, when generating a first feature matrix of corresponding scale according to the number of convolutions based on one or more convolution kernels for a data matrix, a (1*1) convolution kernel is first used to reduce the dimension of the data matrix, and then the data is processed through multi-layer convolution operations, each convolution kernel size is (1*2), and the convolution stride is 2.

[0062] According to an embodiment of the present invention, when processing a data matrix, one or more convolution processes are performed to obtain first feature matrices of multiple scales. Figure 2 FIG. 4 is a schematic diagram showing a convolution process for a data matrix according to an exemplary embodiment of the present invention. Figure 2 As shown, the data matrix is ​​processed once with the convolution kernel (1*1) to obtain the first feature matrix of the first scale; the data matrix is ​​processed with the convolution kernel (1*1) and then processed once with the convolution kernel (1*2) to obtain the first feature matrix of the second scale; the data matrix is ​​processed with the convolution kernel (1*1) and then processed twice with the convolution kernel (1*2) to obtain the first feature matrix of the third scale; the data matrix is ​​processed with the convolution kernel (1*1) and then processed three times with the convolution kernel (1*2) to obtain the first feature matrix of the fourth scale; the data matrix is ​​processed with the convolution kernel (1*1) and then processed four times with the convolution kernel (1*2) to obtain the first feature matrix of the fifth scale; the data matrix is ​​processed with the convolution kernel (1*1) of maxpool and then processed once with the convolution kernel (1*1) to obtain the first feature matrix of the sixth scale. Figure 2 The manner of performing convolution processing on the data matrix shown is only exemplary, and the present invention does not limit the size and type of the convolution kernel for processing the data matrix, and the number of specific convolution processing times.

[0063] After the above convolution processing, the input size of the k-th convolution processing is (n, 1, ), the maximum number of convolutions can be set to 4. The features extracted from each scale are M is a real number matrix with L rows and d columns, representing the dimension of the first feature matrix. The purpose of multi-scale analysis of the data matrix is ​​that since the information contained in different data may exist at different scales, multi-scale analysis can extract the best features of data information at different scales and extract features from the data.

[0064] The feature processing module includes a first convolution block, a second convolution block and a third convolution block. After generating multiple first feature matrices of different scales, the feature processing module performs convolution processing on the first feature matrix at each scale according to the first convolution block, the second convolution block and the third convolution block to generate a second feature matrix, thereby obtaining multiple second feature matrices.

[0065] Figure 3a FIG. 4 shows a schematic diagram of a feature processing module according to an exemplary embodiment of the present invention. Figure 3a As shown, the multi-scale first feature matrix is ​​input into the convolution module to generate the multi-scale second feature matrix. The feature processing module may include a first convolution block, a second convolution block and a third convolution block.

[0066] Figure 3b FIG. 4 shows a schematic diagram of a convolution block according to an exemplary embodiment of the present invention. Figure 3b As shown, the convolution block includes a convolution kernel (conv1D), a Dropout layer, a Maxout layer, and a Max Pool 1D layer. The present invention does not limit the specific composition of the convolution and.

[0067] According to one embodiment of the present invention, the convolution kernel size of the first convolution block (conv1) is (8, 128, 5), the convolution kernel size of the second convolution block (conv2) is (128, 512, 5), and the convolution kernel size of the third convolution block (conv3) is (512, 16, 5). The present invention does not limit the number of convolution blocks included in the convolution module and the size of the convolution kernel corresponding to each convolution block.

[0068] The first feature matrix of each scale is processed by the feature processing module to obtain the second feature matrix F:

[0069] F = conv3(conv2(conv1(M)))

[0070] According to one embodiment of the present invention, after obtaining the second feature matrices of multiple scales, the convolution block attention module determines the channel weight coefficient according to each second feature matrix; performs element-level calculations based on the channel weight coefficients and the second feature matrix to generate an intermediate feature matrix; determines the spatial weight coefficient based on the intermediate feature matrix; and performs element-level calculations based on the spatial weight coefficients and the intermediate feature matrix to generate a third feature matrix.

[0071] According to an embodiment of the present invention, the convolution block attention module includes a channel attention module and a spatial attention module; specifically, each second feature matrix is ​​first processed according to the channel attention module.

[0072] Figure 4a FIG. 4 is a schematic diagram showing a process of processing the second feature matrix according to a channel attention module according to an exemplary embodiment of the present invention. Figure 4a As shown, the second feature matrix F(m*n) is aggregated through the global MaxPool layer and the AvgPool layer to obtain two 1*n feature maps; the two 1*n feature maps are then sent to a two-layer neural network (MLP), where the number of neurons in the first layer is T / r, r is the reduction rate, and the activation function is the Relu function; the number of neurons in the second layer is T. The two-layer neural network shares the input to compress the dimension of the input feature map. The feature map output by the output neural network (MLP) is then element-wise added to obtain the result of the addition operation, and the result of the addition operation is calculated according to the Sigmoid activation function to obtain the channel weight coefficient Mc. Finally, the scaled intermediate feature matrix F' can be obtained by multiplying Mc by the original second feature matrix F. The calculation formula is as follows:

[0073]

[0074] Where σ represents the sigmoid function, W0 is a convolution block with a convolution kernel size of (C, R, 1), and W1 is a convolution block with a convolution kernel size of (R, C, 1). C is the dimension of the convolution layer input, that is, the number of channels of the feature matrix, and R represents the dimension of the convolutional feature. R can be set as needed. To facilitate feature concatenation, it can be set to be consistent with the size of the feature matrix, such as 32. and They represent the global average pooling results and the global maximum pooling results respectively.

[0075] Figure 4b FIG. 4 is a schematic diagram showing a process of processing an intermediate feature matrix according to a spatial attention module according to an exemplary embodiment of the present invention. Figure 4bAs shown in the figure, the intermediate feature matrix F' output by the channel attention module is used as the input of the spatial attention module. After passing through the MaxPool layer and the AvgPool layer, an m*1 feature map is obtained. Then, it is input into the linear connection layer to connect it linearly. The spatial weight coefficient Ms is calculated to generate a two-dimensional spatial attention feature map. Subsequently, the third feature matrix is ​​calculated based on the spatial weight coefficient and the intermediate feature matrix. The calculation formula of the spatial weight coefficient is as follows:

[0076]

[0077] Where W2 is a convolution block, which includes a convolution kernel size of (2, R, 7). Therefore, the calculation method of the second feature matrix according to the convolution block attention module is:

[0078]

[0079] in The third feature matrix F" is obtained by element-wise multiplication as the final output.

[0080] According to one embodiment of the present invention, each second feature matrix generated obtains a third feature matrix according to the convolution block attention module. For example, the second feature matrices of 6 scales calculated according to the above embodiment are processed to obtain a third feature matrix F" of 6 scales, which is then input into the feature warning module for processing.

[0081] According to one embodiment of the present invention, the feature warning module includes a first fully connected layer, a second fully connected layer and a third fully connected layer. The feature warning module is suitable for processing each third feature matrix according to the first fully connected layer, the second fully connected layer and the third fully connected layer to generate a feature probability value at each scale; calculating an epilepsy probability value according to the feature probability values ​​at multiple scales; and determining whether the epilepsy probability value is greater than a preset alarm threshold. If it is greater than the preset alarm threshold, an epilepsy warning is issued.

[0082] Figure 5 FIG. 4 shows a schematic diagram of a feature warning module according to an exemplary embodiment of the present invention. Figure 5 As shown, the dimension of the first fully connected layer can be set to 512, the dimension of the second fully connected layer can be set to 64, and the dimension of the third fully connected layer can be set to 32. The present invention does not limit the specific dimension size of each fully connected layer.

[0083] The characteristic warning module processes the third characteristic matrix of each different scale to obtain characteristic probability values ​​at multiple scales. When calculating the epilepsy probability value according to the characteristic probability value, it can be calculated by weighted average or the like. The present invention does not limit the specific method of calculating the epilepsy probability value.

[0084] According to one embodiment of the present invention, after obtaining the epilepsy probability value, it is determined whether the epilepsy probability value is greater than a preset alarm threshold. The preset alarm threshold can be set to 95%. When performing an epilepsy warning, an alarm sound can be emitted through a connected bracelet, or a message can be sent to a pre-stored emergency contact, etc. The present invention does not limit the specific method of epilepsy warning.

[0085] It should be noted that the storage medium (computer-readable medium) mentioned above in the present application may be a computer-readable signal medium or a non-transitory computer-readable storage medium or any combination of the above two. The non-transitory computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of non-transitory computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0086] In the present application, a non-transitory computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a non-transitory computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0087] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0088] The above description is only a partial embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

[0089] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0090] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An epileptic seizure warning device based on multimodal physiological data acquisition and synchronization, the device comprising a data acquisition module, a data processing module and a data warning module, and the device is suitable for integration in a wearable device; in, The data acquisition module is suitable for collecting data of multiple channels, and the data of each channel is obtained by collecting data of the user according to the sensor in the wearable device; The data processing module is adapted to determine a reference signal from the data of the multiple channels, and perform data synchronization on the data of each channel according to the reference signal to generate displacement data of the multiple channels; The data warning module is suitable for generating a data matrix for the displacement data of multiple channels, generating a feature matrix of multiple scales according to the data matrix, and generating an epilepsy probability value according to the feature matrix of multiple scales. If it is judged that the epilepsy probability value is greater than a preset alarm threshold, an epilepsy warning is performed.

2. The device according to claim 1, wherein: The data processing module is also suitable for: Generate a plurality of different test delays according to the step number, add different test delays to the data of a channel to obtain test data of the channel data, and obtain a test data set including a plurality of test data, wherein the number of test data in the test data set is the same as the number of the test delays; Calculating a correlation coefficient for each test data in the test data set according to the reference signal; The test delay corresponding to the test data with the largest correlation coefficient is used as the synchronization delay, and the synchronization delay is added to the data of the channel to obtain the displacement data corresponding to the channel data.

3. The device according to claim 1, wherein: The data warning module includes a feature extraction module, a feature processing module, a convolution block attention module and a feature warning module; The feature extraction module is adapted to generate a data matrix for the displacement data of multiple channels according to the number of channels and the data length, and to generate a first feature matrix at multiple scales according to the data matrix; The feature processing module is adapted to perform a convolution operation on each first feature matrix to generate a second feature matrix; The convolution block attention module is adapted to determine a channel weight coefficient and a spatial weight coefficient according to each second feature matrix, and perform element-level calculation on each second feature matrix according to the channel weight coefficient and the spatial weight coefficient to generate a third feature matrix; The feature warning module is suitable for determining the corresponding feature probability value for each third feature matrix, obtaining the feature probability values ​​at multiple scales, and calculating the epilepsy probability value based on the feature probability values ​​at multiple scales. If it is judged that the epilepsy probability value is greater than a preset alarm threshold, an epilepsy warning is performed.

4. The device of claim 1, wherein: The data of the multiple channels include multiple items of acceleration data, body temperature data, electromyography data, skin electricity data, pulse data, blood oxygen data, electrocardiogram data, and electroencephalogram data.

5. The device according to any one of claims 2 to 4, wherein: The feature extraction module is also suitable for: Setting the number of convolutions for each convolution kernel, wherein the number of convolutions includes a plurality of different convolution number values; The data matrix is ​​compared with the convolution times according to one or more convolution kernels to generate a first characteristic matrix of a corresponding scale, so as to obtain a plurality of first characteristic matrices of different scales corresponding to different convolution times.

6. The device according to any one of claims 2 to 4, wherein: The feature processing module includes a first convolution block, a second convolution block and a third convolution block. The feature processing module is suitable for performing convolution processing on the first feature matrix at each scale according to the first convolution block, the second convolution block and the third convolution block to generate a second feature matrix, thereby obtaining multiple second feature matrices.

7. The device according to any one of claims 2 to 6, wherein: The convolutional block attention module is adapted to determine a channel weight coefficient according to each second feature matrix; Performing element-level calculation according to the channel weight coefficient and the second characteristic matrix to generate an intermediate characteristic matrix; Determining a spatial weight coefficient according to the intermediate feature matrix; A third characteristic matrix is ​​generated by performing element-level calculation based on the spatial weight coefficient and the intermediate characteristic matrix.

8. The device according to any one of claims 2 to 7, wherein: The feature warning module includes a first fully connected layer, a second fully connected layer and a third fully connected layer, and the feature warning module is suitable for processing each third feature matrix according to the first fully connected layer, the second fully connected layer and the third fully connected layer to generate a feature probability value at each scale; Calculate epilepsy probability values ​​according to characteristic probability values ​​at multiple scales; It is determined whether the epilepsy probability value is greater than a preset alarm threshold. If so, an epilepsy warning is issued.

9. A computing device comprising: one or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in a memory and configured to be executed by one or more processors, and the one or more programs include instructions for executing the steps performed by the epileptic seizure warning device based on multimodal physiological data acquisition and synchronization according to claims 1 to 8.

10. A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, enable the computing device to execute the steps performed by the epileptic seizure warning device based on multimodal physiological data acquisition and synchronization according to claims 1 to 8.

Images