Multi-modal data processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a multi-modal data processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring multi-modal neural signal data corresponding to a target object; extracting consciousness characteristic parameters from the multi-modal neural signal data; the consciousness feature parameters are converted into consciousness feature vectors, the consciousness feature vectors are fused into a target feature vector, and the target feature vector is a vector used for representing the consciousness state of the target object; inputting the feature vector into a pre-training model of consciousness state classification, and outputting a consciousness state category corresponding to the target object; the pre-training model of consciousness state classification is a model which is obtained through training based on historical sample data of a plurality of patients and is used for determining the consciousness state classification of the to-be-tested person. According to the embodiment of the invention, the problem of low classification accuracy of consciousness states of patients with disturbance of consciousness in the prior art can be solved.

Description

Technical Field

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a multi-modal data processing method, apparatus, electronic device, and storage medium. Background Art

[0002] Disorders of consciousness (DOC) refer to a state of loss of consciousness caused by various severe brain injuries, which further leads to a brain function impairment disease in which an individual's perception of themselves and the outside world is weakened or even disappears, including coma state, vegetative state (VS), minimally conscious state (MCS), etc. For a patient to recover from a disorder of consciousness to a normal person, they need to progress from coma to VS and then to MCS. Therefore, detecting the conscious state of a patient is of great significance for the patient's recovery.

[0003] Existing methods for classifying conscious states mainly rely on standardized clinical scales, such as the Coma Recovery Scale–Revised (CRS-R), or use a single neuroimaging or electrophysiological tool, such as electroencephalogram (EEG), functional magnetic resonance imaging (fMRI), and diffusion tensor imaging (DTI). However, the above methods have many limitations. For example, the CRS-R is difficult to identify residual consciousness, so it may misjudge the conscious state of some patients. For example, it may identify an MCS subject with arousal and conscious fluctuations as a VS subject. Therefore, the existing assessment means for conscious states have relatively low accuracy in classifying the conscious states of patients with disorders of consciousness. Summary of the Invention

[0004] In view of the above problems, the present application provides a multi-modal data processing method, apparatus, electronic device, and storage medium to at least solve the technical problem of relatively low accuracy in classifying the conscious states of patients with disorders of consciousness in the related art.

[0005] According to one aspect of the embodiments of the present application, there is provided a multimodal data processing method, including: acquiring multimodal neural signal data corresponding to a target object; extracting consciousness feature parameters from the multimodal neural signal data; converting the consciousness feature parameters into consciousness feature vectors, and fusing the consciousness feature vectors into a target feature vector, where the target feature vector is a vector for characterizing the consciousness state of the target object; inputting the feature vector into a pre-trained model for consciousness state classification, and outputting the consciousness state category corresponding to the target object; the pre-trained model for consciousness state classification is a model for determining the consciousness state classification of a person to be tested, which is trained based on historical sample data of multiple patients.

[0006] According to another aspect of the embodiments of the present application, there is also provided a multimodal data processing device, including: an acquisition unit for acquiring multimodal neural signal data corresponding to a target object; an extraction unit for extracting consciousness feature parameters from the multimodal neural signal data; a conversion and fusion unit for converting the consciousness feature parameters into consciousness feature vectors and fusing the consciousness feature vectors into a target feature vector, where the target feature vector is a vector for characterizing the consciousness state of the target object; a processing unit for inputting the feature vector into a pre-trained model for consciousness state classification and outputting the consciousness state category corresponding to the target object; the pre-trained model for consciousness state classification is a model for determining the consciousness state classification of a person to be tested, which is trained based on historical sample data of multiple patients.

[0007] According to yet another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to execute the above multimodal data processing method through the computer program.

[0008] According to yet another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, in which a computer program is stored, where the computer program is configured to execute the above multimodal data processing method when running.

[0009] In the embodiments of the present application, the following method is adopted: obtaining multi-modal neural signal data corresponding to a target object; extracting consciousness feature parameters from the multi-modal neural signal data; converting the consciousness feature parameters into consciousness feature vectors, and fusing the consciousness feature vectors into a target feature vector, where the target feature vector is a vector used to characterize the consciousness state of the target object; inputting the feature vector into a pre-trained model for consciousness state classification to output the consciousness state category corresponding to the target object; the pre-trained model for consciousness state classification is a model trained based on historical sample data of multiple patients and used to determine the consciousness state classification of a person to be tested. In the above method, by comprehensively analyzing the neural activity pattern of the target object's brain from different perspectives such as time, space, and structure for the multi-modal neural signal data of the target object, the accuracy and comprehensiveness of consciousness state assessment are significantly improved. At the same time, the pre-trained model for consciousness state classification can automatically process and analyze the multi-modal neural signal data, and can also provide a more objective and efficient consciousness state assessment result. This solves the technical problem of the low accuracy of consciousness state classification for patients with disorders of consciousness in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present invention, and constitute a part of this application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention, and do not constitute an improper limitation of the present invention. In the drawings:

[0011] Figure 1 is a schematic diagram of an application environment of an optional multi-modal data processing method according to an embodiment of the present application;

[0012] Figure 2 is a schematic diagram of an application environment of another optional multi-modal data processing method according to an embodiment of the present application;

[0013] Figure 3 is a schematic flowchart of an optional multi-modal data processing method according to an embodiment of the present application;

[0014] Figure 4 is a schematic diagram of the architecture of an optional multi-modal data processing system according to an embodiment of the present application;

[0015] Figure 5 is a schematic diagram of the architecture of an optional multi-modal data processing platform according to an embodiment of the present application;

[0016] Figure 6 is a schematic diagram of the structure of another optional multi-modal data processing device according to an embodiment of the present application;

[0017] Figure 7 is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. Detailed implementation manners

[0018] To enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0019] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above accompanying drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0020] According to one aspect of the embodiments of the present application, a multi-modal data processing method is provided. Optionally, as an alternative implementation manner, the above multi-modal data processing method can be but is not limited to being applied to an application environment as Figure 1 shown. The application environment includes: a terminal device 102 for human-computer interaction with a user, a network 104, and a server 106. Human-computer interaction can be carried out between the user 108 and the terminal device 102, and a multi-modal data processing client runs in the terminal device 102. The above terminal device 102 includes a human-computer interaction screen 1022, a processor 1024, and a memory 1026. The human-computer interaction screen 1022 is used to display the multi-modal neural signal data corresponding to the target object; the processor 1024 is used to send a request for obtaining the classification of the awareness state corresponding to the target object to the server 106. The memory 1026 is used to store the multi-modal neural signal data corresponding to the target object.

[0021] In addition, the server 106 includes a database 1062 and a processing engine 1064. The database 1062 is used to store the multimodal neural signal data corresponding to the target object, and the processing engine 1064 is used to obtain the multimodal neural signal data corresponding to the target object; extract the consciousness feature parameters from the multimodal neural signal data; convert the consciousness feature parameters into consciousness feature vectors, fuse the consciousness feature vectors into target feature vectors, where the target feature vectors are vectors used to characterize the consciousness state of the target object; input the feature vectors into a pre-trained model for consciousness state classification, and output the consciousness state category corresponding to the target object; the pre-trained model for consciousness state classification is a model trained based on the historical sample data of multiple patients and used to determine the consciousness state classification of the person to be tested; and send the consciousness state category corresponding to the target object to the client of the above terminal device 102.

[0022] In one or more embodiments, the above multimodal data processing method of the present application can be applied to Figure 2 the application environment shown. As Figure 2 shown, human-computer interaction can be carried out between the user 202 and the user device 204. The user device 204 includes a memory 206 and a processor 208. In this embodiment, the user device 204 can, but is not limited to, perform the operations performed by the above terminal device 102 and display the consciousness state category corresponding to the target object.

[0023] Optionally, the above terminal device 102 and user device 204 include, but are not limited to, terminals such as mobile phones, tablets, laptops, PCs, medical detection devices, wearable devices, etc. The above network 104 can include, but is not limited to, a wireless network or a wired network. Among them, the wireless network includes: WIFI and other networks that implement wireless communication. The above wired network can include, but is not limited to: wide area network, metropolitan area network, local area network. The above server 106 can include, but is not limited to, any hardware device that can perform calculations. The above server can be a single server, or a server cluster composed of multiple servers, or a cloud server. The above is only an example, and no limitation is made in this embodiment.

[0024] Existing methods for classifying the state of consciousness mainly rely on standardized clinical scales, such as the Coma Recovery Scale–Revised (CRS-R), or use a single neuroimaging or electrophysiological tool, such as Electroencephalogram (EEG), Functional Magnetic Resonance Imaging (fMRI), and Diffusion Tensor Imaging (DTI). However, the above methods have many limitations. For example, the CRS-R is difficult to identify residual consciousness, so it may misjudge the state of consciousness of some patients. For example, patients with minimally conscious state (MCS) with arousal and fluctuating consciousness may be identified as vegetative state (VS) patients. Therefore, the existing assessment methods for the state of consciousness have low accuracy in classifying the state of consciousness of patients with disorders of consciousness.

[0025] To solve the above technical problems, as an optional implementation, as Figure 3 shown, the embodiment of the present application provides a multi-modal data processing method, including the following steps:

[0026] S302, obtaining multi-modal neural signal data corresponding to the target object;

[0027] S304, extracting consciousness characteristic parameters from the multi-modal neural signal data;

[0028] S306, converting the consciousness characteristic parameters into consciousness characteristic vectors, and fusing the consciousness characteristic vectors into a target characteristic vector, where the target characteristic vector is a vector used to characterize the consciousness state of the target object;

[0029] S308, inputting the characteristic vector into a pre-trained model for classifying the state of consciousness, and outputting the class of the state of consciousness corresponding to the target object; the pre-trained model for classifying the state of consciousness is a model trained based on the historical sample data of multiple patients and used to determine the classification of the state of consciousness of the person to be tested.

[0030] Specifically, in the embodiments of the present application, the target object can be a person with normal consciousness or a person with consciousness disorders. Disorders of consciousness (DoC) refer to the state of loss of consciousness caused by various severe brain injuries, such as vegetative state (VS) and minimally conscious state (MCS). The VS state refers to the state of preserving the basic brainstem reflexes and the sleep-wake cycle, with spontaneous eye opening or eye opening in response to stimulation, but without conscious content. The MCS state refers to the clear signs of consciousness that are discontinuous and fluctuating in patients after severe brain injury. Its main feature is the behavioral activities in which the fluctuations of signs of consciousness can be repeatedly detected. From the perspective of the complexity level of behavioral responses, MCS can be subdivided into two subcategories: "MCS+" and "MCS-". "MCS+" refers to the activities such as eye movement, eye opening and closing, or limb movement that follow simple instructions, but still cannot complete functional communication with the outside world or use items purposefully. "MCS-" refers to the activities such as visual tracking, pain localization, and directional voluntary movement, but cannot complete the activities of following simple instructions. Prolonged disorders of consciousness (pDoC) refer to the disorders of consciousness in which the loss of consciousness exceeds 28 days. The patient may have various functional disorders, including disorders of consciousness, cognition, emotion, swallowing, speech, urination and defecation, motor function, etc.

[0031] The above-mentioned multimodal neural signal data can include, for example, electroencephalogram (EEG), functional magnetic resonance imaging (fMRI), and diffusion tensor imaging (DTI). Among them, EEG data records the electrical activities of the human brain. fMRI data provides images of brain activities and structures. DTI data characterizes the integrity and directionality of the white matter fibers in the brain.

[0032] The above-mentioned consciousness characteristic parameters can be, for example, the spectral characteristics in EEG, such as the absolute power and relative power of different frequency bands. The consciousness characteristic parameters can also be the activity intensity of the default mode network DMN or other functional connectivity indicators in fMRI, as well as the FA (fractional anisotropy) and MD (mean diffusivity) values in DTI.

[0033] After converting each obtained consciousness feature parameter into a corresponding consciousness feature vector, the consciousness feature vectors are fused into a high-dimensional target feature vector, which can represent the consciousness state of the target object. Finally, the fused target feature vector is input into a pre-trained consciousness state classification model. The consciousness state classification model is trained based on the historical sample data of multiple patients and is used to determine the consciousness state classification of the person to be tested. The output of the model can be the consciousness state category corresponding to the target object, such as different consciousness state categories like VS (vegetative state), MCS (minimally conscious state), or two subcategories of "MCS+" and "MCS-".

[0034] In the embodiment of the present application, the method includes obtaining multi-modal neural signal data corresponding to a target object; extracting consciousness feature parameters from the multi-modal neural signal data; converting the consciousness feature parameters into consciousness feature vectors, and fusing the consciousness feature vectors into a target feature vector, where the target feature vector is a vector used to represent the consciousness state of the target object; inputting the feature vector into a pre-trained model for consciousness state classification to output the consciousness state category corresponding to the target object; the pre-trained model for consciousness state classification is a model trained based on the historical sample data of multiple patients and is used to determine the consciousness state classification of the person to be tested. In the above method, by comprehensively analyzing the neural activity pattern of the target object's brain from different perspectives such as time, space, and structure of the multi-modal neural signal data of the target object, the accuracy and comprehensiveness of consciousness state evaluation are significantly improved. At the same time, the pre-trained model for consciousness state classification can automatically process and analyze multi-modal neural signal data and can also provide a more objective and efficient consciousness state evaluation result. This solves the technical problem of low accuracy in consciousness state classification for patients with disorders of consciousness in the related art.

[0035] In one or more embodiments, the multi-modal neural signal data includes electroencephalogram signal data and neuroimaging data. The obtaining of the multi-modal neural signal data corresponding to the target object includes at least one of the following:

[0036] Obtaining electroencephalogram signal data corresponding to the target object through an electroencephalogram device; the electroencephalogram signal data includes spontaneous electroencephalogram signals and evoked electroencephalogram signals;

[0037] Obtaining functional neuroimaging data corresponding to the target object through a functional magnetic resonance imaging device;

[0038] Obtaining structural neuroimaging data corresponding to the target object through a diffusion tensor imaging device.

[0039] Specifically, in the embodiments of the present application, the electroencephalogram (EEG) signal data obtained by the EEG device includes two types: (1) Spontaneous EEG signals: These signals are the electrical activities naturally generated by the brain without external stimuli, which can reflect the natural state of the brain and the intrinsic activities of the neural network.

[0040] (2) Evoked EEG signals: These signals are caused by specific external stimuli (such as light, sound, touch, or a specific task), which can reflect the brain's response to these stimuli.

[0041] The functional neuroimaging data obtained by the functional magnetic resonance imaging (fMRI) device can reveal the activity patterns of the brain when performing specific tasks or in specific states. fMRI reflects neural activity by detecting changes in blood flow in the brain.

[0042] Structural neuroimaging data is obtained by a diffusion tensor imaging (DTI) device. This structural neuroimaging data focuses on the structural characteristics of the brain's white matter and evaluates the integrity and directionality of nerve fibers by measuring the diffusion of water molecules in the brain's white matter. DTI can provide detailed information about white matter fiber bundles, such as fractional anisotropy (FA) and mean diffusivity (MD).

[0043] By integrating multimodal neuro-signal data such as EEG, fMRI, and DTI, comprehensive information about brain function and structure can be provided, which helps to more accurately classify and evaluate the consciousness state of the target object.

[0044] In one or more embodiments, the consciousness characteristic parameters include the absolute power of EEG signals in different frequency bands corresponding to the spontaneous EEG signals. Extracting consciousness characteristic parameters from the multimodal neuro-signal data includes:

[0045] Obtaining the absolute power of EEG signals in each frequency band corresponding to the target object through formula (1);

[0046]

[0047] where P absolute represents the absolute power of the EEG signal in the current frequency band, X(f) is the spectrum of the EEG signal, n1 and n2 are the upper and lower limit values of the frequency of the current frequency band respectively, and df represents the integration of the frequency f in the range from n1 to n2.

[0048] Specifically, in the implementation of this application, in the VS state, the commonly seen higher-power Delta (δ) segment and Theta (θ) segment, as well as the lower Alpha (α) segment and Gamma (γ) segment, that is, the low-frequency proportion is relatively high and the high-frequency proportion is relatively low. In the MCS state, the common δ segment, θ segment, and α segment gradually return to the normal level, that is, the low-frequency proportion decreases and the high-frequency proportion increases. Among them, the δ wave is the brain wave in the range of 1 - 4 Hz, the θ wave is the brain wave in the range of 4 - 8 Hz, the α wave is the brain wave in the range of 8 - 12 Hz, the β wave is the brain wave in the range of 12 - 30 Hz, and the γ wave is the brain wave in the range of 30 - 100 Hz.

[0049] In one or more embodiments, the consciousness characteristic parameter further includes the relative power of the electroencephalogram signals in different frequency bands corresponding to the spontaneous electroencephalogram signal. Extracting the consciousness characteristic parameter from the multi-modal neural signal data further includes:

[0050] Obtaining the total power of the electroencephalogram signals in each frequency band corresponding to the target object through formula (2);

[0051] P total = P δ + P θ + P α + P β + P γ (2)

[0052] Among them, P total represents the total power of all frequency band electroencephalogram signals, P δ represents the absolute power of the δ wave (in the range of 1 - 4 Hz), P θ represents the absolute power of the θ wave (in the range of 4 - 8 Hz), P α represents the absolute power of the α wave (in the range of 8 - 12 Hz), P β represents the absolute power of the β wave (in the range of 12 - 30 Hz), P γ represents the absolute power of the γ wave (in the range of 30 - 100 Hz);

[0053] Obtaining the relative power of the electroencephalogram signals in each frequency band corresponding to the target object through formula (3);

[0054]

[0055] Among them, r δ represents the relative power of the δ wave, r θ represents the relative power of the θ wave, r α represents the relative power of the α wave, r β represents the relative power of the β wave, r γ represents the relative power of the γ wave.

[0056] In one or more embodiments, the awareness characteristic parameter further includes the spectral entropy corresponding to the spontaneous electroencephalogram signal, and extracting the awareness characteristic parameter from the multimodal neural signal data further includes:

[0057] Obtaining the spectral entropy of the spontaneous electroencephalogram signal corresponding to the target object through formula (4);

[0058]

[0059] where H(x, f s ) is the spectral entropy of signal x, P(f) is the normalized probability distribution of the signal power spectral density, and f s is the sampling frequency of the signal.

[0060] Specifically, in the implementation of this application, compared with the VS state, the spectral entropy of the electroencephalogram signal in the MCS state is higher, the functional connectivity of the δ-band brain wave decreases, and the functional connectivity of the θ-band brain wave and the α-band brain wave increases.

[0061] In one or more embodiments, the awareness characteristic parameter further includes the clustering coefficient and the participation coefficient corresponding to the spontaneous electroencephalogram signal, and extracting the awareness characteristic parameter from the multimodal neural signal data includes:

[0062] Calculating the correlation between the signals recorded by different electrodes based on the spontaneous electroencephalogram signal to obtain the clustering coefficient and the participation coefficient; the clustering coefficient is used to characterize the density of the mutual connection between the neighbors of a node; the participation coefficient is used to characterize the distribution uniformity of a node among different modules in the brain neural network; the node represents a region of the brain in the electroencephalogram signal.

[0063] Specifically, in the implementation of this application, the brain network characteristics extracted from the functional connectivity of the spontaneous electroencephalogram can also reflect the change of the awareness state. For example, there are differences in the clustering coefficient (measuring the connection density between the neighbors of a node) and the participation coefficient (measuring the distribution uniformity of a node among different modules) in different states. Specifically, the VS state of the δ-band brain wave is higher than that of healthy people, while the VS state of the α-band brain wave is lower than that of healthy people.

[0064] In one or more embodiments, the awareness characteristic parameter includes the mismatch negativity signal induced by auditory stimulation corresponding to the evoked electroencephalogram signal, and the perturbation complexity index calculated after transcranial magnetic stimulation. Extracting the awareness characteristic parameter from the multimodal neural signal data includes:

[0065] Obtaining the mismatch negativity signal corresponding to the target object;

[0066] Obtaining the perturbation complexity index PCI corresponding to the target object through formula (5);

[0067]

[0068] where C(X binary ) is the compression length of the binarized signal, and L is the total length of the signal sequence.

[0069] Specifically, in the embodiments of the present application, such as the MMN wave (Mismatch Negative, MMN) evoked by auditory stimuli, MMN is the brain's automatic response to abnormal auditory stimuli (sounds or changes that do not match expectations), usually related to the brain's automatic perceptual processing ability. The MMN amplitude increases with the recovery of consciousness. Patients with high amplitude and short latency can usually recover to a relatively high level of consciousness. In coma and VS states, common situations of MMN absence or severe attenuation include: coma: amplitude ≤ 0.5 μV, VS: 0.6 - 0.9 μV). In the MCS state, common situations of MMN delay or attenuation include: MCS-: 1.0 - 1.7 μv, MCS+: 1.7 - 2.0 μv), indicating that the patient still has perceptual ability but a low level of consciousness. In addition, there is also PCI after TMS stimulation, which is calculated from the transcranial magnetic stimulation (TMS) - evoked potential (TEP) to evaluate the brain's functional connectivity and consciousness level. In the VS state, the common range of PCI is 0 - 0.31, and in the MCS state, the common range of PCI is 0.31 - 0.44.

[0070] In one or more embodiments, the consciousness characteristic parameter includes the coherence index corresponding to the default mode network in the functional neuroimaging data. The functional neuroimaging data includes the coherence index corresponding to the default mode network, and the default mode network is a group of functionally connected regions in the human brain; extracting the consciousness characteristic parameter from the multimodal neural signal data includes: obtaining the coherence index corresponding to the target object through formula (6);

[0071]

[0072] where C xy (f) represents the common power of signals x(t) and y(t) at frequency f, P xy (f) is the cross - power spectral density of signals x(t) and y(t), P xx (f) is the auto - power spectral density of signal x(t), P yy (f) is the auto - power spectral density of signal y(t).

[0073] Specifically, in the embodiments of the present application, the Default Mode Network (DMN) is a set of functionally connected regions in the brain, which is closely related to consciousness, attention, and emotional regulation. In particular, the activity intensity and connection intensity of the posterior cingulate cortex or the precuneus in the DMN are significantly positively correlated with the consciousness level of DoC patients. Compared with healthy people, the activity intensity of the DMN in DoC patients is usually significantly reduced, especially close to zero in the VS state. The activity intensity of the DMN is often reflected by the above coherence index. Coherence is a frequency-domain index used to quantify the linear correlation between two time-series signals, which reflects the interaction or connection degree between two brain regions, especially the synchrony in the frequency domain.

[0074] In one or more embodiments, the consciousness characteristic parameter includes the fractional anisotropy index FA in the structural neuroimaging data. The fractional anisotropy index is used to characterize the degree of diffusion anisotropy of water molecules in the white matter of the brain. Extracting the consciousness characteristic parameter from the multimodal neural signal data includes: obtaining the FA corresponding to the target object through formula (7);

[0075]

[0076] Wherein, λ1, λ2, and λ3 are the three eigenvalues of the diffusion tensor, respectively representing the diffusion intensities of the diffusion tensor in three main directions. λ1 is the diffusion intensity in the main diffusion direction (along the fiber bundle direction), and λ2 and λ3 represent the diffusion intensities perpendicular to the main direction.

[0077] Specifically, in the embodiments of the present application, fractional anisotropy (FA) is a parameter for measuring the diffusion directionality (or anisotropy) of water molecules. A low FA value usually means damage or lesion of the white matter structure. In the coma and VS states, the FA value often decreases significantly, especially in the brain regions closely related to consciousness function (brainstem, subcortex, thalamus).

[0078] In one or more embodiments, the consciousness characteristic parameter further includes the mean diffusivity MD in the structural neuroimaging data. Extracting the consciousness characteristic parameter from the multimodal neural signal data includes: obtaining the MD corresponding to the target object through formula (8);

[0079]

[0080] Wherein, λ1, λ2, and λ3 are the three eigenvalues of the diffusion tensor, respectively representing the diffusion intensities of the diffusion tensor in three main directions. λ1 is the diffusion intensity in the main diffusion direction (along the fiber bundle direction), and λ2 and λ3 represent the diffusion intensities perpendicular to the main direction.

[0081] Specifically, in the embodiments of the present application, the mean diffusivity (MD) is a measure of the average diffusivity of water molecules in all directions and is commonly used to evaluate the overall water molecule diffusion characteristics of brain tissue, such as cerebral edema, ischemia, or inflammation. The higher the MD value, the more severe the damage to the white matter fiber structure and the more serious the degree of consciousness disorder.

[0082] In one or more embodiments, the converting the consciousness characteristic parameters into consciousness characteristic vectors and fusing the consciousness characteristic vectors into target characteristic vectors includes:

[0083] Based on a preset deep learning model and a principal component analysis model, converting the electroencephalogram signal data into electroencephalogram characteristic vectors and converting the neuroimaging data into neuroimaging vectors;

[0084] Fusing the electroencephalogram characteristic vectors and the neuroimaging vectors to obtain the target characteristic vectors of a preset dimension.

[0085] Specifically, in the embodiments of the present application, a preset deep learning model and a principal component analysis model can be used to process electroencephalogram signal data, extract key features, and convert them into electroencephalogram characteristic vectors. The deep learning model may include a convolutional neural network (CNN), a recurrent neural network (RNN), or a long short-term memory network (LSTM), and these models can automatically learn and extract useful features from EEG data. In addition, deep learning models or image processing techniques are used to convert fMRI and DTI data into neuroimaging vectors. The deep learning model and the principal component analysis model can identify and extract features related to the consciousness state, such as the DMN activity intensity, FA value, and MD value.

[0086] Then, the feature vectors extracted from EEG and neuroimaging data can be fused by weighted summation, feature concatenation, or using additional deep learning layers to learn the interaction between features. Then the fused feature vectors are converted into a target feature vector of a preset dimension, and this vector is used to characterize the consciousness state of the target object.

[0087] In one or more embodiments, the multimodal data processing method further includes:

[0088] Performing denoising processing and artifact rejection processing on the multimodal neural signal data to obtain preprocessed data;

[0089] The extracting the consciousness characteristic parameters from the multimodal neural signal data includes: extracting the consciousness characteristic parameters from the preprocessed data.

[0090] Specifically, in the embodiments of the present application, the above denoising process and artifact rejection process can be, for example, filtering the collected EEG data, mainly including power frequency filtering of 50HZ, band-pass filtering, rereferencing, independent component analysis, rejecting artifact components, interpolating bad leads, rejecting bad segments, etc.; removing unstable time points, time slice correction, head motion correction, spatial normalization, spatial smoothing, removing linear drift, removing spikes, filtering, regression of covariates, and removing time points with excessive head motion from the fMRI data; performing head motion correction, eddy current correction, spatial normalization, noise removal, and fiber direction estimation on the DTI data to improve the data quality of multimodal neural signal data.

[0091] In one or more embodiments, the multimodal data processing method further includes: outputting the probability of consciousness recovery corresponding to the target object.

[0092] In one or more embodiments, the method further includes: generating a consciousness state assessment report, where the assessment report includes at least one of the following: a feature weight map corresponding to each consciousness feature parameter of the target object, the consciousness state category corresponding to the target object, and the probability of consciousness recovery of the target object.

[0093] Specifically, to solve the problems of low accuracy and single information in traditional consciousness state assessment methods. In the embodiments of the present application, a consciousness state assessment system for DoC patients based on multimodal neural signal data combined with an AI algorithm is also provided. As Figure 4 shown, the overall framework of this assessment system consists of six modules, specifically including the following modules:

[0094] (1) Data acquisition module: Use EEG to collect the electroencephalogram data of DoC patients, and use fMRI and DTI to collect the functional and structural neuroimaging data of the patients.

[0095] (2) Data preprocessing module: Filter the collected EEG data (mainly including power frequency filtering of 50HZ, band-pass filtering, rereferencing, independent component analysis, rejecting artifact components, interpolating bad leads, rejecting bad segments); remove unstable time points, time slice correction, head motion correction, spatial normalization, spatial smoothing, removing linear drift, removing spikes, filtering, regression of covariates, and removing time points with excessive head motion from the fMRI data; perform head motion correction, eddy current correction, spatial normalization, noise removal, and fiber direction estimation on the DTI data to ensure data quality.

[0096] (3) Feature extraction module, extract features from data of different modalities.

[0097] (4) Multimodal feature fusion module, which integrates EEG, fMRI, and DTI feature vectors into a high-dimensional feature vector through PCA and deep learning methods to form a complete representation of the consciousness state.

[0098] (5) AI algorithm evaluation module, which inputs the fused feature vector into a pre-trained AI model (such as a convolutional neural network or a long short-term memory network); the model classifies the consciousness state of the patient according to the learned feature patterns, and determines whether the consciousness state belongs to VS, MCS, or others; combined with the data of previous DoC patients, the AI model can predict the probability of the patient's consciousness recovery.

[0099] (6) Result output module, the system generates a consciousness state evaluation report, including an important feature weight map, the consciousness state classification result, and the consciousness recovery probability; doctors can formulate personalized treatment plans based on the report content.

[0100] By integrating EEG, fMRI, and DTI multimodal neural signal data, this system comprehensively analyzes the neural activity patterns of the brains of DoC patients from different perspectives such as time, space, and structure, thereby improving the accuracy and comprehensiveness of consciousness state evaluation. At the same time, the AI algorithm of the system can automatically process and analyze multimodal data, thereby providing more objective and efficient consciousness state evaluation results. In addition, by learning the historical data of a large number of patients, the AI algorithm model can predict the probability of consciousness recovery, providing a scientific basis for formulating personalized rehabilitation plans.

[0101] In an application embodiment, the embodiment of the present application also provides a cloud platform in clinical applications. This cloud platform is based on the above-mentioned DoC patient consciousness state evaluation system that combines multimodal neural signal data with an AI algorithm. Through cloud deployment, multiple application institutions can share the AI algorithm model of the system and obtain the consciousness state evaluation results of DoC patients in a timely manner, improving the practicability and efficiency of the consciousness state evaluation system. As Figure 5 shown, this cloud platform consists of three parts, specifically including:

[0102] (1) Data upload module, application institutions can upload the collected EEG, fMRI, and DTI data of patients to the cloud platform.

[0103] (2) Cloud analysis module, which automatically evaluates the multimodal neural signal data such as EEG, fMRI, and DTI of patients on the cloud platform.

[0104] (3) Report download module, each application institution can download the evaluation report in real time, and doctors can understand the patient's consciousness state and predict consciousness recovery through the report content.

[0105] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0106] According to another aspect of the embodiments of the present application, there is also provided a multimodal data processing device for implementing the above multimodal data processing method. As Figure 6 shown, the device includes:

[0107] An acquisition unit 602, configured to acquire multimodal neural signal data corresponding to a target object;

[0108] An extraction unit 604, configured to extract consciousness feature parameters from the multimodal neural signal data;

[0109] A conversion and fusion unit 606, configured to convert the consciousness feature parameters into consciousness feature vectors, and fuse the consciousness feature vectors into a target feature vector, where the target feature vector is a vector used to represent the consciousness state of the target object;

[0110] A processing unit 608, configured to input the feature vector into a pre-trained model for consciousness state classification, and output a consciousness state category corresponding to the target object; the pre-trained model for consciousness state classification is a model trained based on historical sample data of multiple patients and used to determine the consciousness state classification of a user to be measured.

[0111] In the embodiments of the present application, the method includes: obtaining multi-modal neural signal data corresponding to a target object; extracting consciousness characteristic parameters from the multi-modal neural signal data; converting the consciousness characteristic parameters into consciousness characteristic vectors, and fusing the consciousness characteristic vectors into a target characteristic vector, where the target characteristic vector is a vector used to characterize the consciousness state of the target object; inputting the characteristic vector into a pre-trained model for consciousness state classification to output the consciousness state category corresponding to the target object; the pre-trained model for consciousness state classification is a model trained based on historical sample data of multiple patients for determining the consciousness state classification of a person to be tested. In the above method, by comprehensively analyzing the neural activity pattern of the target object's brain from different perspectives such as time, space, and structure of the multi-modal neural signal data of the target object, the accuracy and comprehensiveness of consciousness state assessment are significantly improved. At the same time, the pre-trained model for consciousness state classification can automatically process and analyze multi-modal neural signal data, and can also provide a more objective and efficient consciousness state assessment result. This solves the technical problem of low accuracy in consciousness state classification for patients with disorders of consciousness in the related art.

[0112] According to another aspect of the embodiments of the present application, there is also provided an electronic device for implementing the above multi-modal data processing method. The electronic device may be Figure 7 the terminal device or server shown in the figure. In this embodiment, the electronic device is taken as an example of a server for illustration. As Figure 7 shown, the electronic device includes a memory 702 and a processor 704. A computer program is stored in the memory 702, and the processor 704 is configured to execute the steps in any one of the above method embodiments through the computer program.

[0113] Optionally, in this embodiment, the above electronic device may be at least one network device among multiple network devices in a computer network.

[0114] Optionally, in this embodiment, the above processor may be configured to execute the following steps through the computer program:

[0115] S1, obtaining multi-modal neural signal data corresponding to a target object;

[0116] S2, extracting consciousness characteristic parameters from the multi-modal neural signal data;

[0117] S3, converting the consciousness characteristic parameters into consciousness characteristic vectors, and fusing the consciousness characteristic vectors into a target characteristic vector, where the target characteristic vector is a vector used to characterize the consciousness state of the target object;

[0118] S4. Input the feature vector into a pre-trained model for classifying the state of consciousness, and output the state-of-consciousness category corresponding to the target object. The pre-trained model for classifying the state of consciousness is a model for determining the state-of-consciousness classification of a user to be tested, which is trained based on the historical sample data of multiple patients.

[0119] Optionally, those of ordinary skill in the art can understand that Figure 7 The structure shown is only schematic. The electronic device can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, and terminal devices such as Mobile Internet Devices (MID), PAD, etc. Figure 7 It does not limit the structure of the above-mentioned electronic device. For example, the electronic device may further include Figure 7 more or fewer components (such as a network interface, etc.) than those shown, or have a different configuration from Figure 7 that shown.

[0120] Among them, the memory 702 can be used to store software programs and modules, such as the program instructions / modules corresponding to the multi-modal data processing method and device in the embodiments of the present application. The processor 704 executes various functional applications and data processing by running the software programs and modules stored in the memory 702, that is, implements the above-mentioned multi-modal data processing method. The memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 702 may further include a memory remotely disposed relative to the processor 704, and these remote memories can be connected to the terminal through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof. Specifically, the memory 702 may be used but is not limited to storing the feature weight maps corresponding to the respective consciousness feature parameters of the target object, the state-of-consciousness category corresponding to the target object, and the consciousness recovery probability of the target object. As an example, as Figure 7 shown, the above memory 702 may include but is not limited to the acquisition unit 702, the extraction unit 704, the conversion and fusion unit 706, and the processing unit 706 in the above multi-modal data processing device. In addition, it may also include but is not limited to other module units in the above multi-modal data processing device, which will not be elaborated in this example.

[0121] Optionally, the above-mentioned transmission device 706 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired network and a wireless network. In one example, the transmission device 707 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers through a network cable, so as to communicate with the Internet or a local area network. In one example, the transmission device 706 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0122] In addition, the above-mentioned electronic device further includes: a display 708, which is used to display the characteristic weight map corresponding to each consciousness characteristic parameter of the target object, the consciousness state category corresponding to the target object, and the consciousness recovery probability of the target object; and a connection bus 710, which is used to connect each module component in the above-mentioned electronic device.

[0123] In other embodiments, the above-mentioned terminal device or server can be a node in a distributed system. Among them, the distributed system can be a blockchain system, and the blockchain system can be a distributed system formed by connecting the multiple nodes through network communication. Among them, the nodes can form a peer-to-peer (P2P, Peer To Peer) network, and any form of computing device, such as servers, terminals and other electronic devices, can become a node in the blockchain system by joining the peer-to-peer network.

[0124] According to one aspect of the present application, there is provided a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned multi-modal data processing method, wherein the computer program is set to execute the steps in any one of the above-mentioned method embodiments when running.

[0125] Optionally, in this embodiment, the above-mentioned computer-readable storage medium can be set to store a computer program for executing the following steps:

[0126] S1, obtaining multi-modal neural signal data corresponding to a target object;

[0127] S2, extracting consciousness characteristic parameters from the multi-modal neural signal data;

[0128] S3, converting the consciousness characteristic parameters into consciousness characteristic vectors, and fusing the consciousness characteristic vectors into a target characteristic vector, where the target characteristic vector is a vector used to represent the consciousness state of the target object;

[0129] S4. Input the feature vector into a pre-trained model for classifying the state of consciousness, and output the category of the state of consciousness corresponding to the target object; the pre-trained model for classifying the state of consciousness is a model for determining the classification of the state of consciousness of a user to be tested, which is trained based on the historical sample data of multiple patients.

[0130] Optionally, in this embodiment, those of ordinary skill in the art can understand that all or part of the steps in the above various methods can be completed by instructing the relevant hardware of the terminal device through a program, and this program can be stored in a computer-readable storage medium. The storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disc, etc.

[0131] The serial numbers of the above embodiments of the present application are only for description and do not represent the superiority or inferiority of the embodiments.

[0132] If the integrated unit in the above embodiments is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing one or more computer devices (which may be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention.

[0133] In the above embodiments of the present invention, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0134] In the several embodiments provided by the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, and the indirect coupling or communication connection of units or modules may be in an electrical or other form.

[0135] The unit described as a separation component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed over multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0136] In addition, each functional unit in various embodiments of the present invention may be integrated in a processing unit, may exist separately as individual physical units, or two or more units may be integrated in one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0137] The above is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A multimodal data processing method, characterized in that The method includes: Obtaining multi-modal neural signal data corresponding to a target object; Extracting consciousness characteristic parameters from the multi-modal neural signal data; Converting the consciousness characteristic parameters into consciousness characteristic vectors, and fusing the consciousness characteristic vectors into a target characteristic vector, where the target characteristic vector is a vector used to characterize the consciousness state of the target object; Inputting the characteristic vector into a pre-trained model for consciousness state classification, and outputting the consciousness state category corresponding to the target object; the pre-trained model for consciousness state classification is a model trained based on historical sample data of multiple patients and used to determine the consciousness state classification of a person to be tested.

2. The method according to claim 1, characterized in that, The multi-modal neural signal data includes electroencephalogram (EEG) signal data and neuroimaging data. Obtaining the multi-modal neural signal data corresponding to the target object includes at least one of the following: Obtaining EEG signal data corresponding to the target object through an EEG device; the EEG signal data includes spontaneous EEG signals and evoked EEG signals; Obtaining functional neuroimaging data corresponding to the target object through a functional magnetic resonance imaging (fMRI) device; Obtaining structural neuroimaging data corresponding to the target object through a diffusion tensor imaging (DTI) device.

3. The method according to claim 2, wherein The consciousness characteristic parameters include the absolute power of EEG signals in different frequency bands corresponding to the spontaneous EEG signals. Extracting the consciousness characteristic parameters from the multi-modal neural signal data includes: Obtaining the absolute power of EEG signals in each frequency band corresponding to the target object through formula (1); where P absolute represents the absolute power of the EEG signal in the current frequency band, X(f) is the spectrum of the EEG signal, n1 and n2 are respectively the upper and lower limit values of the frequency of the current frequency band, and df represents the integration of the frequency f in the range from n1 to n2.

4. The method according to claim 3, characterized in that The consciousness characteristic parameters further include the relative power of EEG signals in different frequency bands corresponding to the spontaneous EEG signals. Extracting the consciousness characteristic parameters from the multi-modal neural signal data further includes: Obtaining the total power of EEG signals in each frequency band corresponding to the target object through formula (2); P total = P δ + P θ + P α + P β + P γ (2) Among them, P total represents the total power of all frequency band EEG signals, P δ represents the absolute power of the δ wave (in the range of 1 - 4 Hz), P θ represents the absolute power of the θ wave (in the range of 4 - 8 Hz), P α represents the absolute power of the α wave (in the range of 8 - 12 Hz), P β represents the absolute power of the β wave (in the range of 12 - 30 Hz), P γ represents the absolute power of the γ wave (in the range of 30 - 100 Hz); Obtaining the relative power of EEG signals in each frequency band corresponding to the target object through formula (3); Among them, r δ represents the relative power of the δ wave, r θ represents the relative power of the θ wave, r α represents the relative power of the α wave, r β represents the relative power of the β wave, r γ represents the relative power of the γ wave.

5. The method according to any one of claims 2 to 4, characterized in that The consciousness characteristic parameters further include the spectral entropy of the spontaneous EEG signals. Extracting the consciousness characteristic parameters from the multi-modal neural signal data further includes: Obtaining the spectral entropy of the spontaneous EEG signals corresponding to the target object through formula (4); Among them, H(x, f s ) is the spectral entropy of the signal x, P(f) is the normalized probability distribution of the signal power spectral density, and f s is the sampling frequency of the signal.

6. The method according to any one of claims 2 to 4, characterized in that, The consciousness characteristic parameters further include the clustering coefficient and participation coefficient of the spontaneous EEG signals. Extracting the consciousness characteristic parameters from the multi-modal neural signal data includes: Calculating the correlation between signals recorded by different electrodes based on the spontaneous EEG signals to obtain the clustering coefficient and participation coefficient; the clustering coefficient is used to characterize the density of the mutual connection between the neighbors of a node; the participation coefficient is used to characterize the distribution uniformity of a node among different modules in the brain neural network; the node represents a region of the brain in the EEG signal.

7. The method according to claim 2, characterized in that The consciousness characteristic parameters include the mismatch negativity (MMN) signal evoked by auditory stimuli corresponding to the evoked EEG signals, and the perturbation complexity index calculated after transcranial magnetic stimulation. Extracting the consciousness characteristic parameters from the multi-modal neural signal data includes: Obtaining the MMN signal corresponding to the target object; Obtaining the perturbation complexity index PCI corresponding to the target object through formula (5); Among them, C(X binary ) is the compression length of the binarized signal, and L is the total length of the signal sequence.

8. The method according to claim 3, wherein The conscious characteristic parameters include the coherence index corresponding to the default mode network in the functional neuroimaging data. The functional neuroimaging data includes the coherence index corresponding to the default mode network, and the default mode network is a set of functionally connected regions in the human brain. Extracting conscious characteristic parameters from the multimodal neural signal data includes: obtaining the corresponding coherence index of the target object through formula (6). Among them, C xy (f) represents the common power of signals x(t) and y(t) at frequency f, P xy (f) is the cross-power spectral density of signals x(t) and y(t), P xx (f) is the auto-power spectral density of signal x(t), P yy (f) is the auto-power spectral density of signal y(t).

9. The method according to claim 3, characterized in that, The conscious characteristic parameters include the fractional anisotropy index FA in the structural neuroimaging data. The fractional anisotropy index is used to characterize the degree of diffusion anisotropy of water molecules in the white matter of the brain. Extracting conscious characteristic parameters from the multimodal neural signal data includes: obtaining the corresponding FA of the target object through formula (7). Where λ1, λ2, and λ3 are the three eigenvalues of the diffusion tensor, representing the diffusion intensities of the diffusion tensor in three main directions respectively. λ1 is the diffusion intensity in the main diffusion direction (along the fiber bundle direction), and λ2 and λ3 represent the diffusion intensities perpendicular to the main direction.

10. The method according to claim 9, characterized in that, The conscious characteristic parameters further include the mean diffusivity MD in the structural neuroimaging data. Extracting conscious characteristic parameters from the multimodal neural signal data includes: obtaining the corresponding MD of the target object through formula (8). Where λ1, λ2, and λ3 are the three eigenvalues of the diffusion tensor, representing the diffusion intensities of the diffusion tensor in three main directions respectively. λ1 is the diffusion intensity in the main diffusion direction (along the fiber bundle direction), and λ2 and λ3 represent the diffusion intensities perpendicular to the main direction.

11. The method according to claim 2, characterized in that Converting the conscious characteristic parameters into a conscious characteristic vector and fusing the conscious characteristic vectors into a target characteristic vector includes: Based on a preset deep learning model and principal component analysis model, converting the electroencephalogram signal data into an electroencephalogram characteristic vector and converting the neuroimaging data into a neuroimaging vector. Fusing the electroencephalogram characteristic vector and the neuroimaging vector to obtain the target characteristic vector of a preset dimension.

12. The method according to claim 1 or 2, characterized in that, The method further includes: Performing denoising processing and artifact removal processing on the multimodal neural signal data to obtain preprocessed data. Extracting conscious characteristic parameters from the multimodal neural signal data includes: extracting conscious characteristic parameters from the preprocessed data.

13. The method according to claim 1 or 2, characterized in that, The method further includes: outputting the consciousness recovery probability corresponding to the target object.

14. The method according to claim 1 or 2, characterized in that, The method further includes: Generating a consciousness state evaluation report, where the evaluation report includes at least one of the following: the characteristic weight map corresponding to each conscious characteristic parameter of the target object, the conscious state category corresponding to the target object, and the consciousness recovery probability corresponding to the target object.

15. A multimodal data processing device, characterized in that, The device includes: An acquisition unit for acquiring multimodal neural signal data corresponding to a target object. An extraction unit for extracting conscious characteristic parameters from the multimodal neural signal data. A conversion and fusion unit for converting the conscious characteristic parameters into a conscious characteristic vector and fusing the conscious characteristic vectors into a target characteristic vector, where the target characteristic vector is a vector used to characterize the conscious state of the target object. A processing unit for inputting the feature vector into a pre-trained model for classifying the state of consciousness, and outputting the state-of-consciousness category corresponding to the target object; the pre-trained model for classifying the state of consciousness is a model for determining the state-of-consciousness classification of a person to be tested, which is trained based on historical sample data of multiple patients.

16. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor runs the computer program to implement the method according to any one of claims 1-14.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the steps of the method according to any one of claims 1-14.

Images