An automatic sleep staging method, system, apparatus, and storage medium

Abstract

The application discloses an automatic sleep staging method, system, device and storage medium, wherein the method comprises the following steps: acquiring a polysomnogram of a sleeper; extracting physiological signal features from the polysomnogram; performing an adversarial enhancement process on the physiological signal features to obtain enhanced physiological signal features; learning high-order correlation relationships between multi-modal physiological signals based on the enhanced physiological signal features; performing multi-modal fusion according to the high-order correlation relationships; and obtaining sleep stage classification results. The application obtains physiological signal data with balanced information quantity through adversarial enhancement of physiological signals, and realizes automatic sleep state detection through data fusion, thereby improving the prediction accuracy. In addition, the high-order correlation between modalities is captured for the collected multi-modal physiological signal features, and the redundant information and correlation information of the multi-modal physiological signals are effectively utilized for feature fusion. The application can be widely applied to the technical field of sleep analysis.

Description

Technical Field

[0001] This invention relates to the field of sleep analysis technology, and in particular to an automatic sleep staging method, system, device, and storage medium. Background Technology

[0002] Sleep is one of the most important physiological activities in humans. Insufficient sleep and poor sleep quality can directly lead to various physiological and mental illnesses, such as sleep apnea syndrome, depression, and cognitive decline. Sleep staging can serve as an effective method for analyzing and assessing a patient's sleep quality. Clinically, polysomnography (PSG) is used to record multimodal physiological signals during sleep. PSG contains various multimodal physiological signals, such as electroencephalography (EEG), electrooculography (EOG), and electromyography (EMG). Currently, the most commonly used staging method divides sleep states into five categories: rapid eye movement (REM), non-rapid eye movement (NREM) type 3, and wakefulness. Current manual staging methods mainly rely on sleep experts observing the waveforms of various physiological signals in PSG for calibration. This method requires sleep experts to spend a significant amount of time professionally judging massive amounts of PSG fragments and is also subject to subjective issues. Therefore, an automated sleep stage staging scheme based on the fusion of multimodal physiological signal features is of great significance.

[0003] In practical applications, single-modal physiological signals often fail to fully reflect the physiological state of the human body, while physiological signals from different modalities carry complementary information. Multimodal physiological signal fusion can fully utilize the complementarity of various signals and reduce the misdiagnosis rate. Common physiological signal fusion methods first extract various features of physiological signals, and then fuse these features based on deep learning methods. These methods mainly include feature concatenation, convolutional neural networks (CNNs), attention-based fusion, and bilinear pooling fusion.

[0004] Existing automatic sleep staging schemes primarily collect multimodal human physiological signals, extract features from these signals, and use machine learning and deep learning algorithms to identify each sleep stage. However, existing solutions have the following drawbacks: 1) PSG mainly contains multiple EEG channels, while some modalities only have a few channels, resulting in an imbalance of modal information. 2) They do not consider the higher-order correlations between multimodal physiological signals, i.e., the synergistic correlations between various physiological signals and sleep stages, thus leading to insufficient information fusion. 3) Existing automatic sleep staging ignores the data distribution differences caused by different data collection groups, resulting in staging errors for some groups. Summary of the Invention

[0005] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide an automatic sleep staging method, system, device and storage medium based on a multimodal hypergraph fusion network.

[0006] The first technical solution adopted in this invention is:

[0007] An automatic sleep staging method includes the following steps:

[0008] Obtain polysomnography during the subject's sleep;

[0009] Physiological signal features are extracted from polysomnography, and adversarial enhancement processing is applied to these physiological signal features to obtain enhanced physiological signal features.

[0010] Based on enhanced physiological signal features, higher-order correlations between multimodal physiological signals are learned, multimodal fusion is performed based on the higher-order correlations, and sleep stage classification results are obtained.

[0011] Furthermore, acquiring the polysomnography of the subject during sleep includes:

[0012] Physiological signals from subjects in multiple modalities were collected, including electroencephalogram (EEG), eye movement (EOG), electromyography (EMG), and electrocardiogram (ECG).

[0013] The collected physiological signals from multiple modalities were amplified and filtered to obtain polysomnography.

[0014] Furthermore, the step of extracting physiological signal features from polysomnography and performing adversarial enhancement processing on these features to obtain enhanced physiological signal features includes:

[0015] After the polysomnography is segmented into time segments according to a preset period, feature extraction is performed to obtain physiological signal features; among which, the extracted features include power spectral density features, differential entropy features, and frequency domain statistical features.

[0016] Based on the obtained physiological signal characteristics, cross-modal physiological signal data conversion is performed using generative adversarial networks. Finally, feature mixing of various physiological signals is performed using channel feature mixing technology to obtain enhanced physiological signal characteristics.

[0017] Furthermore, the generative adversarial network includes a discriminator D and encoders and decoders corresponding to each modality;

[0018] The feature distribution of the encoded mode m1 is defined as follows:

[0019]

[0020] In the formula, It is the encoder equation, θ m1 These are the parameters of the equation; p(x) m1 ) is the distribution of features of mode m1 before encoding; x m1 It is the initial feature of mode m1, that is, the physiological signal feature of mode m1 before it has undergone adversarial enhancement; These are the features of mode m1 after encoding;

[0021] The discriminator D is used to determine whether the modal features generated by each encoder are the corresponding modal features. The discriminant loss function is defined as follows:

[0022]

[0023] The error adversarial loss function is defined as follows:

[0024]

[0025] The correct adversarial loss function is defined as follows:

[0026]

[0027] In the formula, For the features of the encoded mode m2, Let w be the features of the encoded modality m3, w be the trainable parameters, and D be the discriminator.

[0028] The expression for the decoder is:

[0029]

[0030] In the formula, It is the decoder equation, θ dm1 These are the parameters of the equation; It is the distribution of features of mode m1 after decoding; These are the characteristics of mode m1 after being decoded by the decoder;

[0031] For the decoder, define the reconstruction loss as follows:

[0032]

[0033] In the formula, The multimodal physiological signal characteristics after decoding are x f These are physiological signal characteristics that have not been encoded or decoded, and ||||2 is the L2 norm;

[0034] The features of each encoded modality are enhanced by channel blending. The features of the enhanced modality m1 are defined as follows:

[0035]

[0036] In the formula, a and b are learnable parameters, M(x) m1 () is the physiological signal characteristic of the enhanced postmodal m1;

[0037] After channel mixing and enhancement, enhanced physiological signal characteristics are obtained.

[0038] Furthermore, the process of learning higher-order correlations between multimodal physiological signals based on enhanced physiological signal features, performing multimodal fusion based on these higher-order correlations, and obtaining sleep stage classification results includes:

[0039] By constructing a hypergraph fusion module, multimodal fusion of enhanced physiological signal features is performed, and sleep stage classification results are output.

[0040] The hypergraph fusion module includes a hypergraph convolutional layer, a graph feature extraction layer, and a sleep stage classification layer; the hypergraph fusion module operates as follows:

[0041] The enhanced physiological signal features of each time segment are used as features of the hypergraph nodes and input into the hypergraph convolutional layer;

[0042] The hypergraph convolutional layer adaptively learns the hyperedges between hypergraph nodes through model training, learns the high-order correlation of physiological signals, reflects the synergistic effect of various physiological organs during sleep, and updates the hypergraph node features based on the information transmission mechanism.

[0043] The updated hypergraph node features are input into the graph feature reading layer to read the hypergraph features, and the hypergraph representation is classified through the sleep stage classification layer to obtain the corresponding sleep stage classification results.

[0044] Furthermore, the automatic sleep staging method also includes a step of fine-tuning the hypergraph fusion module:

[0045] For the multimodal physiological signals of new subject groups, a portion of the data is sampled and manually labeled, and then compared with the classification results calculated by the hypergraph fusion module to calculate the classification accuracy.

[0046] If the classification accuracy is lower than the preset threshold, the hypergraph fusion module is fine-tuned, and the classification result is obtained.

[0047] The sleep data of the test subjects will be stored in a database.

[0048] Furthermore, the automatic sleep staging method also includes the following steps:

[0049] The sleep quality of the subjects was analyzed based on the sleep stage classification results and a pre-set database.

[0050] The second technical solution adopted in this invention is:

[0051] An automatic sleep staging system includes:

[0052] The polysomnography acquisition module is used to acquire polysomnography data of the subject during sleep.

[0053] The physiological signal adversarial enhancement module is used to extract physiological signal features from polysomnography, perform adversarial enhancement processing on the physiological signal features, and obtain enhanced physiological signal features.

[0054] The hypergraph fusion module is used to learn higher-order correlations between multimodal physiological signals based on enhanced physiological signal features, perform multimodal fusion based on higher-order correlations, and obtain sleep stage classification results.

[0055] The third technical solution adopted in this invention is:

[0056] An automatic sleep staging device includes:

[0057] At least one processor;

[0058] At least one memory for storing at least one program;

[0059] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0060] The fourth technical solution adopted in this invention is:

[0061] A computer-readable storage medium storing a processor-executable program, which, when executed by a processor, performs the method described above.

[0062] The beneficial effects of this invention are as follows: This invention obtains physiological signal data with balanced information content by enhancing the adversarial effect on physiological signals, and achieves automatic sleep state detection through data fusion, thereby improving prediction accuracy. Furthermore, it captures high-order intermodal correlations of the collected multimodal physiological signal features, effectively utilizing the redundant and correlated information of the multimodal physiological signals for feature fusion, making it an accurate and reliable multimodal data fusion method. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a flowchart of an automatic sleep staging system based on a hypergraph fusion model in an embodiment of the present invention;

[0065] Figure 2 This is a schematic diagram of the polysomnography acquisition module in an embodiment of the present invention;

[0066] Figure 3 This is a schematic diagram of the physiological signal countermeasure enhancement module in an embodiment of the present invention;

[0067] Figure 4 This is a schematic diagram of the operation of the hypergraph fusion module in an embodiment of the present invention;

[0068] Figure 5 This is a flowchart illustrating the steps of an automatic sleep staging method in an embodiment of the present invention. Detailed Implementation

[0069] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0070] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0071] In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. If "first" or "second" is used, it is only for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features. Furthermore, "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0072] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0073] like Figure 5 As shown, this embodiment provides an automatic sleep staging method, including the following steps:

[0074] S1. Obtain polysomnography of the subject during sleep.

[0075] First, the subject is placed in a suitable measurement environment, and a polysomnography acquisition module is used to collect polysomnography data during sleep. The data is then sent to a physiological signal countermeasure enhancement module.

[0076] S2. Extract physiological signal features from polysomnography, perform adversarial enhancement processing on the physiological signal features, and obtain enhanced physiological signal features.

[0077] The physiological signal features are extracted by the physiological signal adversarial enhancement module, and the features are then subjected to adversarial enhancement processing to obtain enhanced physiological signal features.

[0078] S3. Based on enhanced physiological signal features, learn the higher-order correlation between multimodal physiological signals, perform multimodal fusion based on the higher-order correlation, and obtain sleep stage classification results.

[0079] The enhanced physiological signal features are input into the hypergraph fusion module to learn the high-order correlation between multimodal physiological signals, and multimodal fusion is performed based on this to obtain sleep stage classification labels.

[0080] S4. Adapt and fine-tune the enhanced data.

[0081] For the multimodal physiological signals of a new subject group, a portion of the data is sampled and manually labeled by clinicians. This data is then compared with the classification results calculated by the hypergraph fusion module to determine the classification accuracy. If the accuracy is below a threshold, the model is fine-tuned using the data adaptation module, and the classification result is obtained. The sleep state data of the subjects is then stored in the database.

[0082] S5. Analyze the sleep quality of the test subjects based on the sleep stage classification results and the preset database.

[0083] Based on the data in the database and the sleep stage classification results of the test subjects, the sleep quality analysis module generates sleep quality reports and early warnings for the test subjects.

[0084] Corresponding to the above methods, such as Figure 1As shown, this embodiment also provides an automatic sleep staging system based on a hypergraph fusion model, including a polysomnography acquisition module, a physiological signal adversarial enhancement module, a hypergraph fusion module, a data adaptation fine-tuning module, and a sleep quality analysis module. After polysomnography acquisition, enhanced physiological signal features are obtained by physiological signal adversarial enhancement, followed by multimodal data fusion. The data is compared with partially manually labeled data by clinicians. If the model labeling accuracy is below a threshold, data adaptation fine-tuning is performed, and the subject's sleep state data is stored in a database. Sleep quality analysis is performed based on the data in the database, generating sleep quality reports and warnings for the subject. Specifically, the detailed operation of each module is as follows:

[0085] (1) Polysomnography acquisition module

[0086] The polysomnography acquisition module collects various physiological signals from the subjects, including electroencephalogram (EEG), eye movement (EOG), electromyography (EMG), and electrocardiogram (ECG), and performs operations such as filtering and power spectral density feature extraction.

[0087] The flowchart of the polysomnography acquisition module is as follows: Figure 2 As shown. The polysomnography acquisition module can be used with a polysomnography monitor for acquisition and processing. First, the electrode pads are placed in contact with the scalp, skin, eyelids, etc., to ensure stable contact between the electrodes and the corresponding acquisition locations. Then, physiological signals from multiple channels are acquired. The acquired physiological signals are pre-amplified to obtain a stronger signal, followed by post-amplification and filtering operations, finally resulting in the processed polysomnography.

[0088] (2) Physiological signal countermeasure enhancement module

[0089] Physiological signal countermeasure enhancement module, such as Figure 3 As shown, the collected physiological signals are segmented into time segments with a period of T, and features are extracted. Common features include power spectral density features, differential entropy features, and frequency domain statistical features. Then, cross-modal physiological signal data conversion is performed based on generative adversarial networks. Finally, feature mixing technology is used to mix the features of each physiological signal to obtain enhanced physiological signal features.

[0090] A generative adversarial network (GAN) consists of a discriminator D and encoders and decoders for each modality. For the data of each modality, taking modality m1 as an example, and m2 and m3 as the other two modalities, the feature distribution of the encoded modality m1 is defined as follows:

[0091]

[0092] in It is the encoder equation, θ m1These are the parameters of the equation, p(x) m1 ) is the distribution of features of mode m1 before encoding, x m1 These are the initial characteristics of mode m1, that is, the physiological signal characteristics of mode m1 before adversarial enhancement. These are the features of mode m1 after encoding.

[0093] The discriminator D is used to determine whether the modal features generated by each encoder are the corresponding modal features. The discriminant loss function is defined as follows:

[0094]

[0095] The error adversarial loss function is defined as follows:

[0096]

[0097] The correct adversarial loss function is defined as follows:

[0098]

[0099] Where w are trainable parameters.

[0100] The decoder is:

[0101]

[0102] in It is the decoder equation, θ dm1 These are the parameters of the equation. It is the distribution of features of mode m1 after decoding. These are the features of mode m1 after being encoded by the encoder. It is the feature of mode m1 after being decoded by the decoder.

[0103] For the decoder, define the reconstruction loss as follows:

[0104]

[0105] Discriminators, encoders, and decoders can be constructed from various deep learning models, such as multilayer perceptrons, recurrent neural networks, and spiking neural networks.

[0106] The features of each encoded modality are enhanced by channel blending. Taking modality m1 as an example, the features of enhanced m1 are defined as follows:

[0107]

[0108] Where a and b are learnable parameters, M(x) m 1) is the physiological signal characteristic of m1 after enhancement.

[0109] After channel mixing and enhancement, enhanced physiological signal characteristics are obtained.

[0110] It should be noted that the physiological signal adversarial enhancement module mentioned above generates enhanced physiological signal features, which solves the problem of modal information imbalance. In addition to the above-mentioned technical means, variational autoencoders, diffusion models, etc. can also be used to perform modal conversion on the data of each modality based on the different heterogeneous features between the physiological signals of each modality.

[0111] (3) Hypergraph Fusion Module

[0112] See Figure 4 The hypergraph fusion module performs multimodal fusion of enhanced physiological signal features. The module includes a hypergraph convolutional layer, a graph feature reading layer, and a sleep stage classification layer. The enhanced physiological signal features of each time segment T are used as features of the hypergraph nodes and input into the hypergraph convolutional layer. Through model training, the hypergraph convolutional layer adaptively learns the hyperedges between hypergraph nodes, learning the high-order correlations of physiological signals, reflecting the synergistic effects of various physiological organs during sleep. It also updates the hypergraph node features based on the information transfer mechanism. The updated hypergraph node features are then input into the graph feature reading layer to read the hypergraph features, and finally, the sleep stage classification layer classifies the hypergraph representation to obtain the corresponding sleep stage prediction result.

[0113] Let the set of hypervertices be V = {v1, v2, ..., vn}, and the set of hyperedges be E = {E1, E2, ..., Em}. Let W be the weight matrix of the hyperedges, and w(e) be the weight of hyperedge e. The degree of a hyperedge is defined as the degree of a hypervertices. De and Dv represent the diagonal matrices of the degree of the hyperedge and the degree of the vertex, respectively.

[0114] Define the model-learnable mapping matrix H to represent the hypergraph structure, where each element h(v, e) in the matrix is ​​defined as:

[0115]

[0116] Define the characteristics of a superpoint:

[0117]

[0118] in This represents the initial characteristics of the nth physiological signal channel.

[0119] The hypergraph convolutional layer is:

[0120]

[0121] in X(l) is the propagation representation of the first-level hypergraph, and X(l) is the original representation of the first-level hypergraph.

[0122] The residual connection layer is:

[0123]

[0124] Where α is a hyperparameter and σ is the activation function.

[0125] The activation function layer can use linear rectified function units:

[0126] RELU(X) = max(0, X)

[0127] Where X represents the input vector of the activation function.

[0128] The image feature reading layer is:

[0129]

[0130] Z represents the fused physiological signal features after reading. After passing the fused physiological signal features through a sleep stage classification layer, a five-category sleep staging result is output. The classification function can be the Softmax function.

[0131] It should be noted that the hypergraph fusion module mentioned above uses a hypergraph neural network to fuse multimodal data. In addition to this technical solution, other algorithms or models can also be used, such as convolutional neural networks (CNN), recurrent neural networks (RNN), bilinear pooling layers, ensemble learning, etc.

[0132] (4) Data adaptation and fine-tuning module

[0133] For data from the new subject group, polysomnography was first collected. Then, clinicians manually labeled a small portion of the data with sleep state stages. The polysomnography was then subjected to physiological signal adversarial enhancement and hypergraph fusion to obtain the model classification results, which were compared with the manually labeled data. If the accuracy was less than a threshold, a new data fine-tuning layer was created in the data adaptation fine-tuning module.

[0134] The data fine-tuning layer is:

[0135] Z′=(W a Z+b)

[0136] Among them W a is a learnable weight matrix, and b is a learnable bias term. Z′ is the fused physiological signal feature obtained after data adaptation. Five-category sleep staging results are output based on a classification function. The classification function can be a softmax function. The sleep staging statistics of each group are stored in the respective group's sleep information database.

[0137] (5) Sleep quality analysis module

[0138] For each participant's physiological signal data, the system analyzes their sleep quality using the sleep quality analysis module and compares it with data from various groups in the database. If a participant's sleep stage distribution is similar to that of a certain group, and that group commonly suffers from a certain disease, a sleep quality report is generated, and an alert is issued to the participant. For example, if a participant's sleep stage distribution is similar to that of a group of patients with obstructive sleep apnea in the database, the sleep quality analysis module will generate a sleep quality report for the participant and issue an alert.

[0139] In summary, compared with the prior art, the present invention has at least the following advantages and beneficial effects:

[0140] (1) This invention proposes a physiological signal adversarial enhancement method to adversarially enhance physiological signal data of various modalities. This method adversarially enhances multimodal physiological signals with imbalanced data volume and channel number, obtaining enhanced physiological signal features, thereby balancing the data volume of different modalities, enhancing the learning ability of modal data with less information, and solving the problem of information imbalance of physiological signals of various modalities during polysomnography acquisition. Specifically, a generative adversarial network is used to perform modality transformation on the data of each modality, converting the modality with more information to other modalities, and further enhancing the physiological signals based on the channel feature mixing method to balance the information volume of each modality.

[0141] (2) This invention proposes a hypergraph fusion method to perform feature fusion on multimodal physiological signal data. Commonly used multimodal fusion methods neglect to capture the high-order correlations between multimodal physiological signal data, making it impossible for the model to fully capture the complementary information of each modality and the modal coordination patterns with sleep stage segmentation. The hypergraph fusion method proposed in this invention can effectively capture the high-order correlations between multimodal physiological signals and perform feature fusion, thereby achieving efficient and accurate automatic sleep stage segmentation.

[0142] (3) This invention proposes a data adaptation fine-tuning method to adapt to the physiological signal characteristics of different groups, and to perform sleep quality analysis and disease early warning based on the data distribution of sleep stages in different groups. Traditional automatic sleep stage segmentation methods neglect data adaptation to the data distribution of different groups. The data adaptation fine-tuning method proposed in this invention realizes the mining and adaptation of relevant patterns of sleep stage segmentation in different groups, and analyzes the sleep quality of the subjects based on this, realizing sleep quality reporting and disease early warning.

[0143] (4) The present invention provides an automatic sleep staging method that integrates a physiological signal countermeasure enhancement module, a hypergraph fusion module, and a data adaptation fine-tuning module. Physiological signal countermeasure enhancement yields physiological signal data with balanced information content. The enhanced data is then processed and classified through data fusion and adaptation fine-tuning to achieve automatic sleep state detection.

[0144] This embodiment also provides an automatic sleep staging device, including:

[0145] At least one processor;

[0146] At least one memory for storing at least one program;

[0147] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 5 The method shown.

[0148] This embodiment of the automatic sleep staging device can execute an automatic sleep staging method provided in the method embodiment of the present invention, and can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0149] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform... Figure 5 The method shown.

[0150] This embodiment also provides a storage medium storing instructions or programs that can execute the automatic sleep staging method provided in the method embodiment of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the method has the corresponding functions and beneficial effects.

[0151] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0152] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0153] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0154] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0155] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0156] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0157] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0158] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0159] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. An automatic sleep staging method characterized by, The method comprises the following steps: obtaining a polysomnogram of a subject; extracting physiological signal features from the polysomnogram, performing adversarial enhancement processing on the physiological signal features, and obtaining enhanced physiological signal features; learning high-order correlation relationships between multi-modal physiological signals based on the enhanced physiological signal features, performing multi-modal fusion according to the high-order correlation relationships, and obtaining sleep stage classification results; performing cross-modal physiological signal data conversion based on a generative adversarial network according to the obtained physiological signal features, and finally performing feature mixing of each physiological signal based on a channel feature mixing technology to obtain enhanced physiological signal features; the generative adversarial network comprises a discriminator D and an encoder and a decoder corresponding to each modality; the feature distribution of the encoded modality m1 is defined as: wherein is the encoder equation, θ m1 are parameters of the equation; p(x m1 ) is the distribution of features of the modality m1 before encoding; x m1 is the initial feature of the modality m1, i.e. the physiological signal feature of the modality m1 without adversarial augmentation; is the feature of the encoded modality m1; the discriminator D is used to determine whether the modality features generated by each encoder are corresponding modalities, and the discrimination loss function is defined as: wherein the error adversarial loss function is defined as: the correct adversarial loss function is defined as: wherein, is a feature of the encoded modality m2, is a feature of the encoded modality m3, w is a trainable parameter, and D is a discriminator. the expression of the decoder is: wherein is the decoder equation, θ dm1 is a parameter of the equation; is the distribution of the features of the modality m1 after decoding; is the features of the modality m1 after decoding by the decoder; for the decoder, the reconstruction loss is defined as: In the formula, is the multi-modal physiological signal feature decoded by the decoder, x f is the physiological signal feature without encoding and decoding, x is the L2 norm. the features of each modality after encoding are mixed in the channel to enhance the features of the modality m1 after enhancement, and the features of the modality m1 after enhancement are defined as: where a and b are learnable parameters, M(x m1 ) is the physiological signal feature of the augmented modality m1. after the channel mixing enhancement, the enhanced physiological signal features are obtained.

2. The automatic sleep staging method of claim 1, wherein, The method comprises the following steps: acquiring physiological signals of multiple modalities of a subject, the physiological signals comprising electroencephalogram signals, eye movement signals, electromyogram signals, and electrocardiogram signals; amplifying and filtering the acquired physiological signals of multiple modalities to obtain a polysomnogram.

3. The automatic sleep staging method of claim 1, wherein, The method comprises the following steps: after the polysomnogram is time-slice segmented at a preset period, features are extracted to obtain physiological signal features; wherein the extracted features comprise power spectral density features, differential entropy features, and frequency domain statistical features.

4. The automatic sleep staging method of claim 1, wherein, The method comprises the following steps: a hypergraph fusion module is constructed to perform multi-modal fusion on the enhanced physiological signal features and output sleep stage classification results; wherein the hypergraph fusion module comprises a hypergraph convolution layer, a graph feature reading layer, and a sleep stage classification layer; the hypergraph fusion module works as follows: the enhanced physiological signal features of each time slice are input into the hypergraph convolution layer as the features of hypergraph nodes; the hypergraph convolution layer learns the hyperedges between hypergraph nodes and the high-order correlation relationships between physiological signals through model training and self-adaptive learning, and reflects the synergistic effect of various physiological organs in the sleep process; and the hypergraph node features are updated based on an information transmission mechanism; the updated hypergraph node features are input into the graph feature reading layer to read the hypergraph features, and the hypergraph features are classified by the sleep stage classification layer to obtain corresponding sleep stage classification results.

5. The automatic sleep staging method of claim 4, wherein, The automatic sleep staging method further comprises the step of fine-tuning the hypergraph fusion module. For the new subject group of multi-modal physiological signals, part of the data is sampled and manually labeled, and the classification result calculated by the hypergraph fusion module is compared to calculate the classification accuracy; If the classification accuracy is lower than the preset threshold, the model of the hypergraph fusion module is fine-tuned, and the classification result is obtained; The sleep state data of the subject is stored in the database.

6. The automatic sleep staging method of claim 1, wherein, The automatic sleep staging method further comprises the following steps: According to the sleep stage classification result and the preset database, the sleep quality of the subject is analyzed.

7. An automatic sleep staging system for use in the method of any one of claims 1-6, wherein, Comprise: A polysomnogram acquisition module for acquiring polysomnogram of the subject during sleep; A physiological signal adversarial enhancement module for extracting physiological signal features from the polysomnogram, performing adversarial enhancement processing on the physiological signal features, and obtaining enhanced physiological signal features; A hypergraph fusion module for learning high-order correlation between multi-modal physiological signals based on the enhanced physiological signal features, performing multi-modal fusion according to the high-order correlation, and obtaining sleep stage classification results.

8. An automatic sleep staging apparatus characterized by, Comprise: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-6.

9. A computer readable storage medium having stored therein a program which is executable by a processor, characterized in that, The program executable by the processor is used to execute the method of any one of claims 1-6 when executed by the processor.

Images