A sleep state detection method and system based on deep reinforcement learning

Abstract

The application discloses a sleep state detection method and system based on deep reinforcement learning, comprising the following steps: collecting multi-source sleep original data, preprocessing, determining the current sleep time slice and the subsequent verification time window, and simultaneously acquiring the user's historical sleep record; generating a candidate sleep state set; generating the counterfactual sleep continuation segment corresponding to each candidate sleep state; performing consistency verification with the counterfactual sleep continuation segment, generating state continuation verification evidence; mapping the sleep time sequence feature segment to the personal sleep inertia trajectory field, generating the current sleep trajectory point and the personal rhythm constraint evidence; generating signal credible features and performing deep reinforcement learning state correction processing, generating detection correction actions; performing state confirmation and sequence correction on the candidate sleep state set, and generating a sleep state detection result. The application is based on deep reinforcement learning and counterfactual verification, realizes dynamic correction of the sleep state, and has strong continuity and high accuracy.

Description

Technical Field

[0001] This invention relates to the field of sleep state detection technology, and in particular to a sleep state detection method and system based on deep reinforcement learning. Background Technology

[0002] Current sleep state detection typically involves collecting physiological signals such as electroencephalogram (EEG), heart rate, respiration, blood oxygenation, body movement, and snoring. The collected data is then denoised, aligned, windowed, and feature-analyzed. A classification model is then used to identify states of wakefulness, light sleep, deep sleep, REM sleep, or abnormal arousal, thereby obtaining sleep staging results.

[0003] However, existing methods mostly focus on the static identification of the current time slice, lacking verification of the subsequent evolution of candidate sleep states. They are difficult to dynamically correct by combining real subsequent sleep segments, user's historical sleep rhythms, and signal reliability. This can easily lead to short-term state jumps, confusion between light sleep and REM sleep, and misjudgment of abnormal awakenings, affecting the continuity and accuracy of sleep state detection results.

[0004] Therefore, how to provide a sleep state detection method and system based on deep reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a sleep state detection method and system based on deep reinforcement learning. This invention achieves dynamic correction of sleep state based on deep reinforcement learning and counterfactual verification, with strong continuity and high accuracy.

[0006] A sleep state detection method based on deep reinforcement learning according to an embodiment of the present invention includes the following steps: Collect multi-source raw sleep data during the user's sleep process and obtain the user's historical sleep records. Preprocess the multi-source raw sleep data to generate sleep time-series feature segments, and determine the current sleep time slice and subsequent verification time window from the sleep time-series feature segments. Based on the sleep time sequence feature segments corresponding to the current sleep time slice, a set of candidate sleep states is generated; Based on the user's historical sleep records, the set of candidate sleep states, and the sleep time sequence feature segments corresponding to the current sleep time slice, a counterfactual sleep continuation segment corresponding to each candidate sleep state is generated. Based on the subsequent verification time window, a true subsequent sleep segment is generated. The consistency of the true subsequent sleep segment with the counterfactual sleep continuation segment corresponding to each candidate sleep state is verified to generate state continuation verification evidence. A personal sleep inertial trajectory field is generated based on the user's historical sleep records. The sleep time sequence feature segment corresponding to the current sleep time slice is mapped to the personal sleep inertial trajectory field to generate the current sleep trajectory point and personal rhythm constraint evidence. Based on multi-source sleep raw data and sleep time-series feature fragments, reliable signal features are generated, and deep reinforcement learning state correction processing is performed to generate detection correction actions; The candidate sleep state set is confirmed and the sequence is corrected according to the detection and correction actions, and the sleep state detection result corresponding to the current sleep time slice is generated.

[0007] Optionally, the multi-source sleep raw data includes electroencephalogram (EEG) signals, heart rate signals, respiratory signals, blood oxygenation signals, body movement signals, and snoring signals; The preprocessing includes noise reduction, time alignment, sampling unification, and time window segmentation.

[0008] Optionally, the generation of the candidate sleep state set includes: The sleep time sequence feature segments corresponding to the current sleep time slice are processed by time domain change processing, frequency domain change processing, and rhythm continuity processing to generate the current sleep state representation features; Based on the current sleep state representation features, response features for the waking state, the falling asleep state, the light sleep state, the deep sleep state, the REM sleep state, and the abnormal arousal state are generated respectively. Based on the response features, state confidence is organized to generate candidate state confidence results, and a set of candidate sleep states corresponding to the current sleep time slice is generated based on the candidate state confidence results.

[0009] Optionally, the generation of the counterfactual sleep continuation segment includes: Using each candidate sleep state in the candidate sleep state set as an index, determine the historical sleep state segment that is consistent with each candidate sleep state from the user's historical sleep record, and extract the historical sleep continuation segment that is consistent with the length of the current sleep time slice after the historical sleep state segment to generate the historical sleep continuation sample set corresponding to each candidate sleep state. The sleep time sequence feature segment corresponding to the current sleep time slice is identified by initial amplitude recognition, change direction recognition, and channel continuity recognition, and the continuity constraint feature of the current sleep segment is generated. Based on the current sleep segment continuation constraint characteristics, the historical sleep continuation sample set corresponding to each candidate sleep state is subjected to initial amplitude connection, change direction matching and channel continuity screening to generate candidate state continuation benchmark segments corresponding to each candidate sleep state. Based on the current sleep segment continuation constraint characteristics, the continuation trend correction is performed on the candidate state continuation benchmark segment corresponding to each candidate sleep state to generate the counterfactual sleep continuation segment corresponding to each candidate sleep state.

[0010] Optionally, the generation of the state continuation verification evidence includes: Based on the subsequent verification time window, continuous segments are extracted from the sleep time sequence feature segments, while maintaining the time order and feature dimension order in the sleep time sequence feature segments, to generate real subsequent sleep segments; Using real subsequent sleep segments as a reference, the counterfactual sleep continuation segments corresponding to each candidate sleep state are aligned in terms of time length and feature dimension to generate aligned counterfactual sleep continuation segments corresponding to each candidate sleep state. The actual subsequent sleep segments and the aligned counterfactual sleep continuation segments corresponding to each candidate sleep state are processed for same-position difference to generate a point-by-point continuation difference sequence corresponding to each candidate sleep state. The point-by-point continuity difference sequence corresponding to each candidate sleep state is continuously aggregated in time and feature dimension to generate the state continuity difference feature corresponding to each candidate sleep state. Based on the state continuity difference features corresponding to each candidate sleep state, the state continuity labeling results corresponding to each candidate sleep state are generated. Based on the state continuation labeling results corresponding to each candidate sleep state, state continuation verification evidence is generated.

[0011] Optionally, the generation of the personal rhythm constraint evidence includes: The system divides users’ historical sleep records into time slices and organizes sleep state tags to generate a set of continuous segments of historical sleep states. Based on a set of continuous segments of historical sleep states, trajectory anchor points are constructed for the start time position, end time position, duration, connection direction of adjacent segments, and state recovery interval of continuous segments of historical sleep states, generating a set of personal rhythm trajectory anchor points; Based on the temporal sequence and the connection direction of adjacent segments in the personal rhythm trajectory anchor point set, the personal rhythm trajectory anchor point set is connected to generate a personal sleep inertial trajectory field. The current sleep time slice is encoded with state tendency, rhythm stage and change direction to generate the current sleep segment trajectory features; Map the current sleep segment trajectory features to the individual sleep inertial trajectory field to generate the current sleep trajectory points; Based on the current sleep trajectory point and the individual's sleep inertial trajectory field, trajectory position deviation processing, adjacent segment connection direction deviation processing, and state recovery interval deviation processing are performed to generate personal rhythm constraint evidence.

[0012] Optionally, the generation of the detection correction action includes: Channel missing identification, noise fluctuation identification, body motion interference identification, and feature continuity identification are performed on multi-source sleep raw data and sleep time sequence feature segments to generate reliable signal features; Based on state continuity verification evidence, personal rhythm constraint evidence, current sleep trajectory point, personal sleep inertial trajectory field and signal reliability characteristics, time sequence correspondence and evidence relationship are organized to generate state correction observation features; Generate a candidate set of detection correction actions based on state correction observation features; Based on the continuity consistency evidence, continuity conflict evidence, and jump anomaly evidence in the state continuity verification evidence, combined with personal rhythm constraint evidence and signal credibility characteristics, state correction reward characteristics are generated. Based on the state correction observation features, the detection correction action candidate set, and the state correction reward features, deep reinforcement learning state correction processing is performed. The action value of each detection correction action in the detection correction action candidate set is iterated to generate an action value sequence. Actions are selected from the candidate set of detection and correction actions based on the action value sequence, and detection and correction actions are generated.

[0013] Optionally, the generation of the sleep state detection results includes: The target sleep state corresponding to the current sleep time slice is determined from the candidate sleep state set based on the detection and correction action. The target sleep state is processed according to the detection and correction actions, including state confirmation, state replacement, state extension, state rollback, boundary adjustment, or abnormal transfer marking, to generate the current sleep state correction result. Based on the current sleep state correction results and the detection correction actions, the start and end positions of the state, the continuity of the state, and the state correction type are sorted out to generate the current sleep state sequence correction results. The sleep state detection result corresponding to the current sleep time slice is generated based on the correction result of the current sleep state sequence.

[0014] A sleep state detection system based on deep reinforcement learning according to an embodiment of the present invention includes: The data acquisition and processing module is used to collect multi-source raw sleep data and user historical sleep records, and generate sleep time sequence feature segments, current sleep time slices and subsequent verification time windows; The candidate state generation module is used to generate a set of candidate sleep states based on the sleep time sequence feature segments corresponding to the current sleep time slice. The counterfactual continuation generation module is used to generate counterfactual sleep continuation segments based on the user's historical sleep records, the candidate sleep state set, and the sleep time sequence feature segments corresponding to the current sleep time slice. The state continuation verification module is used to generate real subsequent sleep segments based on the subsequent verification time window, and to perform consistency verification between the real subsequent sleep segments and the counterfactual sleep continuation segments to generate state continuation verification evidence. The personal rhythm constraint module is used to generate a personal sleep inertial trajectory field, the current sleep trajectory point, and personal rhythm constraint evidence based on the user's historical sleep records and the sleep time sequence feature segments corresponding to the current sleep time slice; The enhanced state correction module is used to generate detection and correction actions based on state continuation verification evidence, personal rhythm constraint evidence, current sleep trajectory points, personal sleep inertial trajectory field, and signal credibility features. The detection result generation module is used to generate the sleep state detection result corresponding to the current sleep time slice based on the detection correction action.

[0015] The beneficial effects of this invention are: This invention collects EEG signals, heart rate signals, respiratory signals, blood oxygen signals, body movement signals, and snoring signals, and preprocesses the multi-source raw sleep data to form sleep time sequence feature segments. This makes the identification of the current sleep time slice no longer solely dependent on a single physiological signal or a single moment feature, but rather based on a comprehensive characterization of the temporal correlation, direction of change, and continuity between multiple signals. This enhances the adaptability of sleep state identification to signal fluctuations, short-term interference, and channel anomalies, and improves the stability of sleep state detection results.

[0016] This invention further generates counterfactual sleep continuation segments corresponding to each candidate sleep state based on a set of candidate sleep states, and performs consistency verification between the counterfactual sleep continuation segments and the actual subsequent sleep segments to generate state continuation verification evidence. This allows the system to extend beyond simply determining whether the current state resembles a certain sleep state to whether that state can reasonably continue in the subsequent time. This approach effectively suppresses short-term state transitions caused solely by the characteristics of the current segment, reducing the probability of confusion between light sleep, REM sleep, and aberrant arousal states.

[0017] Meanwhile, this invention generates a personal sleep inertial trajectory field based on the user's historical sleep records and maps the current sleep time slice to form the current sleep trajectory point, thereby generating personal rhythm constraint evidence. This allows the sleep state detection process to incorporate the user's own sleep rhythm, state transition habits, and recovery characteristics, avoiding the problem of insufficient adaptation due to individual differences caused by using a uniform recognition scale. Furthermore, by combining signal reliability features and deep reinforcement learning state correction processing to generate detection correction actions, the candidate sleep state set is confirmed and the sequence is corrected. This improves the continuity of the sleep state sequence, individual adaptability, and accuracy of abnormal arousal recognition, ultimately obtaining more reliable sleep state detection results. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a sleep state detection method based on deep reinforcement learning proposed in this invention; Figure 2 This is a schematic diagram illustrating the process of verifying the state continuation between counterfactual sleep continuation segments and real subsequent sleep segments in a sleep state detection method based on deep reinforcement learning proposed in this invention. Figure 3 This is a schematic diagram illustrating the process of generating detection correction actions based on deep reinforcement learning state correction processing in the sleep state detection method proposed in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] refer to Figures 1-3 A sleep state detection method based on deep reinforcement learning includes the following steps: Collect multi-source raw sleep data during the user's sleep process and obtain the user's historical sleep records. The multi-source raw sleep data includes EEG signals, heart rate signals, respiratory signals, blood oxygen signals, body movement signals and snoring signals. Preprocess the multi-source raw sleep data to generate sleep time sequence feature segments, and determine the current sleep time slice and subsequent verification time window from the sleep time sequence feature segments. Based on the sleep time sequence feature segments corresponding to the current sleep time slice, a candidate sleep state set is generated. The candidate sleep state set includes the awake state, the sleep state, the light sleep state, the deep sleep state, the REM sleep state, and the abnormal arousal state. Based on the user's historical sleep records, the set of candidate sleep states, and the sleep time sequence feature segments corresponding to the current sleep time slice, a counterfactual sleep continuation segment corresponding to each candidate sleep state is generated. Based on the subsequent verification time window, real subsequent sleep segments are generated from sleep time sequence feature segments. The consistency of the real subsequent sleep segments with the counterfactual sleep continuation segments corresponding to each candidate sleep state is verified to generate state continuation verification evidence. A personal sleep inertial trajectory field is generated based on the user's historical sleep records. The sleep time sequence feature segment corresponding to the current sleep time slice is mapped to the personal sleep inertial trajectory field to generate the current sleep trajectory point. Personal rhythm constraint evidence is generated based on the current sleep trajectory point and the personal sleep inertial trajectory field. Based on multi-source sleep raw data and sleep time-series feature fragments, reliable signal features are generated. Based on state continuity verification evidence, personal rhythm constraint evidence, current sleep trajectory point, personal sleep inertial trajectory field and reliable signal features, deep reinforcement learning state correction processing is performed to generate detection correction actions. The candidate sleep state set is confirmed and the sequence is corrected according to the detection and correction actions, and the sleep state detection result corresponding to the current sleep time slice is generated.

[0021] In this embodiment, the multi-source sleep raw data includes electroencephalogram (EEG) signals, heart rate signals, respiratory signals, blood oxygenation signals, body movement signals, and snoring signals; The preprocessing includes noise reduction, time alignment, sampling unification, and time window segmentation.

[0022] In this embodiment, the generation of the candidate sleep state set includes: The sleep time sequence feature segments corresponding to the current sleep time slice are processed by time domain change processing, frequency domain change processing, and rhythm continuity processing to generate the current sleep state representation features; Based on the current sleep state representation features, response features for the waking state, the falling asleep state, the light sleep state, the deep sleep state, the REM sleep state, and the abnormal arousal state are generated respectively. Based on the response features, state confidence is organized to generate candidate state confidence results, and based on the candidate state confidence results, a candidate sleep state set corresponding to the current sleep time slice is generated. The candidate sleep state set includes awake state, sleep state, light sleep state, deep sleep state, REM sleep state and abnormal awakening state. The generation of the candidate sleep state set specifically includes: taking the wakefulness response features, sleep onset response features, light sleep response features, deep sleep response features, REM sleep response features, and abnormal arousal response features as the state confidence processing objects; performing same-dimensional normalization and same-state aggregation on each state response feature to generate the state confidence value corresponding to each candidate sleep state; arranging the candidate sleep states according to each state confidence value to generate the candidate state confidence results, and generating the candidate sleep state set from the candidate state confidence results.

[0023] In this embodiment, the generation of counterfactual sleep continuation segments includes: Using each candidate sleep state in the candidate sleep state set as an index, determine the historical sleep state segment that is consistent with each candidate sleep state from the user's historical sleep record, and extract the historical sleep continuation segment that is consistent with the length of the current sleep time slice after the historical sleep state segment to generate the historical sleep continuation sample set corresponding to each candidate sleep state. The generation of the historical sleep continuation sample set specifically includes: reading the status label, start and end time, and duration of historical sleep state segments from the user's historical sleep records; mapping the status label to each candidate sleep state in the candidate sleep state set; continuously extracting historical sleep continuation segments with the same length as the current sleep time slice after the end position of the corresponding historical sleep state segment; and aggregating the historical sleep continuation segments corresponding to the same candidate sleep state to generate a historical sleep continuation sample set corresponding to each candidate sleep state. The sleep time sequence feature segment corresponding to the current sleep time slice is identified by initial amplitude recognition, change direction recognition, and channel continuity recognition, and the continuity constraint feature of the current sleep segment is generated. The generation of the current sleep segment continuation constraint feature specifically includes: organizing the first and last feature values ​​of the sleep time sequence feature segment corresponding to the current sleep time slice to generate the initial amplitude feature; sequentially encoding the feature change direction between adjacent sampling positions to generate the change direction feature; organizing the continuity of each feature dimension within the current sleep time slice to generate the channel continuity feature; and the current sleep segment continuation constraint feature is composed of the initial amplitude feature, the change direction feature, and the channel continuity feature. Based on the current sleep segment continuation constraint characteristics, the historical sleep continuation sample set corresponding to each candidate sleep state is subjected to initial amplitude connection, change direction matching and channel continuity screening to generate candidate state continuation benchmark segments corresponding to each candidate sleep state. The generation of the candidate state continuation benchmark segment specifically includes: connecting the initial amplitude feature in the current sleep segment continuation constraint feature with the first-end feature of each historical sleep continuation segment in the historical sleep continuation sample set to generate a historical sleep continuation segment after amplitude connection; adjusting the direction of change feature with the historical sleep continuation segment after amplitude connection to generate a historical sleep continuation segment after direction matching; and adjusting the channel continuity feature with the historical sleep continuation segment after direction matching to generate a candidate state continuation benchmark segment corresponding to each candidate sleep state. Based on the current sleep segment continuation constraint characteristics, the continuation trend correction is performed on the candidate state continuation benchmark segment corresponding to each candidate sleep state to generate the counterfactual sleep continuation segment corresponding to each candidate sleep state. The generation of the counterfactual sleep continuation segment specifically includes: using the initial amplitude feature in the current sleep segment continuation constraint feature as the connection benchmark, performing amplitude translation on the first feature of the candidate state continuation benchmark segment; performing trend continuity adjustment on the amplitude-translated candidate state continuation benchmark segment based on the change direction feature; and supplementing the feature dimensions of the trend-continuously adjusted candidate state continuation benchmark segment based on the channel continuity feature to generate the counterfactual sleep continuation segment corresponding to each candidate sleep state.

[0024] In this embodiment, the generation of state continuation verification evidence includes: Based on the subsequent verification time window, continuous segments are extracted from the sleep time sequence feature segments, while maintaining the time order and feature dimension order in the sleep time sequence feature segments, to generate real subsequent sleep segments; Using real subsequent sleep segments as a reference, the counterfactual sleep continuation segments corresponding to each candidate sleep state are aligned in terms of time length and feature dimension to generate aligned counterfactual sleep continuation segments corresponding to each candidate sleep state. The actual subsequent sleep segments and the aligned counterfactual sleep continuation segments corresponding to each candidate sleep state are processed for same-position difference to generate a point-by-point continuation difference sequence corresponding to each candidate sleep state. The specific steps of performing positional difference processing include: pairing the real subsequent sleep segments with the aligned counterfactual sleep continuation segments corresponding to each candidate sleep state according to the same time position and the same feature dimension position; performing difference processing on the real feature value and counterfactual feature value of each paired position to generate the continuation difference value of the corresponding position; arranging the continuation difference values ​​according to the time order and feature dimension order to generate the point-by-point continuation difference sequence corresponding to each candidate sleep state; The point-by-point continuity difference sequence corresponding to each candidate sleep state is continuously aggregated in time and feature dimension to generate the state continuity difference feature corresponding to each candidate sleep state. The generation of the state continuity difference feature specifically includes: continuously aggregating the point-by-point continuity difference sequence corresponding to each candidate sleep state along the time order to generate a time continuous difference result; aggregating the continuity difference values ​​within the same feature dimension to generate a feature dimension difference result; and merging the time continuous difference result and the feature dimension difference result to generate the state continuity difference feature corresponding to each candidate sleep state. Based on the state continuation difference characteristics of each candidate sleep state, a state continuation labeling result is generated for each candidate sleep state. The state continuation labeling result includes a continuation consistency label, a continuation conflict label, and a jump anomaly label. The generation of the state continuation labeling results specifically includes: based on the temporal continuity difference results and feature dimension difference results in the state continuation difference features, sorting out the continuous proximity relationship, dimension deviation relationship, and short-term abrupt change relationship between the real subsequent sleep segments and the aligned counterfactual sleep continuation segments; converting the continuous proximity relationship into a continuation consistency label, the dimension deviation relationship into a continuation conflict label, and the short-term abrupt change relationship into a jump anomaly label; the state continuation labeling results are composed of the continuation consistency label, the continuation conflict label, and the jump anomaly label. Based on the state continuation marking results corresponding to each candidate sleep state, state continuation verification evidence is generated. The state continuation verification evidence includes continuation consistency evidence, continuation conflict evidence, and jump anomaly evidence corresponding to each candidate sleep state.

[0025] In this embodiment, the generation of personal rhythm constraint evidence includes: The system divides users’ historical sleep records into time slices and organizes sleep state tags to generate a set of continuous segments of historical sleep states. Based on a set of continuous segments of historical sleep states, trajectory anchor points are constructed for the start time position, end time position, duration, connection direction of adjacent segments, and state recovery interval of continuous segments of historical sleep states, generating a set of personal rhythm trajectory anchor points; The generation of the personal rhythm trajectory anchor point set specifically includes: reading the start time position, end time position, duration, adjacent segment connection direction, and state recovery interval of each historical sleep state continuous segment from the historical sleep state continuous segment set; binding the above information with the corresponding sleep state tag to generate personal rhythm trajectory anchor points; arranging the personal rhythm trajectory anchor points according to the temporal order of the historical sleep state continuous segments to generate a personal rhythm trajectory anchor point set. Based on the temporal sequence and the connection direction of adjacent segments in the personal rhythm trajectory anchor point set, the personal rhythm trajectory anchor point set is connected to generate a personal sleep inertial trajectory field. The generation of the personal sleep inertial trajectory field specifically includes: connecting adjacent personal rhythmic trajectory anchor points according to the temporal order in the personal rhythmic trajectory anchor point set to generate a rhythmic trajectory connection result; arranging the rhythmic trajectory connection result according to the connection direction of adjacent segments to generate a trajectory structure with directional continuity; and forming a personal sleep inertial trajectory field by the personal rhythmic trajectory anchor point set and the trajectory structure with directional continuity. The current sleep time slice is encoded with state tendency, rhythm stage and change direction to generate the current sleep segment trajectory features; The generation of the current sleep segment trajectory features specifically includes: performing state tendency encoding on the sleep time sequence feature segment corresponding to the current sleep time slice to generate state tendency features; performing rhythm phase encoding on the time position of the current sleep time slice in the sleep time sequence feature segment to generate rhythm phase features; encoding the feature change direction of adjacent sampling positions within the current sleep time slice to generate change direction features; and the current sleep segment trajectory features are composed of state tendency features, rhythm phase features, and change direction features. Map the current sleep segment trajectory features to the individual sleep inertial trajectory field to generate the current sleep trajectory points; The generation of the current sleep trajectory point specifically includes: mapping the state tendency features in the current sleep segment trajectory features to the state position in the personal sleep inertial trajectory field; mapping the rhythm stage features to the time position in the personal sleep inertial trajectory field; mapping the change direction features to the connection direction position in the personal sleep inertial trajectory field; and merging the state position, time position, and connection direction position to generate the current sleep trajectory point. Based on the current sleep trajectory point and the personal sleep inertial trajectory field, trajectory position deviation processing, adjacent segment connection direction deviation processing, and state recovery interval deviation processing are performed to generate personal rhythm constraint evidence; The generation of the personal rhythm constraint evidence specifically includes: sorting out the trajectory position deviation between the current sleep trajectory point and the personal sleep inertial trajectory field to generate a trajectory position deviation result; sorting out the connection direction deviation between adjacent segments between the current sleep trajectory point and the personal sleep inertial trajectory field to generate a connection direction deviation result; sorting out the state recovery interval deviation between the current sleep trajectory point and the personal sleep inertial trajectory field to generate a state recovery interval deviation result; and the trajectory position deviation result, connection direction deviation result, and state recovery interval deviation result constitute the personal rhythm constraint evidence.

[0026] In this embodiment, the generation of the detection correction action includes: Channel missing identification, noise fluctuation identification, body motion interference identification, and feature continuity identification are performed on multi-source sleep raw data and sleep time sequence feature segments to generate reliable signal features; The generation of the reliable signal features specifically includes: sorting out the sampling integrity of each signal channel in the multi-source sleep raw data to generate channel missing features; sorting out short-term fluctuations and adjacent sampling abrupt changes in sleep time sequence feature segments to generate noise fluctuation features; sorting out the synchronous changes of body movement signals and other signal channels at the same time position to generate body movement interference features; sorting out the feature preservation status of sleep time sequence feature segments at continuous time positions to generate feature continuity features; and the reliable signal features are composed of channel missing features, noise fluctuation features, body movement interference features, and feature continuity features. Based on state continuity verification evidence, personal rhythm constraint evidence, current sleep trajectory point, personal sleep inertial trajectory field and signal reliability characteristics, time sequence correspondence and evidence relationship are organized to generate state correction observation features; The process of generating the state-corrected observation features needs to be supplemented. The supplement is as follows: convert the state continuation verification evidence, personal rhythm constraint evidence, current sleep trajectory point, personal sleep inertial trajectory field, and signal credibility features into evidence vectors of the same dimension; splice the evidence vectors according to the time order corresponding to the current sleep time slice to generate the corrected observation spliced ​​vector; perform dimension normalization and evidence relationship organization on the corrected observation spliced ​​vector to generate the state-corrected observation features. A candidate set of detection and correction actions is generated based on the state correction observation features. The candidate set of detection and correction actions includes state confirmation actions, state replacement actions, state extension actions, state rollback actions, boundary adjustment actions, and abnormal transfer marking actions. Based on the continuity consistency evidence, continuity conflict evidence, and jump anomaly evidence in the state continuity verification evidence, combined with personal rhythm constraint evidence and signal credibility characteristics, state correction reward characteristics are generated. The generation of the state correction reward feature specifically includes: converting the continuity consistency evidence, continuity conflict evidence, and jump anomaly evidence in the state continuity verification evidence into continuity verification reward components respectively; converting the personal rhythm constraint evidence into personal rhythm reward components; converting the signal credibility feature into signal credibility reward components; and fusing the continuity verification reward components, personal rhythm reward components, and signal credibility reward components to generate the state correction reward feature. Based on the state correction observation features, the detection correction action candidate set, and the state correction reward features, deep reinforcement learning state correction processing is performed. The action value of each detection correction action in the detection correction action candidate set is iterated to generate an action value sequence. The generation of the action value sequence is as follows: the state correction observation features are used as the current state representation, and the state confirmation action, state replacement action, state extension action, state rollback action, boundary adjustment action, and abnormal transition marking action in the detection correction action candidate set are used as optional actions; the state correction reward features are used as reward feedback, and action values ​​are generated for each optional action; the value is iterated according to the correspondence between the state correction observation features, optional actions, and reward feedback to generate the action value sequence. Based on the action value sequence, actions are selected from the candidate set of detection and correction actions to generate detection and correction actions; The generation of the detection correction action specifically includes: normalizing and organizing the action values ​​corresponding to each selectable action in the action value sequence to generate action selection weights; binding the action selection weights to each detection correction action in the candidate set of detection correction actions to generate action selection results; and generating the detection correction action corresponding to the current sleep time slice based on the action selection results.

[0027] In this embodiment, the generation of sleep state detection results includes: The target sleep state corresponding to the current sleep time slice is determined from the candidate sleep state set based on the detection and correction action. The target sleep state is processed according to the detection and correction actions, including state confirmation, state replacement, state extension, state rollback, boundary adjustment, or abnormal transfer marking, to generate the current sleep state correction result. The generation of the current sleep state correction result specifically includes: binding the detection correction action with the target sleep state to generate a correction processing object corresponding to the target sleep state; performing state confirmation processing, state replacement processing, state extension processing, state rollback processing, boundary adjustment processing, or abnormal transfer marking processing on the correction processing object according to the action type of the detection correction action; merging the processed target sleep state and the corresponding action type to generate the current sleep state correction result. Based on the current sleep state correction results and the detection correction actions, the start and end positions of the state, the continuity of the state, and the state correction type are sorted out to generate the current sleep state sequence correction results. The generation of the current sleep state sequence correction result specifically includes: based on the current sleep time slice and the current sleep state correction result, organizing the start and end positions of the target sleep state to generate the state start and end positions; organizing the state correction type corresponding to the detection correction action based on the detection correction action; organizing the continuous maintenance relationship of the target sleep state within the current sleep time slice to generate the state continuity relationship; and writing the state start and end positions, the state continuity relationship, and the state correction type corresponding to the detection correction action into the current sleep state correction result to generate the current sleep state sequence correction result. Based on the current sleep state sequence correction results, a sleep state detection result corresponding to the current sleep time slice is generated. The sleep state detection result includes the target sleep state corresponding to the current sleep time slice, the start and end positions of the state, and the state correction type corresponding to the detection correction action.

[0028] A sleep state detection system based on deep reinforcement learning includes: The data acquisition and processing module is used to collect multi-source raw sleep data and user historical sleep records, and generate sleep time sequence feature segments, current sleep time slices and subsequent verification time windows; The candidate state generation module is used to generate a set of candidate sleep states based on the sleep time sequence feature segments corresponding to the current sleep time slice. The counterfactual continuation generation module is used to generate counterfactual sleep continuation segments based on the user's historical sleep records, the candidate sleep state set, and the sleep time sequence feature segments corresponding to the current sleep time slice. The state continuation verification module is used to generate real subsequent sleep segments based on the subsequent verification time window, and to perform consistency verification between the real subsequent sleep segments and the counterfactual sleep continuation segments to generate state continuation verification evidence. The personal rhythm constraint module is used to generate a personal sleep inertial trajectory field, the current sleep trajectory point, and personal rhythm constraint evidence based on the user's historical sleep records and the sleep time sequence feature segments corresponding to the current sleep time slice; The enhanced state correction module is used to generate detection and correction actions based on state continuation verification evidence, personal rhythm constraint evidence, current sleep trajectory points, personal sleep inertial trajectory field, and signal credibility features. The detection result generation module is used to generate the sleep state detection result corresponding to the current sleep time slice based on the detection correction action.

[0029] Example 1: To verify the feasibility of this invention in practice, it was applied to a health sleep monitoring scenario. This scenario was an independent rest room equipped with sleep signal acquisition capabilities. The room was equipped with an EEG headband, wrist-mounted heart rate and blood oxygenation devices, a chest strap respiratory acquisition device, a mattress-mounted body movement acquisition device, and an environmental snoring acquisition device. The acquisition period was four consecutive weeks, with each day's acquisition time covering the user's complete nighttime sleep. The subjects were users experiencing difficulty falling asleep, short-term nighttime awakenings, or unstable REM sleep recognition, resulting in 504 hours of sleep sample data. The above sample data is only used to illustrate the application and technical effects of this invention in a real sleep monitoring scenario and does not constitute a limitation on the scope of protection of this invention.

[0030] In this scenario, after a user enters the rest room, EEG signals, heart rate signals, respiratory signals, blood oxygen signals, body movement signals, and snoring signals are simultaneously collected. The collected multi-source raw sleep data undergoes denoising, time alignment, sampling unification, and time window segmentation to form continuous sleep time sequence feature segments. A 30-second period is defined as a current sleep time slice, and the three consecutive time slices following the current sleep time slice serve as subsequent verification time windows. This allows the system to simultaneously retain the verification effect of subsequent sleep changes on the current state when identifying the current sleep state. For example, within a certain current sleep time slice, if low-frequency components in the EEG signal increase, the heart rate decreases from 72 beats per minute to 68 beats per minute, the respiratory rate decreases from 16 breaths per minute to 14 breaths per minute, and the body movement signal continues to weaken, the system first generates a set of candidate sleep states corresponding to wakefulness, sleep onset, light sleep, deep sleep, REM sleep, and abnormal arousal, rather than directly fixing the time slice to a single state.

[0031] After generating a set of candidate sleep states, this invention combines the user's historical sleep records to generate a corresponding counterfactual sleep continuation segment for each candidate sleep state. Using the candidate sleep state as an index, the system identifies historical sleep state segments with the same state label from historical sleep records and extracts their subsequent continuation segments, forming a historical sleep continuation sample set. Simultaneously, it organizes the initial amplitude features, change direction features, and channel continuity features from the current sleep time slice to generate current sleep segment continuation constraint features. Then, it performs initial amplitude connection, change direction matching, and channel continuity organization on the historical sleep continuation sample set to generate candidate state continuation benchmark segments corresponding to each candidate sleep state, thereby obtaining the counterfactual sleep continuation segments. Through this processing, the system can identify the possible continuation changes of different candidate sleep states in subsequent time periods, providing a comparison basis for subsequent verification.

[0032] Subsequently, this invention extracts real subsequent sleep segments from sleep time-series feature segments based on a subsequent verification time window, and performs consistency verification between the real subsequent sleep segments and the counterfactual sleep continuation segments corresponding to each candidate sleep state. Taking a sample time slice of a certain night as an example, conventional sleep detection methods identify it as an abnormal awakening state because of the short-term body movement and heart rate increase in this segment. However, in the processing of this invention, the EEG signal within the subsequent verification time window does not show continuous awakening characteristics, the respiratory signal and blood oxygen signal remain stable, and the snoring signal does not show obvious discontinuous changes. The system performs positional difference processing on the real subsequent sleep segments and the counterfactual sleep continuation segments corresponding to the abnormal awakening state, generating a point-by-point continuation difference sequence, and further obtaining state continuation difference features and state continuation labeling results. The results show that the differences are mainly concentrated in the short-term body movement position, and the counterfactual sleep continuation segments corresponding to the light sleep state are closer to the real subsequent sleep segments. Therefore, it generates continuation consistency evidence for the light sleep state and forms continuation conflict evidence and jump abnormal evidence for the abnormal awakening state, thereby avoiding misjudgment of abnormal awakening caused by short-term disturbances within a single time slice.

[0033] To accommodate different users' sleep habits, this invention also generates a personal sleep inertial trajectory field based on the user's historical sleep records. This trajectory field organizes the start and end times, durations, connection directions of adjacent segments, and state recovery intervals of continuous segments of historical sleep states, forming a set of personal rhythm trajectory anchor points. These anchor points are then connected according to chronological order and the connection directions of adjacent segments. Taking a user's data from multiple consecutive nights as an example, this user typically enters a light sleep state 18 to 35 minutes after falling asleep, and the first deep sleep state usually occurs between 42 and 68 minutes after falling asleep. After short periods of body movement at night, most users return to their original sleep stage within 2 minutes. With this invention, the sleep time sequence feature segments corresponding to the current sleep time slice are mapped to the personal sleep inertial trajectory field, generating the current sleep trajectory point. This is combined with trajectory position deviations, adjacent segment connection direction deviations, and state recovery interval deviations to generate personal rhythm constraint evidence, allowing the same heart rate fluctuations or body movement changes to be explained in conjunction with the user's own sleep rhythm.

[0034] After generating state continuation verification evidence and personal rhythm constraint evidence, this invention further combines multi-source raw sleep data and sleep temporal feature fragments to generate reliable signal features. For signal abrupt changes caused by short-term EEG contact instability, short-term loosening of wrist devices, and body movement, the system respectively organizes channel missing features, noise fluctuation features, body movement interference features, and feature continuity features to form reliable signal features. Subsequently, the state continuation verification evidence, personal rhythm constraint evidence, current sleep trajectory point, personal sleep inertial trajectory field, and reliable signal features are temporally correlated and evidence relationships are organized to generate state correction observation features. Then, deep reinforcement learning state correction processing is performed by combining the candidate set of detection correction actions and state correction reward features to generate detection correction actions, and state confirmation and sequence correction are performed on the candidate sleep state set based on the detection correction actions.

[0035] In 504 hours of sleep sample data, the sleep state sequences obtained using the conventional single-segment classification method contained 286 short-term state transition segments, of which 94 were abnormal arousals lasting less than 90 seconds. After applying this invention, the number of short-term state transition segments decreased to 117, and the number of abnormal arousals lasting less than 90 seconds decreased to 38. Regarding segments easily confused between light sleep and REM sleep states (as identified through manual review), the conventional method resulted in 41 instances of confusion in 126 samples; this invention reduced the number of confused segments to 18. Regarding misjudgments of abnormal arousals caused by nighttime body movement, the conventional method resulted in 32 misjudgments in 78 body movement-related samples; this invention reduced the number of misjudgments to 11. Therefore, this invention effectively improves the continuity, individual adaptability, and accuracy of sleep state detection results, and reduces the probability of short-term state transitions, confusion between light sleep and REM sleep, and misjudgments of abnormal arousals.

[0036] Table 1. Comparison of sleep state detection performance under counterfactual continuation check and personal rhythm correction.

[0037] The table above is based on 504 hours of multi-source sleep sample data, including EEG signals, heart rate signals, respiratory signals, blood oxygen signals, body movement signals, and snoring signals. Manually verified sleep state labels are used for comparison. The overall accuracy and macro-average F1 score in the table reflect the comprehensive recognition capabilities of different methods for wakefulness, sleep onset, light sleep, deep sleep, REM sleep, and aberrant arousal. The confusion rate between light sleep and REM sleep reflects the ability to distinguish between two similar sleep stages. The aberrant arousal misjudgment rate reflects the impact of short-term body movement, heart rate fluctuations, and signal perturbations on the detection results. The number of short-term state transition segments and the average state boundary deviation reflect the continuity of the sleep state sequence and the stability of the state transition boundaries.

[0038] As shown in Table 1, the overall accuracy of the single-segment rule-based classification method is 76.8%, with a macro-average F1 score of 0.723. The confusion rate between light sleep and REM sleep reaches 35.7%, and the misjudgment rate of abnormal awakening reaches 43.6%. This indicates that this type of method mainly relies on signal amplitude, frequency band changes, or body movement within the current time slice for judgment, lacking verification of subsequent sleep evolution trends. It easily misinterprets short-term body movement or short-term heart rate increases as abnormal awakening and is also prone to confusion between light sleep and REM sleep states. Although the single-segment convolutional classification method can learn local features from the current sleep time slice, improving the overall accuracy to 82.4%, it still relies primarily on the features of the current segment. The number of short-term state transition segments is still 50.2 segments / 100 hours, indicating that relying solely on the deep feature extraction of the current segment is insufficient to fully address the problem of discontinuous sleep state sequences.

[0039] Recurrent neural network (RNN) and Transformer time-series classification methods can model the temporal relationships between consecutive time slices, thus outperforming single-slice methods. The Transformer method achieves an overall accuracy of 87.6%, a macro-average F1 score of 0.852, and reduces the number of short-term state transition segments to 34.7 segments / 100 hours, indicating that temporal modeling can improve the continuity of sleep state transitions. However, these methods typically classify states based on existing time series, lacking a counterfactual verification process to determine whether candidate sleep states reasonably continue in the future. They also fail to fully integrate historical sleep records to form a personal sleep inertia trajectory field. Therefore, there is still room for improvement in terms of the confusion rate between light sleep and REM sleep, and the misjudgment rate of abnormal awakenings.

[0040] The overall accuracy of the method of this invention reaches 91.8%, which is 4.2 percentage points higher than that of the Transformer time-series classification method. The macro-average F1 score reaches 0.901, which is 0.049 higher than that of the Transformer time-series classification method. This indicates that the present invention not only improves the overall recognition accuracy but also enhances the ability to recognize the balance between different sleep states. The confusion rate between light sleep and REM sleep of the present invention is 14.3%, which is about 30.6% lower than that of the Transformer time-series classification method (20.6%). The false alarm rate is 14.1%, which is about 42.2% lower than that of the Transformer time-series classification method (24.4%). These results show that the present invention, by generating counterfactual sleep continuation segments corresponding to each candidate sleep state and verifying their consistency with the actual subsequent sleep segments, can determine whether the current candidate state has a reasonable continuation relationship in the subsequent time, thereby avoiding erroneous judgments caused by short-term body movements or short-term fluctuations in the current time slice.

[0041] Meanwhile, the number of short-term state transition segments in this invention is reduced to 23.2 segments / 100 hours, a reduction of approximately 33.1% compared to the Transformer temporal classification method; the average state boundary deviation is reduced to 19.6 seconds, a reduction of approximately 29.5% compared to the Transformer temporal classification method. This is mainly because this invention not only generates state continuity verification evidence but also generates a personal sleep inertial trajectory field based on the user's historical sleep records, mapping the current sleep time slice to the current sleep trajectory point and generating personal rhythm constraint evidence. This processing enables sleep state detection results to be constrained by the user's own sleep rhythm, state transition habits, and state recovery intervals, avoiding the problem of insufficient adaptation to individual differences caused by using a uniform model scale. Combined with signal credibility features and deep reinforcement learning state correction processing, this invention can generate more reasonable detection correction actions from state confirmation actions, state replacement actions, state extension actions, state regression actions, boundary adjustment actions, and abnormal transition marking actions, thereby improving the continuity, accuracy, and anti-interference ability of sleep state detection results.

[0042] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A sleep state detection method based on deep reinforcement learning, characterized in that, Includes the following steps: Collect multi-source raw sleep data during the user's sleep process and obtain the user's historical sleep records. Preprocess the multi-source raw sleep data to generate sleep time-series feature segments, and determine the current sleep time slice and subsequent verification time window from the sleep time-series feature segments. Based on the sleep time sequence feature segments corresponding to the current sleep time slice, a set of candidate sleep states is generated; Based on the user's historical sleep records, the set of candidate sleep states, and the sleep time sequence feature segments corresponding to the current sleep time slice, a counterfactual sleep continuation segment corresponding to each candidate sleep state is generated. Based on the subsequent verification time window, a true subsequent sleep segment is generated. The consistency of the true subsequent sleep segment with the counterfactual sleep continuation segment corresponding to each candidate sleep state is verified to generate state continuation verification evidence. A personal sleep inertial trajectory field is generated based on the user's historical sleep records. The sleep time sequence feature segment corresponding to the current sleep time slice is mapped to the personal sleep inertial trajectory field to generate the current sleep trajectory point and personal rhythm constraint evidence. Based on multi-source sleep raw data and sleep time-series feature fragments, reliable signal features are generated, and deep reinforcement learning state correction processing is performed to generate detection correction actions; The candidate sleep state set is confirmed and the sequence is corrected according to the detection and correction actions, and the sleep state detection result corresponding to the current sleep time slice is generated.

2. The sleep state detection method based on deep reinforcement learning according to claim 1, characterized in that, The multi-source sleep raw data includes electroencephalogram (EEG) signals, heart rate signals, respiratory signals, blood oxygenation signals, body movement signals, and snoring signals; The preprocessing includes noise reduction, time alignment, sampling unification, and time window segmentation.

3. The sleep state detection method based on deep reinforcement learning according to claim 1, characterized in that, The generation of the candidate sleep state set includes: The sleep time sequence feature segments corresponding to the current sleep time slice are processed by time domain change processing, frequency domain change processing, and rhythm continuity processing to generate the current sleep state representation features; Based on the current sleep state representation features, response features for the waking state, the falling asleep state, the light sleep state, the deep sleep state, the REM sleep state, and the abnormal arousal state are generated respectively. Based on the response features, state confidence is organized to generate candidate state confidence results, and a set of candidate sleep states corresponding to the current sleep time slice is generated based on the candidate state confidence results.

4. The sleep state detection method based on deep reinforcement learning according to claim 1, characterized in that, The generation of the counterfactual sleep continuation fragment includes: Using each candidate sleep state in the candidate sleep state set as an index, determine the historical sleep state segment that is consistent with each candidate sleep state from the user's historical sleep record, and extract the historical sleep continuation segment that is consistent with the length of the current sleep time slice after the historical sleep state segment to generate the historical sleep continuation sample set corresponding to each candidate sleep state. The sleep time sequence feature segment corresponding to the current sleep time slice is identified by initial amplitude recognition, change direction recognition, and channel continuity recognition, and the continuity constraint feature of the current sleep segment is generated. Based on the current sleep segment continuation constraint characteristics, the historical sleep continuation sample set corresponding to each candidate sleep state is subjected to initial amplitude connection, change direction matching and channel continuity screening to generate candidate state continuation benchmark segments corresponding to each candidate sleep state. Based on the current sleep segment continuation constraint characteristics, the continuation trend correction is performed on the candidate state continuation benchmark segment corresponding to each candidate sleep state to generate the counterfactual sleep continuation segment corresponding to each candidate sleep state.

5. The sleep state detection method based on deep reinforcement learning according to claim 1, characterized in that, The generation of the state continuation verification evidence includes: Based on the subsequent verification time window, continuous segments are extracted from the sleep time sequence feature segments, while maintaining the time order and feature dimension order in the sleep time sequence feature segments, to generate real subsequent sleep segments; Using real subsequent sleep segments as a reference, the counterfactual sleep continuation segments corresponding to each candidate sleep state are aligned in terms of time length and feature dimension to generate aligned counterfactual sleep continuation segments corresponding to each candidate sleep state. The actual subsequent sleep segments and the aligned counterfactual sleep continuation segments corresponding to each candidate sleep state are processed for same-position difference to generate a point-by-point continuation difference sequence corresponding to each candidate sleep state. The point-by-point continuity difference sequence corresponding to each candidate sleep state is continuously aggregated in time and feature dimension to generate the state continuity difference feature corresponding to each candidate sleep state. Based on the state continuity difference features corresponding to each candidate sleep state, the state continuity labeling results corresponding to each candidate sleep state are generated. Based on the state continuation labeling results corresponding to each candidate sleep state, state continuation verification evidence is generated.

6. The sleep state detection method based on deep reinforcement learning according to claim 1, characterized in that, The generation of the personal rhythm constraint evidence includes: The system divides users’ historical sleep records into time slices and organizes sleep state tags to generate a set of continuous segments of historical sleep states. Based on a set of continuous segments of historical sleep states, trajectory anchor points are constructed for the start time position, end time position, duration, connection direction of adjacent segments, and state recovery interval of continuous segments of historical sleep states, generating a set of personal rhythm trajectory anchor points; Based on the temporal sequence and the connection direction of adjacent segments in the personal rhythm trajectory anchor point set, the personal rhythm trajectory anchor point set is connected to generate a personal sleep inertial trajectory field. The current sleep time slice is encoded with state tendency, rhythm stage and change direction to generate the current sleep segment trajectory features; Map the current sleep segment trajectory features to the individual sleep inertial trajectory field to generate the current sleep trajectory points; Based on the current sleep trajectory point and the individual's sleep inertial trajectory field, trajectory position deviation processing, adjacent segment connection direction deviation processing, and state recovery interval deviation processing are performed to generate personal rhythm constraint evidence.

7. The sleep state detection method based on deep reinforcement learning according to claim 1, characterized in that, The generation of the detection correction action includes: Channel missing identification, noise fluctuation identification, body motion interference identification, and feature continuity identification are performed on multi-source sleep raw data and sleep time sequence feature segments to generate reliable signal features; Based on state continuity verification evidence, personal rhythm constraint evidence, current sleep trajectory point, personal sleep inertial trajectory field and signal reliability characteristics, time sequence correspondence and evidence relationship are organized to generate state correction observation features; Generate a candidate set of detection correction actions based on state correction observation features; Based on the continuity consistency evidence, continuity conflict evidence, and jump anomaly evidence in the state continuity verification evidence, combined with personal rhythm constraint evidence and signal credibility characteristics, state correction reward characteristics are generated. Based on the state correction observation features, the detection correction action candidate set, and the state correction reward features, deep reinforcement learning state correction processing is performed. The action value of each detection correction action in the detection correction action candidate set is iterated to generate an action value sequence. Actions are selected from the candidate set of detection and correction actions based on the action value sequence, and detection and correction actions are generated.

8. The sleep state detection method based on deep reinforcement learning according to claim 1, characterized in that, The generation of the sleep state detection results includes: The target sleep state corresponding to the current sleep time slice is determined from the candidate sleep state set based on the detection and correction action. The target sleep state is processed according to the detection and correction actions, including state confirmation, state replacement, state extension, state rollback, boundary adjustment, or abnormal transfer marking, to generate the current sleep state correction result. Based on the current sleep state correction results and the detection correction actions, the start and end positions of the state, the continuity of the state, and the state correction type are sorted out to generate the current sleep state sequence correction results. The sleep state detection result corresponding to the current sleep time slice is generated based on the correction result of the current sleep state sequence.

9. A sleep state detection system based on deep reinforcement learning, comprising executing the sleep state detection method based on deep reinforcement learning as described in any one of claims 1 to 8, characterized in that, include: The data acquisition and processing module is used to collect multi-source raw sleep data and user historical sleep records, and generate sleep time sequence feature segments, current sleep time slices and subsequent verification time windows; The candidate state generation module is used to generate a set of candidate sleep states based on the sleep time sequence feature segments corresponding to the current sleep time slice. The counterfactual continuation generation module is used to generate counterfactual sleep continuation segments based on the user's historical sleep records, candidate sleep state sets, and sleep time sequence feature segments corresponding to the current sleep time slice. The state continuation verification module is used to generate real subsequent sleep segments based on the subsequent verification time window, and to perform consistency verification between the real subsequent sleep segments and the counterfactual sleep continuation segments to generate state continuation verification evidence. The personal rhythm constraint module is used to generate a personal sleep inertial trajectory field, the current sleep trajectory point, and personal rhythm constraint evidence based on the user's historical sleep records and the sleep time sequence feature segments corresponding to the current sleep time slice; The enhanced state correction module is used to generate detection and correction actions based on state continuation verification evidence, personal rhythm constraint evidence, current sleep trajectory points, personal sleep inertial trajectory field, and signal credibility features; The detection result generation module is used to generate the sleep state detection result corresponding to the current sleep time slice based on the detection correction action.

Images