Sleep monitoring method and sleep monitoring equipment

Abstract

The embodiment of the invention provides a sleep monitoring method and sleep monitoring equipment. The sleep monitoring method comprises the following steps: acquiring continuous multi-frame two-dimensional temperature images obtained by performing temperature monitoring on a sleep scene by a two-dimensional temperature sensor, and acquiring a space sensing signal obtained by performing space sensing on the sleep scene by a body motion sensing radar at the same time, the body movement sensing radar is a radar capable of sensing body movement characteristics reflecting vital signs of an object; performing recognition processing on the multiple frames of two-dimensional temperature images to obtain a posture prediction sequence representing the body posture change of the object; performing joint recognition processing on the multiple frames of two-dimensional temperature images and the spatial sensing signals to obtain a sleep state prediction sequence representing the sleep feature state of the object; and identifying the sleep time of the object based on the posture prediction sequence and the sleep state prediction sequence. By adopting the scheme of the embodiment of the invention, the problem that the sleep time of the object determined through the information representing the vital signs of the user is not accurate can be found.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data processing, and particularly relates to a falling asleep monitoring method and a sleep monitoring device. Background Art

[0002] A reasonable falling asleep time, a reasonable sleep duration, and good sleep quality contribute to maintaining physical health. In daily life filled with intelligent electronic devices, many users are addicted to using intelligent electronic devices. After they have already planned to go to bed and lie down, they still use electronic devices for a long time to view social media information, which may have formed a falling asleep disorder, and the aforementioned falling asleep disorder problem has had a negative impact on normal daily routines and physical health.

[0003] Monitoring the falling asleep time of users can provide an objective basis for evaluating the sleep mechanism of users, help detect early sleep disorder problems of users at an early stage, and help provide data feedback support for users to purposefully improve their sleep habits. In order to monitor the sleep state of users without interfering with their normal life, relevant technologies propose to use non-contact sensors such as millimeter-wave radars to monitor the body movement of objects in the sleep scenario, and predict the sleep time of the object based on the body movement monitoring results. However, product research and development and verification tests have found that there are systematic errors in the estimated falling asleep time obtained by using body movement sensors such as millimeter-wave radars to monitor the sleep scenarios of some objects. Summary of the Invention

[0004] To be able to perform non-contact falling asleep monitoring, embodiments of the present disclosure provide a falling asleep monitoring method and a sleep monitoring device.

[0005] In a first aspect, embodiments of the present disclosure provide a falling asleep monitoring method, including: Obtaining a plurality of consecutive two-dimensional temperature images obtained by a two-dimensional temperature sensor monitoring the temperature of a sleep scenario, and obtaining a spatial perception signal obtained by a body movement perception radar simultaneously performing spatial perception on the sleep scenario, where the body movement perception radar is a radar capable of perceiving body movement characteristics reflecting the vital signs of an object; Performing recognition processing on the plurality of two-dimensional temperature images to obtain a posture prediction sequence characterizing the body posture change of the object; Performing joint recognition processing on the plurality of two-dimensional temperature images and the spatial perception signal to obtain a falling asleep state prediction sequence characterizing the falling asleep characteristic state of the object; Based on the posture prediction sequence and the falling asleep state prediction sequence, identifying the falling asleep time of the object.

[0006] Optionally, performing fusion processing on the plurality of two-dimensional temperature images and the spatial perception signal to obtain a falling asleep state prediction sequence characterizing the falling asleep characteristic state of the object, including: Process the multi-frame two-dimensional temperature images, determine the pixel regions representing typical body surface regions in each frame of the two-dimensional temperature images, determine the temperatures of the typical body surface regions based on the pixel regions representing the typical body surface regions, and sort the temperatures of the typical body surface regions according to the frame sorting of the two-dimensional temperature images to obtain a typical body surface temperature change sequence; wherein the typical body surface region is a region with significant temperature change characteristics during the sleep process. Process the spatial perception signals to obtain the vital signs of the object, and construct a vital sign change sequence based on the vital signs. Perform joint recognition processing based on the typical body surface temperature change sequence and the vital sign change sequence to obtain the sleep state prediction sequence.

[0007] Optionally, the typical body surface region includes at least one of the mouth and nose region and the forehead region, and the vital signs include at least one of respiratory characteristics, heart rate signs, pulse signs, and blinking characteristics.

[0008] Optionally, it further includes: obtaining an audio signal by a pick-up microphone for audio perception of the sleep scenario at the same time. Perform recognition processing on the audio signal to obtain the vocal characteristics of the object in each audio frame, and construct a vocal characteristic sequence based on the vocal characteristics. The joint recognition processing based on the typical body surface temperature change sequence and the vital sign change sequence includes: performing joint recognition processing based on the typical body surface temperature change sequence, the vital sign change sequence, and the vocal characteristic sequence to obtain the sleep state prediction sequence.

[0009] Optionally, based on the posture prediction sequence and the sleep state prediction sequence, identifying the sleep time of the object includes: Determining the preliminary sleep time of the object based on the sleep state prediction sequence. Correcting the preliminary sleep time with the posture prediction sequence to obtain the corrected sleep time.

[0010] Optionally, the sleep scenario includes a bed; the sleep time is the time when the object lies on the bed and falls asleep. The method further includes: determining the time when the object lies on the bed based on the posture prediction sequence. Determining the sleep efficiency of the object based on the time when the object lies on the bed and the sleep time.

[0011] Optionally, performing recognition processing on the multi-frame two-dimensional temperature images to obtain a posture prediction sequence representing the change of the object's body posture includes: Perform fusion processing on the multi-frame two-dimensional temperature images to obtain a fused temperature feature sequence; the fused temperature feature sequence includes at least one of the difference images of adjacent frames and the heat source distribution features of each frame of two-dimensional temperature images; Input the fused temperature feature sequence into a pose classification prediction model to obtain the predicted limb poses in each frame of two-dimensional temperature images; Sort the predicted limb poses according to the frame order of the two-dimensional temperature images to obtain the pose prediction sequence.

[0012] Optionally, perform recognition processing on the multi-frame two-dimensional temperature images to obtain a pose prediction sequence representing the change of the object's body pose, including: Perform difference processing on the multi-frame two-dimensional temperature images and the initial temperature images corresponding to the initial sleep scenario respectively to obtain difference images; the initial sleep scenario is a sleep scenario without an object; Based on the difference images corresponding to each frame of two-dimensional temperature images, perform recognition processing to determine the predicted limb poses of the object in each frame of two-dimensional temperature images; Sort the predicted limb poses according to the frame order of the two-dimensional temperature images to obtain the pose prediction sequence.

[0013] Optionally, perform recognition processing on the multi-frame two-dimensional temperature images to obtain a pose prediction sequence representing the change of the object's body pose, including: Perform preliminary recognition processing on each frame of two-dimensional temperature images respectively to determine the estimated pose, the corresponding confidence probability, and the pixel regions of each limb part of the object in each frame of two-dimensional temperature images; Take the estimated pose with a larger confidence probability as the predicted limb pose of the object in the corresponding two-dimensional temperature image, and take the corresponding two-dimensional temperature image as the initial reference temperature image; Calculate the difference features between the reference temperature image and the adjacent two-dimensional temperature images, and based on the difference features, the pixel regions of each limb part of the object in the reference temperature image, and the predicted limb pose of the reference temperature image, determine the predicted limb pose of the adjacent two-dimensional temperature images and the pixel regions of each limb part; and, Take the adjacent two-dimensional temperature images as the new reference temperature images and repeat the operation of determining the difference features, as well as the operations of determining the predicted limb pose of the object and the pixel regions of each limb part in the new reference temperature images, until the predicted limb poses of the object in all two-dimensional temperature images are determined; Sort the predicted limb poses according to the frame order of the two-dimensional temperature images to obtain the pose prediction sequence.

[0014] Optionally, it further includes: processing the multi-frame two-dimensional temperature images or spatial perception signals, estimating the number of objects in the sleep scenario, and the estimated positions of each object in the sleep scenario; Based on the estimated positions, perform target region cutting on the corresponding two-dimensional temperature images to determine the pixel regions representing the same object in each frame of two-dimensional temperature maps; And based on the estimated positions, determine the spatial perception signals representing the same object in the spatial perception signals; The recognition processing of the multi-frame two-dimensional temperature images to obtain a posture prediction sequence representing the body posture changes of the object includes: performing recognition processing on the pixel regions representing the same object in the multi-frame two-dimensional temperature images to obtain a posture prediction sequence representing the body posture changes of the same object; The joint recognition processing of the multi-frame two-dimensional temperature images and the spatial perception signals to obtain a sleep state prediction sequence representing the sleep characteristic state of the object includes: performing joint recognition processing on the pixel regions representing the same object in the multi-frame two-dimensional temperature images and the filtered fluctuation signals representing the same object to obtain a sleep state prediction sequence representing the sleep characteristic state of the same object.

[0015] In a second aspect, an embodiment of the present disclosure provides a sleep monitoring device, including: a two-dimensional temperature sensor, a body movement perception radar, and a processor; The two-dimensional temperature sensor is used to monitor the temperature of the sleep scenario and generate two-dimensional temperature images, and the body movement perception radar is used to perform spatial perception on the sleep scenario to obtain spatial perception signals; The signal input end of the processor is connected to the signal output ends of the two-dimensional temperature sensor and the body movement perception radar, and is used to determine the falling asleep time of the object according to the falling asleep monitoring method according to any one of claims 1-9 when receiving the two-dimensional temperature images and the spatial perception signals.

[0016] Adopting the solution of the embodiment of the present disclosure, based on the two-dimensional temperature images obtained by monitoring the sleep scenario by the two-dimensional temperature sensor, determine the posture prediction sequence of the object, and the sleep state sequence initially representing the falling asleep state of the object based on the two-dimensional temperature images and the spatial perception signals, and then determine the corrected falling asleep time based on the posture state prediction sequence and the sleep state sequence. Adopting the solution of the embodiment of the present disclosure can discover some problems that it is not accurate to determine the falling asleep time of the object by the information representing the user's vital signs. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings here are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0018] To more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings, where: Figure 1 is a schematic structural diagram of a sleep monitoring device provided by an embodiment of the present disclosure; Figure 2 is a flowchart of a falling asleep monitoring method provided by some embodiments of the present disclosure; Figure 3 is a flowchart of a method for obtaining a pose prediction sequence based on a two-dimensional temperature image in some example embodiments; Figure 4 is a flowchart of a method for obtaining a pose prediction sequence based on a two-dimensional temperature image in some other example embodiments; Figure 5 is a flowchart of a method for obtaining a pose prediction sequence based on a two-dimensional temperature image in still some other example embodiments; Figure 6 is a flowchart of a method for obtaining a falling asleep state prediction sequence in some embodiments of the present disclosure. Detailed Embodiments

[0019] The following will describe the embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the accompanying drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0020] As used herein, the term "including" and its variants are open-ended, that is, "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description. In this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0021] As mentioned in the background art, when using a millimeter-wave radar as a non-contact sensor to monitor the body movement characteristics of an object and predicting the sleep time of the object using the body movement data, systematic errors will occur. Through the analysis of the behavior of the object before going to sleep, it is found that the specific objects with the aforementioned systematic errors have the habit of reading books or using electronic devices before going to sleep, and tend to involuntarily fall asleep during the process of reading books and using electronic devices. When the object is reading a book and using an electronic device, due to its relaxed and low-activity state, body movement characteristics such as the respiration rate that characterize vital signs will be actively controlled by the autonomic nerves and decrease. However, during the process of the decrease of the aforementioned body movement characteristics of the object, the object is still awake and has not fallen asleep. That is to say, based on the body movement characteristics that characterize the vital signs of the object for sleep monitoring, there may be a problem that the predicted sleep time is earlier than the actual sleep time.

[0022] To solve the problem of inaccurate prediction of the sleep time in the aforementioned scenario, an embodiment of the present disclosure provides a new sleep time monitoring method. The sleep time monitoring method provided by the embodiment of the present disclosure is implemented on the premise of a specific sleep monitoring device. First, a brief introduction to the sleep monitoring device used in the embodiment of the present disclosure will be given below, and then an analysis of the sleep monitoring method based on the aforementioned sleep monitoring device will be made.

[0023] Figure 1 It is a schematic structural diagram of the sleep monitoring device provided by the embodiment of the present disclosure. As Figure 1 shown, the sleep monitoring device 100 used in the embodiment of the present disclosure includes a two-dimensional temperature sensor 101, a body movement sensing radar 102, a data processing device 103, and an output device 104.

[0024] The two-dimensional temperature sensor 101 is a sensor that performs two-dimensional temperature sensing in a non-contact manner, and it can sense the ambient temperature at various places in the three-dimensional environment to generate a two-dimensional temperature image. The temperature data at each position of the two-dimensional temperature image represents the temperature state of the heat source in the actual environment.

[0025] In specific implementation, the two-dimensional temperature sensor 101 can be an infrared camera, a non-imaging infrared sensor, a fiber optic distributed temperature sensor, etc. It should be noted here that in order to be able to identify the limb posture of the object in the following text, the number of sensing units in the two-dimensional temperature sensor 101 and the resolution of the corresponding two-dimensional temperature image formed should be coordinated with the usage scenario to achieve the ability to identify the limb position and relative position relationship of the object.

[0026] The body movement sensing radar is a radar that uses electromagnetic waves with a reasonable band and form to sense the spatial state, and realizes distance measurement, speed measurement, and azimuth determination (or angle determination) of obstacles in the space.

[0027] In practical applications, in order to monitor the active body movements of an object and the body movement characteristics representing vital signs in the sleep scenario, a body movement perception radar is a radar that can achieve millimeter-level precision perception. In specific implementations, the body movement perception radar can be in the form of a millimeter-wave radar, a UWB radar, etc., and preferably a frequency-modulated continuous-wave or pulsed millimeter-wave radar is adopted. In addition, in order to be able to measure fluctuations in more point data and identify changes in the positions of obstacles at different positions, according to the requirements of the actual scenario and the detection target, the body movement perception radar may be a radar with multiple transmitting antennas and multiple receiving antennas, such as a MIMO radar.

[0028] The data processing device 103 is a device for processing the two-dimensional temperature image and the signal output by the body movement perception radar, and estimating the object's falling asleep time based on the two-dimensional temperature image and the signal output by the body movement perception radar.

[0029] In specific implementations, the data processing device 103 can be a device that integrates a processor (such as a CPU), a memory, a data memory, etc. with a PCB circuit board as the carrier substrate. After the data processing device is powered on, the processor can load the program code stored in the data memory to form executable instructions, and process the signals transmitted by the two-dimensional temperature sensor and the body movement perception radar to obtain corresponding processing results. In some other embodiments, the processor, memory, and data memory in the data processing device 103 can be integrated in one chip, so that the data processing device 103 is as miniaturized as possible.

[0030] The output device 104 is a device for outputting the recognition and determination result. In specific implementations, the output device 104 can be a display-type output device 104 (such as various types of display screens), or an audio output device 104 such as a speaker. In some other embodiments, the output device 104 can also be replaced by a communication device, that is, the sleep monitoring device sends the corresponding data results to other devices such as a smart phone through the communication device, and the foregoing other devices output the corresponding results.

[0031] In addition, in some specific applications, the sleep monitoring device can also include a pick-up microphone. The pick-up microphone is used to pick up the sound signal in the sleep scenario and send it to the data processing device 103. The corresponding data processing device determines the object's falling asleep time based on the processing of the sound signal and other signals.

[0032] The following analyzes the falling asleep monitoring method provided by the embodiments of the present disclosure. Figure 2 It is a flowchart of the falling asleep monitoring method provided by some embodiments of the present disclosure. As Figure 2 shown, the falling asleep monitoring method provided by the embodiments of the present disclosure includes S110 - S140.

[0033] S110: Acquire a continuous multi-frame two-dimensional temperature image obtained by a two-dimensional temperature sensor performing temperature monitoring on a sleeping scene, and acquire a spatial perception signal obtained by a body motion perception radar performing spatial perception on a sleeping scene at the same time.

[0034] As analyzed above, the body motion sensing radar in the embodiments of the present disclosure is a radar that can sense the body motion characteristics of the user's vital signs. In specific applications, it is mostly the millimeter wave radar exemplified above or a UWB radar with millimeter wave monitoring accuracy.

[0035] In the disclosed embodiment, the two-dimensional temperature sensor and body motion sensing radar in the aforementioned sleep monitoring device are deployed at a suitable position in the sleep scene, and can monitor the sleep scene (mainly the bed) at a suitable angle. In the case where the sleep scene is a bed, it is more preferred to install the two-dimensional temperature sensor and the body motion sensing radar directly above the bed, at a high position at the head of the bed, or on a bedside table, so that the two-dimensional temperature sensor and the body motion sensing radar can both sense the sleep scene in a manner facing the bed, and all parts of the sleep scene are as close to the middle field of view as possible in the field of view of the two-dimensional temperature sensor and the body motion sensing radar, so as to obtain a two-dimensional temperature image and a spatial sensing signal with better resolution.

[0036] After completing the deployment of the aforementioned two-dimensional temperature sensor and starting the sleep monitoring device, the two-dimensional temperature sensor and body motion sensing radar begin to sense the environmental status according to the pre-configured sampling frequency and signal receiving and transmitting frequency.

[0037] Here is an explanation of the space perception signal obtained by the body motion perception radar. Based on the radar monitoring principle, the body motion perception radar uses a transmitting antenna to transmit electromagnetic waves, uses a receiving antenna to receive electromagnetic waves, and performs mixing processing based on the space perception signal, and realizes space state perception based on the characteristics of the mixed signal. The specific implementation uses a mixer to mix the transmitted electromagnetic wave and the received electromagnetic wave (for example, to obtain the instantaneous frequency difference and the instantaneous phase difference) to obtain a mixed signal based on the space perception signal processing, and then processes the mixed signal to obtain space perception information.

[0038] S120: performing recognition processing on multiple frames of two-dimensional temperature images to obtain a posture prediction sequence representing changes in the body posture of the object.

[0039] After acquiring multiple frames of two-dimensional temperature images, the data processing device recognizes and processes each frame of the two-dimensional temperature image, identifies the predicted limb posture of the object in each frame of the two-dimensional temperature image, and then sorts the predicted limb posture according to the frame sorting of the two-dimensional temperature image to obtain a posture prediction sequence.

[0040] Limb posture is a posture that represents the relative position relationship between the limbs of an object. Accordingly, the predicted limb posture is obtained by recognizing the two-dimensional temperature.

[0041] From the perspective of the whole body, the limb posture can be a squatting posture, a sitting posture, or a lying posture (recumbent posture); from the perspective of the upper limb relative to the drive, the limb posture can be a raised hand (figuratively speaking, at least there is a large angle between the extension direction of the forearm of the upper limb and the extension direction of the torso, or the forearm of the upper limb is very close to the user's face area. In this case, the posture represents the situation where the user is reading a book, using an electronic device such as a smart phone, etc.), natural hanging (more specifically, when the user is standing or lying, the upper limb is naturally placed on the side of the torso, and the extension direction of the upper limb is basically the same as the extension direction of the torso. At this time, it is already difficult to distinguish the boundary position between the torso and the upper limb), covering the stomach (figuratively speaking, the user's hand is placed on the chest and abdomen, and a large angle is formed between the forearm and the upper arm), etc.

[0042] In specific implementation, the data processing device can adopt various processing strategies to obtain the predicted limb posture of the object in the two-dimensional temperature image, and then obtain the posture prediction sequence.

[0043] Figure 3 It is a flowchart of a method for obtaining a posture prediction sequence based on a two-dimensional temperature image in some example embodiments. As Figure 3 shown, in some embodiments, the data processing device can adopt the following S121 - S123 to obtain the posture prediction sequence.

[0044] S121: Perform fusion processing on multiple frames of two-dimensional temperature images to obtain a fused temperature feature sequence.

[0045] In specific implementation, performing fusion processing on multiple frames of two-dimensional temperature images may include the following (1) - (2) (1) Perform differential comparison processing on adjacent frames of two-dimensional temperature images to determine the differential image of the adjacent frames of two-dimensional temperature images. Specifically, subtract the temperature value of the two-dimensional temperature image of the subsequent frame from the corresponding temperature value at the corresponding position in the two-dimensional temperature image of the previous frame to obtain the temperature difference value, and then perform two-dimensional sorting on the temperature difference values according to the two-dimensional temperature image to obtain the differential image.

[0046] (2) Extract significant features based on each frame of two-dimensional temperature image to determine the significant high-temperature region of each frame (in practical applications, because a person is the core heat source in the sleep environment, and their body surface temperature is much higher than that of the bed or bedding), and determine the spatial temperature distribution feature of each two-dimensional temperature image based on the significant high-temperature region. When determining the spatial temperature distribution feature of a certain two-dimensional temperature image here, it is necessary to infer by referring to the spatial temperature distribution feature of the adjacent two-dimensional temperature image before or after. This is considered to be fusion processing here.

[0047] In the case of obtaining the aforementioned differential image and spatial temperature distribution characteristics simultaneously, the differential image and spatial temperature distribution characteristics corresponding to the same frame of two-dimensional temperature image can be put together to form a fusion feature for a two-dimensional temperature image. Subsequently, the aforementioned fusion features are sorted according to the frame identifier of the two-dimensional temperature image to form a fusion temperature feature sequence.

[0048] In other embodiments, the fusion features included in each position of the fusion feature array sequence may include, in addition to the aforementioned differential image features and spatial temperature distribution features, other features that can contain information about the type of user's limb movement.

[0049] S122: Input the fusion temperature feature sequence into the pose classification prediction model to obtain the predicted limb poses of the object in each frame of the two-dimensional temperature image.

[0050] The pose classification prediction model is a model that processes the fusion temperature feature sequence with implicit pose transformation relationships and identifies the limb pose features of the object at a specific moment. The pose classification prediction model can be trained using a sample fusion temperature feature sequence and the corresponding pose label sequence. In a specific implementation, the pose classification prediction model can be a Long Short-term Memory (LSTM) model.

[0051] In a specific implementation, a specific number of fusion features before a certain frame in the fusion temperature feature sequence can be used as the input, and the object limb pose label corresponding to this frame can be used as the output to train the initial pose classification prediction model to obtain a usable pose classification prediction model. Correspondingly, by inputting the fusion temperature feature sequence into the aforementioned trained pose classification prediction model, the predicted limb poses of the object in each frame of the two-dimensional temperature image can be obtained.

[0052] S123: Sort the predicted limb poses according to the frame sorting of the two-dimensional temperature image to obtain a pose prediction sequence.

[0053] Figure 4 is a flowchart of a method for obtaining a pose prediction sequence based on two-dimensional temperature images in some other embodiments. As Figure 4 shown, in some other embodiments, the data processing device can obtain the pose prediction sequence by using the following S124 - S126.

[0054] S124: Perform differential processing on multiple frames of two-dimensional temperature images respectively with the initial temperature image corresponding to the initial sleep scenario to obtain differential images.

[0055] The initial sleep scene is a predetermined sleep scene without an object. In some embodiments, after installing the two-dimensional temperature sensor, the user can follow the instructions for use, leave the bedroom or other sleep scenes (the initial sleep scene is formed at this time) and trigger the two-dimensional temperature sensor for temperature monitoring to obtain an initial temperature image corresponding to the initial sleep scene. It can be understood from the aforementioned strategy for obtaining the initial sleep scene that the initial temperature image is an image that does not contain the temperature characteristics of the object.

[0056] It can be imagined that the two-dimensional temperature image and the initial temperature image are subjected to differential processing. If the scene area corresponding to a point in the two-dimensional temperature image is not covered by the object's limbs, or the object's limbs are covered by a quilt or other covering, the differential value corresponding to this point is small (the value of most areas is 0). If the scene area corresponding to a point in the two-dimensional temperature image is not covered by the object's limbs, and the object's limbs are exposed, the temperature value of this point is large, and the corresponding differential value is large. The differential image composed of the above differential values ​​represents the difference between the two-dimensional temperature image and the initial temperature image caused by the entry of the object.

[0057] S125: Perform recognition processing based on the differential image corresponding to each frame of the two-dimensional temperature image to determine the predicted limb posture of the object in each frame of the two-dimensional temperature image.

[0058] In some embodiments, the recognition processing is performed based on the differential images corresponding to each frame of the two-dimensional temperature image, and the differential images of each frame of the two-dimensional temperature image are respectively recognized and processed to determine the predicted limb posture of the object therein. Similar to the processing method described above, the aforementioned processing for each differential image can be performed using a pre-trained posture classification prediction model, except that the posture classification prediction model at this time can only input a differential temperature sequence of one frame.

[0059] In some other embodiments, the data processing device further executes the following S125A-S125B to realize recognition of the posture of the object in each frame.

[0060] S125A: Perform fusion processing based on the differential images corresponding to each frame of the two-dimensional temperature image to obtain a fused temperature feature sequence.

[0061] S125B: Input the fused temperature feature sequence into a pre-trained posture classification prediction model to determine the predicted limb posture in each frame of the two-dimensional temperature image of the object.

[0062] The fused temperature feature sequence obtained by the fusion processing is a sequence that implicitly represents the posture features of the object in the corresponding period of multiple consecutive two-dimensional temperature images. In specific implementation, the method mentioned in S122 can be used to process the differential image corresponding to each frame of two-dimensional temperature image to obtain the fused temperature feature sequence.

[0063] S126: Sort the predicted limb postures according to the frame order of the two-dimensional temperature image to obtain a posture prediction sequence.

[0064] Figure 5 The following is a flow chart of a method for obtaining a posture prediction sequence based on a two-dimensional temperature image in some embodiments. Figure 5 As shown, in some further embodiments, the data processing device may adopt the following S127-S12C to obtain the predicted limb posture of the object in the two-dimensional temperature image.

[0065] S127: Perform recognition processing on each frame of the two-dimensional temperature image to determine the estimated posture of the object in each frame of the two-dimensional temperature image, the corresponding confidence probability, and the pixel area of ​​each limb part.

[0066] In a specific implementation, a model processing method may be used to process the two-dimensional temperature image (it may be to process only a single frame of the two-dimensional temperature image, or it may be to process or fuse multiple frames of the two-dimensional temperature image simultaneously) to obtain the estimated state of the object therein.

[0067] In a specific implementation, the posture prediction model is used to process the posture to obtain the output probability, and the posture with the highest probability is selected as the estimated state. Accordingly, the output probability corresponding to the aforementioned estimated posture can be used as the corresponding confidence probability. That is to say, at this time, the data processing device uses the posture with the highest probability as the estimated posture and the aforementioned maximum probability as the confidence probability.

[0068] In the process of determining the aforementioned estimated posture, the data processing device can also obtain the identification frame of the object, and determine the pixel area of ​​each limb part through the aforementioned estimated posture and the temperature distribution characteristics of the two-dimensional temperature image. The pixel area of ​​each limb part is the area of ​​the two-dimensional temperature image that represents the corresponding limb part. It should be noted that the aforementioned "each limb part" should be understood as a part that can be represented by a part of the sub-area in the two-dimensional temperature image. For example, a limb part that is covered by a quilt in an actual scene and cannot obtain the corresponding temperature is no longer considered to be included in the aforementioned "each limb part".

[0069] S128: Using the estimated posture with a larger confidence probability as the predicted limb posture of the object in the corresponding two-dimensional temperature image, and using the corresponding two-dimensional temperature image as the initial reference temperature image.

[0070] After determining the estimated posture and corresponding confidence probability in each frame of the two-dimensional temperature image, the confidence probabilities of each frame are then compared to determine the larger confidence probability. In a specific implementation, the larger confidence probability can be the maximum confidence probability, or a confidence probability greater than a set probability value.

[0071] According to the processing strategy of the neural network model and the foregoing method for obtaining the confidence probability, the greater the confidence probability, the greater the possibility that the first pose prediction model believes that the corresponding pose is the actual object pose. Based on this, by comparing the confidence probabilities horizontally, it can be determined which predicted pose corresponding to a two-dimensional temperature image is more credible, and the foregoing more credible predicted pose can be used as the predicted limb pose of the corresponding two-dimensional temperature image.

[0072] In the following text, based on the two-dimensional temperature images for which it is relatively easy to determine the predicted limb poses, for other two-dimensional temperature images S129: Calculate the difference features between the reference temperature image and the adjacent two-dimensional temperature image, and determine the predicted limb pose of the adjacent two-dimensional temperature image, as well as the pixel regions of each limb part, based on the difference features, the pixel regions of each limb part of the object in the reference temperature image, and the predicted limb pose of the reference temperature image.

[0073] The adjacent two-dimensional temperature image mentioned here can be the adjacent two-dimensional temperature image in the front or the adjacent two-dimensional temperature image in the back. Calculating the difference features between the reference temperature image and the adjacent two-dimensional temperature image can be directly calculating the difference array between the reference temperature image and the adjacent two-dimensional temperature image, or calculating other difference features by other means.

[0074] Since the foregoing difference features are the features reflecting the change of the object pose, based on the foregoing difference features and the pixel regions of each limb part of the object in the reference temperature image, the relative change of the object limbs over time can be determined. Subsequently, by integrating the predicted limb pose of the reference temperature truth path and the foregoing relative change, the predicted limb pose of the object in the adjacent two-dimensional temperature image can be determined. In specific implementation, a state transition recognition model can be used, for example, to determine the predicted limb pose of the adjacent two-dimensional temperature image based on the difference features, the pixel regions of each limb part of the object in the reference temperature image, and the predicted limb pose of the reference temperature image. In other words, in specific implementation, a model with content reasoning ability can be used to process the foregoing data to determine the recognition pose features of the adjacent two-dimensional temperature image, as well as the pixel regions of each limb part.

[0075] S12A: Determine whether there is still a two-dimensional temperature queue for which the predicted limb pose has not been determined; if so, execute S12B; if not, execute S12C.

[0076] S12B: Take the adjacent two-dimensional temperature image as the new reference temperature image and execute S127.

[0077] S12C: Sort the predicted limb poses according to the frame sorting of the two-dimensional temperature images to obtain a pose prediction sequence.

[0078] Using the foregoing S127-S12B, the predicted limb postures corresponding to the two-dimensional temperature with the highest confidence probability can be used to calculate the predicted limb postures of other two-dimensional temperature images.

[0079] In specific implementation, for a two-dimensional temperature image with a relatively low confidence probability, the foregoing method may obtain two predicted limb postures for it from the previous reference temperature image and the subsequent reference temperature image. The following two situations may occur: (1) The two foregoing predicted limb postures are the same; (2) The front and back predicted limb postures are different. In the foregoing situation (1), the predicted limb posture can be directly determined; in the foregoing situation (2), the difference magnitude between the two-dimensional temperature image and the reference temperature within the previous and subsequent reference temperatures can be identified, and the reference temperature image with a smaller difference and the limb posture determined by the corresponding difference feature are selected as the final predicted limb posture, or the final predicted limb posture can also be obtained through a re-fusion method.

[0080] S130: Jointly identify and process multiple frames of two-dimensional temperature images and spatial perception signals to obtain a sleep state prediction sequence representing the sleep characteristic state of the object.

[0081] The sleep state prediction sequence is a sequence for predicting the sleep state of the object at each frame moment within the time period of the foregoing collected data.

[0082] In some embodiments, the data processing device may directly input multiple frames of two-dimensional temperature images and spatial perception signals into a pre-trained deep learning model, and the deep learning model directly processes the input data to obtain a sleep state prediction sequence.

[0083] Figure 6 It is a flowchart of a method for obtaining a sleep state prediction sequence according to some embodiments of the present disclosure. As Figure 6 shown, in some embodiments, the data processing device may use the following S131-S133 to obtain a sleep state prediction sequence.

[0084] S131: Process multiple frames of two-dimensional temperature images, determine the pixel regions representing the typical body surface regions in each frame of two-dimensional temperature image, and based on the pixel regions representing the typical body surface regions, determine the temperatures of the typical body surface regions, and sort the temperatures of the typical body surface regions according to the frame sorting of the two-dimensional temperature images to obtain a typical body surface temperature change sequence.

[0085] The typical body surface region is a region with significant temperature change characteristics during the sleep process, specifically a region where the temperature characteristics of the object are significantly different in the sleep state and the waking state. In actual implementation, the target body surface region may be the forehead region or the mouth and nose region of the object. Here, from the perspective of biological principles, an analysis is made on why the temperature characteristics of the foregoing body surface regions can reflect the sleep state of the object.

[0086] Taking the oral and nasal region as an example, since the oral and nasal region is the area where the exhaled gas of the object directly diffuses and conducts heat, with the switching of the breathing state, the air temperature in the oral and nasal region will show obvious fluctuations, manifested as a higher temperature in the exhalation state and a lower temperature in the inhalation state. Because the breathing frequency of the object is significantly different when it enters the sleep state and the waking state, correspondingly, the frequency of temperature fluctuations in the oral and nasal region is significantly different in different states. In addition, under normal circumstances, the user will not sleep with the head covered, and the oral and nasal region will be exposed to the monitoring field of the two-dimensional temperature sensor, so that the two-dimensional temperature image formed can include the temperature characteristics at different times. In addition, the temperature of the oral and nasal region may be higher than that of other body surface regions (combined with its significant temperature volatility), so it is possible to determine the pixel region representing the oral and nasal region.

[0087] Similarly, it is also possible to obtain the pixel region representing the user's forehead region.

[0088] After determining the pixel regions representing the typical body surface regions as described above, perform a weighted sum processing on the temperature values of each pixel point in the aforementioned pixel regions, and it is possible to obtain the temperature of the typical body surface region at the corresponding frame moment of the typical region. Sort the temperatures of the typical body surface regions of the object according to each frame of the two-dimensional temperature image, and a typical body surface temperature change sequence can be obtained.

[0089] In specific implementation, when there are more than one typical body surface regions, the typical body surface temperature change sequences corresponding to each typical body surface region can be spliced to form a multi-region typical body surface temperature change sequence.

[0090] S132: Process the spatial perception signal to obtain the vital signs of the object, and construct a vital sign change sequence based on the vital signs.

[0091] As mentioned before, the vital signs in the embodiments of the present disclosure can be at least one of respiratory characteristics, heart rate characteristics, pulse characteristics, and blinking characteristics. In addition, although the blinking characteristic is not considered a vital sign from the perspective of physiology or medicine, it is also considered here as a vital characteristic that may indicate whether the object enters the sleep state.

[0092] According to general physiological experience, after the object enters the sleep state, its breathing frequency and heart rate (corresponding to the pulse frequency) also change typically (specifically, because the body power consumption characteristics decrease, the aforementioned characteristics decrease).

[0093] In specific implementation, to process the spatial perception signal to obtain the body movement characteristics of the object, it can first adopt methods such as discrete Fourier transform to identify the regions that reflect respiratory changes (such as the chest and abdomen), heart rate changes (such as the neck area), and pulse changes (such as the inner side of the wrist) at each sampling frame. Subsequently, the displacement characteristics of the aforementioned regions over time are identified to determine the object's respiratory rate, heart rate, pulse rate, etc., and the aforementioned frequency characteristics are used as the body movement characteristics representing the object's vital signs. Based on the aforementioned body movement characteristics, a body movement change sequence is constructed. Of course, the distances and angles from the aforementioned regions such as the chest and abdomen and the neck area to the body movement perception radar can also be used as the body movement characteristics of the object to construct a vital sign change sequence.

[0094] Similar to the previous analysis, in the case of obtaining multiple vital sign change sequences, the multiple vital sign change sequences can be spliced to form a multi-sign vital sign change sequence.

[0095] S133: Perform joint recognition processing based on the typical body surface temperature change sequence and the vital sign change sequence to obtain the falling asleep state prediction sequence.

[0096] In specific implementation, a pre-trained falling asleep state recognition model can be used to process the typical body surface temperature change sequence and the vital sign change sequence to obtain the falling asleep state prediction sequence.

[0097] After obtaining the posture prediction sequence and the falling asleep state prediction sequence, then execute S140.

[0098] S140: Based on the posture prediction sequence and the falling asleep state prediction sequence, identify the falling asleep time of the object.

[0099] As previously analyzed, the falling asleep state prediction sequence is already a sequence representing whether the object enters the falling asleep state at each frame moment. To identify the falling asleep time of the object based on the posture prediction sequence and the falling asleep state prediction sequence, regression analysis is performed on the falling asleep state prediction sequence using the posture prediction sequence to correct the determination of the falling asleep time in the falling asleep state prediction sequence, and then the falling asleep time of the object is determined.

[0100] As previously analyzed, during the falling asleep process, the object is in a relaxed and low-activity state, and body movement characteristics representing vital signs such as the respiratory rate will be actively controlled by the autonomic nervous system and decrease; but at this time, the object may still be reading a book or using an electronic device to watch video content, and thus maintain a specific posture because it needs to hold the book or electronic device; and it can be determined from the actions of the object maintaining the aforementioned posture that the object has not entered the sleeping state at this time. Correspondingly, when the object truly enters the sleeping state and changes its body posture (for example, placing the hand on the side of the body or on the chest and abdomen), combined with the falling asleep state sequence determined based on the body movement characteristics, it can be confirmed that the object has truly entered the sleeping state, and the corresponding time is the more accurate falling asleep time.

[0101] In specific implementation, the data processing device can directly perform overall processing on the falling asleep state prediction sequence and the posture prediction sequence by using a correction processing model to obtain a relatively accurate falling asleep time.

[0102] In some embodiments, the data processing device can also first determine a preliminary falling asleep time of the object based on the falling asleep state prediction sequence, and then correct the preliminary falling asleep time with the posture prediction sequence to obtain a corrected falling asleep time. Specifically, the preliminary falling asleep time can be used to determine the posture actions at the corresponding frame moments in the posture prediction sequence, and then retrieve backward from this posture action to determine the posture action that reflects the object entering the sleep state, and use the frame moment corresponding to this posture action as the falling asleep time.

[0103] Adopting the solution of the embodiment of the present disclosure, a posture prediction sequence of the object is determined based on the two-dimensional temperature image obtained by monitoring the sleep scenario with a two-dimensional temperature sensor, a falling asleep state sequence that preliminarily characterizes the falling asleep state of the object is obtained based on the two-dimensional temperature image and the spatial perception signal, and then a corrected falling asleep time is determined based on the posture state prediction sequence and the falling asleep state sequence. By adopting the solution of the embodiment of the present disclosure, it can be found that there are some problems in accurately determining the falling asleep time of the object by using the information characterizing the user's vital signs.

[0104] In some embodiments, the sleep monitoring device further includes a microphone. During the process of obtaining the foregoing two-dimensional temperature image and spatial perception signal, the microphone will also simultaneously perform audio perception on the sleep scenario to obtain an audio signal. After the data processing device obtains the audio signal, it will also perform recognition processing on the audio signal to obtain the vocalization characteristics of the object in each audio frame, and construct a vocalization characteristic sequence based on the vocalization characteristics.

[0105] In the case of obtaining the foregoing vocalization characteristic sequence, the data processing device can perform joint recognition based on the typical body surface temperature change sequence, the vital sign change sequence, and the vocalization characteristic sequence to obtain a falling asleep state prediction sequence.

[0106] According to life experience, most subjects sleep in bed, and the subject needs to go to bed and lie down before entering the sleep state after a period of time. Most sleeping scenes are sleeping scenes in the bedroom including the bed, and the corresponding sleeping time is the time when the subject implants and falls asleep. In order to reasonably determine the sleep quality of the subject, before execution, the present disclosure will determine the implantation time of the subject based on the posture prediction sequence after executing the aforementioned S120 to obtain the posture prediction sequence that characterizes the changes in the subject's body posture. In a specific implementation, the two-dimensional temperature sensor always monitors the bedroom scene, obtains all the motion features of the subject's activities in the bedroom, including the posture prediction sequence of the subject's implantation motion features (for example, from standing upright to sitting down, and then lying down or lying on the side, which is reflected from the perspective of the bed position as no one, someone, and someone lying down on the bed), and obtains the implantation time of the subject through the aforementioned sequence.

[0107] When the implantation time and the sleep time of the subject are determined, the sleep efficiency of the subject can also be determined based on the implantation time and the sleep time. The sleep efficiency in the disclosed embodiment is characterized by the time period length of the sleep time and the implantation time. The longer the time period length, the lower the sleep efficiency of the subject.

[0108] The above solution assumes that there is only one object in the sleep scene. In actual situations, there may be more than one object in the sleep scene. If the above solution is used directly, it may cause problems such as body motion characteristics and posture recognition errors, which will interfere with the prediction of the sleep time of each object. To solve this problem, before executing the above S120 and S130, the data processing device can also execute the following S150-S170.

[0109] S150: Process multiple frames of two-dimensional temperature images or spatial perception signals to estimate the number of objects in the sleeping scene and the estimated position of each object in the sleeping scene.

[0110] In a specific implementation, the method for estimating the number and position of objects in a sleeping scene can be determined based on a two-dimensional temperature image, can be determined based on a spatial perception signal, or can be determined based on a fusion of the two.

[0111] In some embodiments, the data processing device can use deep models such as convolutional neural networks, support vector machines, K-nearest neighbor algorithms, connectivity graph methods, etc. to process two-dimensional temperature images or spatial perception signals to obtain the number of objects, and determine the region cutting strategy based on the recognition results to obtain the estimated position of each object in the sleep scene.

[0112] After the estimated position is obtained, the following S160 and S170 may be executed.

[0113] S160: Cut the target area of the corresponding two-dimensional temperature image based on the estimated position, and determine the pixel areas representing the same object in each frame of the two-dimensional temperature map.

[0114] In specific implementation, the data processing device can determine the pixel areas representing the same object in the two-dimensional temperature map according to the estimated position based on the corresponding relationship between each position and the pixel areas in the two-dimensional image.

[0115] S170: Determine the spatial perception signals representing the same object in the spatial perception signals based on the estimated position.

[0116] In specific implementation, the data processing device determines the spatial perception signals representing the same object based on the spatial position of the object reflected in the spatial perception signals and the aforementioned estimated position.

[0117] In the case of executing the aforementioned S150 - S170, the aforementioned S120 specifically refers to performing recognition processing on the pixel areas representing the same object in multiple frames of two-dimensional temperature images to obtain a posture prediction sequence representing the body posture changes of the same object, and the aforementioned S130 specifically refers to performing joint recognition processing on the pixel areas representing the same object in multiple frames of two-dimensional temperature images and the screened fluctuation signals representing the same object to obtain a sleep state prediction sequence representing the sleep characteristic state of the same object.

[0118] An embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium. This computer program contains program codes for executing the method shown in the flowchart. In such an embodiment, when this computer program is executed by the data processing device, it executes the above functions defined in the method of the embodiment of the present disclosure.

[0119] The aforementioned computer-readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or combined with an instruction execution system, device, or component.

[0120] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or combinations thereof, including but not limited to object-oriented programming languages such as Java, Smalltalk, C++, and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the tester's computer, partially on the tester's computer, as a stand-alone software package, partially on the tester's computer and partially on a remote computer, or entirely on a remote computer or computing device. In the case of a remote computer, the remote computer may be connected to the tester's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a special hardware-based system that performs the specified functions or operations, or by a combination of special hardware and computer instructions.

[0122] The above are only specific embodiments of the present disclosure to enable those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but rather will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for monitoring falling asleep, characterized in that: include: Acquire a continuous multi-frame two-dimensional temperature image obtained by a two-dimensional temperature sensor for temperature monitoring of a sleeping scene, and acquire a spatial sensing signal obtained by a body motion sensing radar for spatial sensing of the sleeping scene at the same time, wherein the body motion sensing radar is a radar capable of sensing body motion characteristics reflecting vital signs of the subject; Performing recognition processing on the multiple frames of two-dimensional temperature images to obtain a posture prediction sequence representing changes in the body posture of the object; Performing joint recognition processing on the multiple frames of two-dimensional temperature images and the spatial perception signal to obtain a sleeping state prediction sequence representing a characteristic sleeping state of the subject; Based on the posture prediction sequence and the sleeping state prediction sequence, the sleeping time of the subject is identified.

2. The sleep monitoring method according to claim 1, characterized in that: The multiple frames of two-dimensional temperature images and the spatial perception signals are fused to obtain a sleep state prediction sequence representing a characteristic state of the subject falling asleep, including: Processing the multiple frames of two-dimensional temperature images, determining a pixel area representing a typical body surface area in each frame of the two-dimensional temperature image, determining a typical body surface area temperature based on the pixel area representing the typical body surface area, and sorting the typical body surface area temperature according to the frame sorting of the two-dimensional temperature image to obtain a typical body surface temperature change sequence; wherein the typical body surface area is an area whose temperature has a significant change characteristic with the sleep process; Processing the spatial perception signal to obtain the vital signs of the subject, and constructing a vital sign change sequence based on the vital signs; The sleeping state prediction sequence is obtained by performing joint recognition processing based on the typical body surface temperature change sequence and the vital sign change sequence.

3. The sleep monitoring method according to claim 2, characterized in that: The typical body surface area includes at least one of the mouth and nose area and the forehead area, and the vital signs include at least one of breathing characteristics, heart rate signs, pulse signs, and blinking characteristics.

4. The sleep monitoring method according to claim 2, characterized in that: Also includes: Acquire the sound pickup and simultaneously perform audio perception on the sleeping scene to obtain an audio signal; Performing recognition processing on the audio signal to obtain the vocalization features of the object in each audio frame, and constructing a vocalization feature sequence based on the vocalization features; The joint recognition processing based on the typical body surface temperature change sequence and the vital sign change sequence includes: performing joint recognition processing based on the typical body surface temperature change sequence, the vital sign change sequence and the vocal feature sequence to obtain the sleeping state prediction sequence.

5. The sleep monitoring method according to claim 1, characterized in that: The sleeping scene includes a bed; the sleeping time is the time when the subject lands in bed and falls asleep; The method further comprises: determining the implantation time of the subject based on the posture prediction sequence; The sleep onset efficiency of the subject is determined based on the subject's implantation time and the sleep onset time.

6. The sleep monitoring method according to any one of claims 1 to 5, characterized in that: Based on the posture prediction sequence and the sleeping state prediction sequence, identifying the sleeping time of the subject, comprising: determining a preliminary sleeping time of the subject based on the sleeping state prediction sequence; The preliminary sleeping time is corrected according to the posture prediction sequence to obtain a corrected sleeping time.

7. The sleep monitoring method according to any one of claims 1 to 5, characterized in that: The multiple frames of two-dimensional temperature images are subjected to recognition processing to obtain a posture prediction sequence representing changes in the body posture of the object, including: The multiple frames of two-dimensional temperature images are fused to obtain a fused temperature feature sequence; the fused temperature feature sequence includes at least one of a differential image of adjacent frames and a heat source distribution feature of each frame of the two-dimensional temperature image; Inputting the fused temperature feature sequence into a posture classification prediction model to obtain a predicted limb posture in each frame of the two-dimensional temperature image; The predicted limb postures are sorted according to the frame sorting of the two-dimensional temperature image to obtain the posture prediction sequence.

8. The sleep monitoring method according to any one of claims 1 to 5, characterized in that: The multiple frames of two-dimensional temperature images are subjected to recognition processing to obtain a posture prediction sequence representing changes in the body posture of the object, including: Performing differential processing on the multiple frames of two-dimensional temperature images and initial temperature images corresponding to an initial sleeping scene to obtain a differential image; the initial sleeping scene is a sleeping scene without an object; Performing recognition processing based on the differential image corresponding to each frame of the two-dimensional temperature image to determine the predicted limb posture of the object in each frame of the two-dimensional temperature image; The predicted limb postures are sorted according to the frame sorting of the two-dimensional temperature image to obtain the posture prediction sequence.

9. The sleep monitoring method according to any one of claims 1 to 5, characterized in that: The multiple frames of two-dimensional temperature images are subjected to recognition processing to obtain a posture prediction sequence representing changes in the body posture of the object, including: Performing preliminary recognition processing on each frame of the two-dimensional temperature image, respectively, to determine the estimated posture of the object in each frame of the two-dimensional temperature image, the corresponding confidence probability, and the pixel area of ​​each limb part; Using the estimated posture with a larger confidence probability as the predicted limb posture of the object in the corresponding two-dimensional temperature image, and using the corresponding two-dimensional temperature image as an initial reference temperature image; Calculating difference features between the reference temperature image and the adjacent two-dimensional temperature image, and determining the predicted limb posture of the adjacent two-dimensional temperature image and the pixel area of ​​each limb part based on the difference features, the pixel area of ​​each limb part of the object in the reference temperature image, and the predicted limb posture of the reference temperature image; and, The adjacent two-dimensional temperature images are used as new reference temperature images to repeatedly perform the operation of determining the difference feature, and the operation of determining the predicted limb posture of the object and the pixel area of ​​each limb part in the new reference temperature image, until the predicted limb posture of the object in all the two-dimensional temperature images is determined; The predicted limb postures are sorted according to the frame sorting of the two-dimensional temperature image to obtain the posture prediction sequence.

10. The sleep monitoring method according to any one of claims 1 to 5, characterized in that: Also includes: Processing the multiple frames of two-dimensional temperature images or spatial perception signals to estimate the number of objects in the sleep scene and the estimated position of each object in the sleep scene; Performing target area cutting on the corresponding two-dimensional temperature image based on the estimated position, and determining pixel areas representing the same object in each frame of the two-dimensional temperature image; and determining, based on the estimated position, a spatial perception signal in the spatial perception signal representing the same object; The step of performing recognition processing on the multiple frames of two-dimensional temperature images to obtain a posture prediction sequence representing changes in the body posture of the object comprises: performing recognition processing on pixel areas representing the same object in the multiple frames of two-dimensional temperature images to obtain a posture prediction sequence representing changes in the body posture of the same object; The method of jointly identifying and processing the multiple frames of two-dimensional temperature images and the spatial perception signals to obtain a sleeping state prediction sequence representing the characteristic sleeping state of the object includes: jointly identifying and processing the pixel areas of the multiple frames of two-dimensional temperature images representing the same object and the screening fluctuation signals representing the same object to obtain a sleeping state prediction sequence representing the characteristic sleeping state of the same object.

11. A sleep monitoring device, characterized in that: include: Two-dimensional temperature sensor, body motion sensing radar and processor; The two-dimensional temperature sensor is used to monitor the temperature of the sleeping scene and generate a two-dimensional temperature image, and the body motion sensing radar is used to perform spatial sensing of the sleeping scene to obtain a spatial sensing signal; The signal input end of the processor is connected to the signal output ends of the two-dimensional temperature sensor and the body motion sensing radar, and is used to determine the sleep time of the object according to the sleep monitoring method as described in any one of claims 1-10 when receiving the two-dimensional temperature image and the spatial sensing signal.

Images