Infant sleep state recognition method and device based on thermal infrared and robot

Abstract

The invention relates to the technical field of infant nursing robots, solves the problem of low infant sleep state recognition accuracy in the prior art, and provides an infant sleep state recognition method and device based on thermal infrared and a robot. The method comprises the following steps: continuously acquiring multiple frames of thermal infrared images, wherein the thermal infrared images comprise a first target area and / or a second target area of an infant; segmenting the multiple frames of thermal infrared images frame by frame to obtain time sequence data of a first target area and / or time sequence data of a second target area; detecting the position change of the time sequence data of the first target area and / or the time sequence data of the second target area of the infant, and obtaining the movement change information of the infant; and identifying the sleep state of the infant according to the motion change information. According to the invention, different sleep states of the infant can be accurately identified, the sleep state of the infant can be tracked in real time, technical support is provided for infant sleep monitoring, the infant sleep safety and comfort are improved, and intelligent nursing is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of infant care robots, and in particular relates to a method, device and robot for identifying the sleeping state of infants based on thermal infrared. Background Art

[0002] An infant care robot is an intelligent assistive device designed to help parents or guardians take care of infants and young children. It provides functions such as analyzing sleep status, allowing it to care for infants and young children in a more scientific and convenient way, reducing the burden on parents or guardians.

[0003] Infants and young children face numerous challenges in different sleep states. For example, when falling asleep, parents struggle to accurately detect sleepiness signals, leading to difficulty coaxing their children to sleep and false alarms. When awake, delayed parental responses can lead to escalating crying or uncontrolled activity (such as falling).

[0004] Therefore, intelligently identifying an infant's sleep state is crucial in infant care. However, existing infant sleep state recognition technologies generally suffer from low accuracy, severely limiting their practical application. For example, traditional technologies face numerous challenges: contact-based wearable devices can affect infants' sleep comfort and pose a risk of falling off; mattress pressure sensors struggle to detect subtle physiological changes; and non-contact microwave radar, while requiring no direct contact, is expensive and susceptible to interference. These technologies often suffer from high misjudgment rates and delayed responses in practice, failing to meet the demands of precise care.

[0005] Therefore, how to more accurately identify the sleeping state of infants and young children during the infant care process, provide technical support for infant sleep monitoring, and realize intelligent care is an urgent problem to be solved. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a method, device, and robot for identifying the sleeping state of infants based on thermal infrared, so as to solve the problem of low accuracy in identifying the sleeping state of infants in the prior art.

[0007] In a first aspect, an embodiment of the present invention provides a method for identifying the sleeping state of an infant based on thermal infrared, the method comprising:

[0008] Continuously acquiring multiple frames of thermal infrared images, wherein the thermal infrared images include the first target area and / or the second target area of ​​the infant;

[0009] Segmenting the multiple frames of thermal infrared images frame by frame to obtain time series data of the first target area and / or time series data of the second target area;

[0010] detecting position changes of the time series data of the first target area and / or the time series data of the second target area of ​​the infant to obtain movement change information of the infant;

[0011] The sleeping state of the infant is identified based on the movement change information.

[0012] Preferably, the step of segmenting the multiple frames of thermal infrared images frame by frame to obtain time series data of the first target area and / or time series data of the second target area includes:

[0013] Performing preliminary segmentation on the multiple frames of thermal infrared images according to a first temperature threshold to obtain multiple frames of preliminary segmented images, wherein the preliminary segmented images include the first target area and / or the second target area of ​​the infant;

[0014] Performing dynamic threshold segmentation on the multiple frames of preliminary segmented images according to a dynamic temperature threshold range and an iteration step, and determining a segmentation threshold that meets preset conditions by analyzing changes in connected domain features;

[0015] The multiple frames of preliminary segmented images are segmented frame by frame according to the segmentation threshold to obtain time series data of the first target area and / or time series data of the second target area.

[0016] Preferably, the multiple frames of thermal infrared images are preliminarily segmented according to the first temperature threshold to obtain multiple frames of preliminarily segmented images, wherein the preliminarily segmented images include the first target area and / or the second target area of ​​the infant, including:

[0017] Acquiring the multiple frames of thermal infrared images;

[0018] Setting a first temperature threshold, wherein the first temperature threshold is lower than the effective surface temperature of a human body;

[0019] The pixel values ​​in the multiple frames of thermal infrared images that are less than the first temperature threshold are set as background values ​​frame by frame, and the remaining pixel values ​​remain unchanged, and the multiple frames of preliminary segmented images are output, where the preliminary segmented images include the first target area and / or the second target area of ​​the infant.

[0020] Preferably, the dynamic threshold segmentation is performed on the multiple frames of preliminary segmented images based on the dynamic temperature threshold range and the iteration step size, and the segmentation threshold that meets the preset conditions is determined by analyzing the connected domain features, including:

[0021] Setting the dynamic temperature threshold range and the iteration step size;

[0022] Taking the lower limit of the dynamic temperature threshold range as the starting threshold, gradually increasing the segmentation threshold according to the iterative step size, and iteratively segmenting the multiple frames of preliminary segmented images until the upper limit of the dynamic temperature threshold range is reached;

[0023] Calculate the feature ratio of the first connected domain and the second connected domain in each iteration;

[0024] When the feature ratio reaches a preset threshold, determining the segmentation threshold;

[0025] The upper limit value is a preset body surface temperature value that is greater than an ambient temperature reference value, and the lower limit value is greater than the ambient temperature reference value.

[0026] Preferably, segmenting the preliminary segmented image frame by frame based on the segmentation threshold to obtain the time series data of the first target area and / or the time series data of the second target area includes:

[0027] Binarizing the multiple frames of preliminary segmented images frame by frame according to the segmentation threshold, removing the first target region frame by frame, and selecting the connected region with the largest area from the remaining regions as the second candidate target region;

[0028] Checking the number of adjacent pixels between the first target area and the second candidate target area frame by frame, and identifying the second candidate target area whose number of adjacent pixels is greater than a preset threshold as the second target area;

[0029] The first target area and / or the second target area of ​​each frame are integrated to obtain time series data of the first target area and / or time series data of the second target area.

[0030] Preferably, the method further comprises: dynamically adjusting the dynamic temperature threshold range:

[0031] Acquire a preset number of frames of the preliminary segmented image, extract the feature ratio between the first connected region and the second connected region frame by frame, and generate a feature ratio sequence;

[0032] Calculating the coefficient of variation of the characteristic ratio sequence;

[0033] When the coefficient of variation is greater than a preset stability threshold, adjusting the upper limit and the lower limit of the dynamic temperature threshold range according to the current characteristic ratio sequence;

[0034] The upper limit of the dynamic temperature threshold range is a preset body surface temperature value that is greater than an ambient temperature reference value, and the lower limit is greater than the ambient temperature reference value.

[0035] Preferably, detecting the position change of the time series data of the first target area and / or the time series data of the second target area of ​​the infant to obtain the movement change information of the infant includes:

[0036] Calculating the aspect ratio of the time series data of the first target area frame by frame to obtain an aspect ratio sequence of the first target area;

[0037] Calculating the aspect ratio difference between every two adjacent frames in the aspect ratio sequence of the first target area to obtain an aspect ratio difference sequence of the first target area;

[0038] and / or,

[0039] Calculating the aspect ratio of the time series data of the second target area frame by frame to obtain an aspect ratio sequence of the second target area;

[0040] calculating the aspect ratio difference between every two adjacent frames in the aspect ratio sequence of the second target area to obtain an aspect ratio difference sequence of the second target area;

[0041] The movement change information of the infant includes a sequence of aspect ratio differences of the first target area and / or a sequence of aspect ratio differences of the second target area.

[0042] Preferably, identifying the infant's sleep state based on the movement change information includes:

[0043] When the difference value of the first consecutive frames in the aspect ratio difference sequence of the first target area is less than or equal to a first preset difference threshold and shows a decreasing trend from the difference value in the second consecutive frames, the sleeping state of the infant is identified as falling asleep;

[0044] When a difference value in a third consecutive frame group in the aspect ratio difference value sequence of the first target area is equal to a second preset difference threshold, identifying the sleeping state as a deep sleep state;

[0045] When the difference in the fourth consecutive frame group in the aspect ratio difference sequence of the second target area is equal to the second preset difference threshold, and the difference in the fifth consecutive frame group in the aspect ratio difference sequence of the second target area or the aspect ratio difference sequence of the first target area is greater than the second preset difference threshold, the sleeping state is identified as the waking state.

[0046] In a second aspect, an embodiment of the present invention provides a device for identifying infant sleep status based on thermal infrared, the device comprising:

[0047] a thermal infrared image acquisition module, configured to continuously acquire multiple frames of thermal infrared images, wherein the thermal infrared images include the first target area and / or the second target area of ​​the infant;

[0048] a segmentation module, configured to segment the plurality of thermal infrared images frame by frame to obtain time series data of the first target area and / or time series data of the second target area;

[0049] a detection module, configured to detect position changes in the time series data of the first target area and / or the time series data of the second target area of ​​the infant, and obtain movement change information of the infant;

[0050] The sleeping state recognition module is used to recognize the sleeping state of the infant based on the movement change information.

[0051] In a third aspect, an embodiment of the present invention provides an infant care robot, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method of the first aspect of the above embodiment.

[0052] In a fourth aspect, an embodiment of the present invention provides a storage medium having computer program instructions stored thereon, which implements the method of the first aspect of the above-mentioned embodiment when the computer program instructions are executed by a processor.

[0053] In summary, the beneficial effects of the present invention are as follows:

[0054] The present invention provides a method, device and robot for identifying the sleeping state of infants and young children based on thermal infrared. The method continuously acquires multiple frames of thermal infrared images, where the thermal infrared images include a first target area and / or a second target area of ​​the infant; the multiple frames of thermal infrared images are segmented frame by frame to obtain time series data of the first target area and / or the time series data of the second target area; the position changes of the time series data of the first target area and / or the time series data of the second target area of ​​the infant are detected to obtain movement change information of the infant; the sleeping state of the infant is identified based on the movement change information, and the different sleeping states of the infant can be accurately identified, the sleeping state of the infant can be tracked in real time, technical support is provided for infant sleep monitoring, the safety and comfort of the infant's sleep are improved, and intelligent care is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work, and these are all within the scope of protection of the present invention.

[0056] Figure 1 This is a schematic diagram of the overall process of a method for identifying infant sleep status based on thermal infrared provided by the present invention;

[0057] Figure 2 This is a specific flow chart of step S1 of a method for identifying infant sleep status based on thermal infrared provided by the present invention;

[0058] Figure 3 This is a schematic diagram of a specific flow of step S2 of a method for identifying infant sleep status based on thermal infrared provided by the present invention;

[0059] Figure 4This is a schematic diagram of a specific flow of step S21 of a method for identifying infant sleep status based on thermal infrared provided by the present invention;

[0060] Figure 5 This is a schematic diagram of a specific flow of step S22 of a method for identifying infant sleep status based on thermal infrared provided by the present invention;

[0061] Figure 6 This is a schematic diagram of a specific flow of step S23 of a method for identifying infant sleep status based on thermal infrared provided by the present invention;

[0062] Figure 7 This is a specific flow chart of step S3 of a method for identifying infant sleep status based on thermal infrared provided by the present invention;

[0063] Figure 8 This is a schematic diagram of a specific flow of step S4 of a method for identifying infant sleep status based on thermal infrared provided by the present invention;

[0064] Figure 9 This is a schematic structural diagram of a device for identifying infant sleep status based on thermal infrared provided by an embodiment of the present invention;

[0065] Figure 10 The figure is a schematic structural diagram of an infant care robot provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In order to make the objects, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only configured to explain the present invention and are not configured to limit the present invention. For those skilled in the art, the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present invention by illustrating examples of the present invention.

[0067] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, the elements defined by the phrase "comprising..." do not exclude the presence of other identical elements in the process, method, article, or device comprising the elements.

[0068] It should be noted that all actions of acquiring signals, information or data in the present invention are performed in compliance with the corresponding local data protection laws and policies and with authorization from the corresponding device owner.

[0069] First, see Figures 1 to 8 The embodiment of the present invention provides a method for identifying the sleeping state of infants and young children based on thermal infrared, the method comprising:

[0070] S1. Continuously acquiring multiple frames of thermal infrared images, wherein the thermal infrared images include a first target area and / or a second target area of ​​the infant;

[0071] Thermal infrared imaging uses thermal infrared imaging technology to capture and display the temperature distribution of an object's surface. Thermal infrared images are not based on visible light reflected from an object, but rather on the object's own thermal radiation. All objects (as long as their temperature is above absolute zero) radiate infrared light, which the thermal infrared imaging module detects and converts into a thermal image.

[0072] Thermal infrared imaging is a non-contact monitoring method. It captures target images of infants and young children in a dark environment and combines them with computer vision technology to analyze their sleep patterns. Thermal infrared imaging overcomes the shortcomings of contact methods and can also monitor infant sleep in dark environments. Thermal infrared imaging does not require contact with the infant's skin or the use of other sensors, thus avoiding discomfort and reducing the risk of infection.

[0073] In one embodiment, the thermal infrared imaging module can be implemented using a low-resolution thermal infrared module. The use of a low-resolution thermal infrared module has the advantages of low cost, low power consumption, high frame rate and real-time performance, privacy protection, and miniaturization, and can collect and process data in real time. Among them, the low cost is reflected in the fact that the price of a low-resolution thermal infrared module is lower than that of a high-resolution thermal infrared module. The low power consumption is reflected in the fact that the computing resource requirements of the low-resolution thermal infrared module are low, making it suitable for battery-powered portable devices or long-term continuous monitoring scenarios. The high frame rate and real-time performance are reflected in the fast processing speed of small-resolution images, which can achieve higher frame rates and is suitable for detecting real-time changes such as infant body movements in dynamic environments. The privacy protection is reflected in the fact that small-resolution images cannot restore clear human faces or environmental details, which is suitable for privacy-sensitive scenarios such as home care. The miniaturization and ease of integration are reflected in the small size of the module, which can be embedded in a crib, wearable device, or mobile terminal without the need for complex optical components.

[0074] In an optional embodiment, the resolution of the low-resolution thermal infrared module can be 82x60 and the frame rate can be 15 frames per second. Of course, the resolution of the low-resolution thermal infrared module can also be other resolutions such as 80x60, which is not limited here.

[0075] It can be understood that each frame of the continuously acquired thermal infrared image includes the infant in the monitored environment, and the thermal infrared image further includes the first target area and / or the second target area of ​​the infant that can be used for sleep state identification.

[0076] In one embodiment, the first target area may be one of the torso area and the head area, and the second target area may be the other of the torso area and the head area. The torso area refers to the main body part of the human body excluding the head and limbs, generally including the chest, abdomen, and back areas.

[0077] In an exemplary embodiment, a method for identifying infant sleep status based on thermal infrared uses a thermal infrared imaging module such as a thermal infrared camera or a small-resolution thermal infrared module to continuously acquire multiple frames of thermal infrared images at a set sampling frequency, wherein the thermal infrared images include the infant's body area such as the torso area and / or the head area.

[0078] It is worth mentioning that the continuous acquisition of multiple frames of thermal infrared images can be the continuous acquisition of multiple frames of thermal infrared images within a preset time. Specifically, the acquisition of multiple frames of thermal infrared images within the preset time is set to segment and detect the movement changes of infants and young children, which can avoid the problem of insufficient or excessive frames of thermal infrared images. Of course, it is also possible not to set a preset time, but to limit the number of frames of thermal infrared images to be acquired, for example, 10 frames, which is not limited here.

[0079] In one embodiment, if Figure 2As shown, the continuous acquisition of multiple frames of thermal infrared images in step S1 may also include a preprocessing method to ensure the accuracy of subsequent segmentation, specifically including:

[0080] S11, continuously acquiring multiple frames of initial thermal infrared images, where the initial thermal infrared images include the first target area and / or the second target area of ​​the infant;

[0081] It can be understood that the initial thermal infrared image is a thermal infrared image that has not been preprocessed, and the initial thermal infrared image includes at least one of the first target area and the second target area of ​​the infant.

[0082] S12: Preprocess the multiple frames of initial thermal infrared images to obtain the multiple frames of thermal infrared images.

[0083] It can be understood that the thermal infrared image includes at least one of the first target area and the second target area of ​​the infant.

[0084] In one embodiment, a method of pre-processing the multiple frames of initial thermal infrared images includes but is not limited to at least one of consistency setting and normalization.

[0085] In an exemplary embodiment, the method adopts a method of setting consistency of the thermal infrared image sequence. In this case, step S12 specifically includes:

[0086] S121. performing consistency setting on the multiple frames of initial thermal infrared images frame by frame according to a temperature threshold interval to obtain the multiple frames of thermal infrared images;

[0087] It's important to note that, compared to other preprocessing methods, using consistency settings for initial thermal infrared images provides a stable and reliable foundation for subsequent image segmentation and identification of infants' sleep states, while also improving processing efficiency. Compared to computationally complex methods like dynamic ROI extraction or motion correction, consistency settings can improve data quality overall at a lower computational cost while ensuring the integrity of key information.

[0088] In one embodiment, the step S121 adopts a consistency setting method to pre-process the multiple frames of initial thermal infrared images frame by frame to obtain the multiple frames of thermal infrared images, which may include:

[0089] S1211 a. Scan the multiple frames of initial thermal infrared images frame by frame to obtain temperature distribution information of each frame of the initial thermal infrared image;

[0090] The initial thermal infrared image includes temperature information. In this embodiment, the initial thermal infrared image obtained includes the temperature of various parts of the infant's body. That is, the thermal infrared image can clearly present the temperature distribution of various regions of the infant's body. Furthermore, the body regions include, but are not limited to, a first target region and a second target region. For example, the first target region is the head region, while the second target region is the torso region.

[0091] Scanning the multiple frames of initial thermal infrared images frame by frame can specifically include scanning all pixel values ​​in the multiple frames of initial thermal infrared images frame by frame to identify the maximum temperature value and the minimum temperature value of each frame of the initial thermal infrared image. After identifying the maximum temperature and the minimum temperature of each frame of the initial thermal infrared image, the maximum temperature and the minimum temperature of each frame of the initial thermal infrared image are counted to obtain the temperature distribution of each frame of the initial thermal infrared image, i.e., the temperature distribution information.

[0092] S1212a, determining a temperature threshold interval according to the temperature distribution information;

[0093] After obtaining the temperature distribution of each frame of the initial thermal infrared image according to the above steps, the temperature range where most of the initial thermal infrared images are located is used as the temperature threshold interval, that is, the mapping temperature range.

[0094] In an exemplary embodiment, after obtaining the maximum temperature and the minimum temperature of each frame of the initial thermal infrared image in step S1211 a, the distribution range of the maximum temperature and the minimum temperature of each frame of the initial thermal infrared image is counted. If most of the initial thermal infrared images are within a certain temperature range, then the temperature range is used as the temperature threshold interval. For example, if most of the initial thermal infrared images are within the range of 18°C ​​to 43°C, then 18°C ​​to 43°C is used as the temperature threshold interval.

[0095] In addition, the temperature threshold interval can also be set according to actual needs and the temperature range. For example, the temperature range of most thermal infrared images can be adjusted up or down according to actual needs to obtain the temperature threshold interval, which is not limited here.

[0096] S1213a. Based on the temperature threshold interval, normalize the multiple frames of initial thermal infrared images frame by frame to obtain the multiple frames of thermal infrared images.

[0097] Specifically, when the temperature threshold interval is 18° C. to 43° C., the initial thermal infrared image is normalized frame by frame based on 18° C. to 43° C. to obtain multiple frames of thermal infrared images.

[0098] It should be noted that applying the normalization processing method to the method for identifying the sleeping state of infants and young children can avoid misjudgment caused by changes in external environmental temperature and improve the accuracy of identifying the sleeping state of infants and young children.

[0099] In addition, the above-mentioned normalization processing of the temperature range (such as 18°C ​​to 43°C) in which most of the initial thermal infrared images are located has the advantage of being the temperature threshold interval: by dynamically statistically obtaining the temperature distribution of multiple frames of initial thermal infrared images, extreme outliers (such as occasional low-temperature noise or short-term high-temperature interference) are automatically excluded, so that the normalization processing is focused on the effective temperature threshold interval, thereby significantly improving the utilization rate of the continuously obtained multiple frames of initial thermal infrared image data.

[0100] In another embodiment, the step S121 adopts a consistency setting method to pre-process the multiple frames of initial thermal infrared images frame by frame to obtain the multiple frames of thermal infrared images, which may include:

[0101] S1211 b. Setting a temperature threshold range;

[0102] The temperature threshold range consists of multiple temperature values, including a lower limit and an upper limit, i.e., the minimum and maximum temperatures. The maximum and minimum temperatures can be selected based on the physiological characteristics of infants and children and sensor performance optimization to ensure the capture of key physiological signals while effectively suppressing environmental noise.

[0103] In an optional embodiment, a temperature threshold interval can be pre-set based on the physiological characteristics of infants and young children. That is, the temperature threshold area in this case is a preset temperature threshold interval, and the maximum temperature of the temperature threshold interval can be set to 43°C, and the minimum temperature can be set to 18°C. Selecting 18°C ​​to 43°C as the temperature threshold interval has the following advantages: this temperature threshold interval can cover the effective physiological temperature of the human body, such as the surface temperature characteristics of infants and young children: the normal surface temperature range is 32°C-38°C (head / torso), and the temperature of the extremities is 25°C-32°C (hands and feet); the lower limit of 18°C ​​can filter out environmental cold noise (such as cold air from air conditioners), and the upper limit of 43°C can exclude abnormal fever (such as fever or local inflammation).

[0104] In other implementations, other minimum temperature values ​​and maximum temperatures may be set as the temperature threshold intervals according to the needs of the scenarios in actual applications, and are not limited here.

[0105] S1212b, screening the multiple frames of the initial thermal infrared images based on the temperature threshold interval and the ambient temperature to obtain multiple frames of valid initial thermal infrared images;

[0106] Specifically, metadata can be obtained for a thermal infrared image sequence. This metadata includes at least the ambient temperature corresponding to each initial thermal infrared image frame, and may also include the acquisition timestamp of each frame, the device's temperature measurement range, and other information. The ambient temperature can be obtained by reading the average temperature of a 5% pixel area around the edge of the image, or by directly recording it using the ambient temperature sensor built into the thermal infrared imaging module. The multiple frames of initial thermal infrared images are then filtered based on the ambient temperature from this metadata, and initial thermal infrared images with ambient temperatures exceeding the specified temperature threshold are eliminated to obtain multiple frames of valid thermal infrared images.

[0107] S1213b. Perform consistency processing on the multiple frames of valid initial thermal infrared images to obtain the multiple frames of thermal infrared images.

[0108] Consistency processing includes temporal alignment, spatial normalization, and metadata tagging. After obtaining multiple frames of valid initial thermal infrared images in step S1212b, each frame of the multiple frames can be processed using at least one of temporal alignment, spatial normalization, and metadata tagging to obtain multiple frames of thermal infrared images. The coordinated use of at least one of temporal alignment, spatial normalization, and metadata tagging can improve the accuracy of subsequent thermal infrared image segmentation and infant sleep state identification. Application of this method to infant sleep state identification can also enable intelligent infant care.

[0109] It should be noted that the above-mentioned method automatically adjusts and determines the temperature threshold range through real-time statistical temperature distribution, and then constrains it to the temperature threshold range and normalizes it. It has strong environmental adaptability and can dynamically relax or tighten the threshold according to the actual scenario to avoid data loss due to drastic environmental changes. At the same time, it retains the relative change trend of out-of-limit data, which is suitable for scenarios such as infant care; and the consistency processing method of directly filtering valid frames according to the preset temperature threshold range and then performing normalization has extremely high execution efficiency, and only simple comparison operations are required to complete the screening.

[0110] S2, segmenting the multiple frames of thermal infrared images frame by frame to obtain time series data of the first target area and / or time series data of the second target area, including:

[0111] Specifically, segmentation includes at least one of static segmentation and dynamic segmentation. Static segmentation relies solely on a single threshold, which cannot adapt to the complex distribution of human body temperature and results in poor segmentation results. Dynamic segmentation combines static and dynamic thresholds. By dynamically adjusting the threshold, it can better identify infants' temperature changes and improve segmentation accuracy, especially in terms of edge details and contour processing. It can resist noise and unnecessary details in the image, reduce missegments through dynamic thresholding, maintain the integrity of the infant's body area, and perform fine-grained segmentation of each body area, which can improve the accuracy of infant sleep state identification.

[0112] In one embodiment, the segmentation may be performed in a dynamic manner, such as Figure 3 As shown, step S2 specifically includes the following sub-steps:

[0113] S21, performing preliminary segmentation on the multiple frames of thermal infrared images according to a first temperature threshold to obtain multiple frames of preliminary segmented images, wherein the preliminary segmented images include the first target area and / or the second target area of ​​the infant;

[0114] In one achievable manner, Figure 4 As shown, step S21 performs preliminary segmentation of the multiple frames of thermal infrared images according to the first temperature threshold to obtain multiple frames of preliminary segmented images, wherein the preliminary segmented images include the first target area and / or the second target area of ​​the infant, and specifically may include:

[0115] S211, acquiring the multiple frames of thermal infrared images;

[0116] S212: Setting a first temperature threshold, where the first temperature threshold is lower than the effective human body surface temperature;

[0117] Among them, the first temperature threshold can also be called the static temperature threshold (i.e., the static threshold), which can be a temperature value lower than the effective surface temperature of the human body; when the normal range of the effective surface temperature of the human body is 32-35°C, the first temperature threshold is less than 32°C. Further, a temperature value suitable for infants and young children to sleep and lower than the effective surface temperature of the human body can be selected as the first temperature threshold. For example, 26°C is a temperature value suitable for infants and young children to sleep and lower than the effective surface temperature of the human body. In a feasible manner, the first temperature threshold can be set to 26°C.

[0118] S213. Set the pixel values ​​in the multiple frames of thermal infrared images that are less than the first temperature threshold as the background value frame by frame, and keep the other pixel values ​​unchanged, and output the multiple frames of preliminary segmented images, wherein the preliminary segmented images include the first target area and / or the second target area of ​​the infant.

[0119] This method can effectively separate the target area of ​​the infant from the background by setting a first temperature threshold that is lower than the effective body surface temperature of the infant but suitable for the infant's sleep, while retaining the integrity of the original data information. It can provide a basis for subsequent position change detection of the infant, improve the accuracy of detection, and thus improve the accuracy of identifying the infant's sleep state.

[0120] S22, performing dynamic threshold segmentation on the multiple frames of preliminary segmented images based on a dynamic temperature threshold range and an iteration step size, and determining a segmentation threshold that meets preset conditions by analyzing connected domain features;

[0121] In one embodiment, if Figure 5 As shown, step S32 may specifically include:

[0122] S221, setting the dynamic temperature threshold range and the iteration step size;

[0123] The dynamic temperature threshold range includes an upper limit and a lower limit, which is composed of temperature values ​​from the lower limit to the upper limit and can be an integer. The upper limit is a preset body surface temperature value greater than the ambient temperature reference value, and the lower limit is greater than the ambient temperature reference value.

[0124] In one embodiment, the preset body surface temperature value may be less than or equal to the highest effective temperature of the human body surface temperature but greater than the ambient temperature reference value. In addition, the lower limit value may be further less than the lowest effective temperature of the human body surface temperature.

[0125] It should be noted that the maximum effective temperature and the minimum effective temperature correspond to the highest temperature value and the lowest temperature value within the effective surface temperature range of the human body. For example, the effective surface temperature of the human body is 32-35°C, the maximum effective temperature is 35°C, and the minimum effective temperature is 32°C.

[0126] In an exemplary embodiment, the effective surface temperature of the human body is 32-35°C, the ambient temperature reference value is 26°C, the upper limit of the dynamic temperature threshold range can be set to 35°C, and the lower limit can be set to 27°C, then the dynamic temperature threshold range is 27-35°C.

[0127] It should be noted that when the first target area of ​​an infant is the head area, the second target area is the torso area. The temperature of the torso area is generally lower than that of the head area, but higher than the ambient temperature. Therefore, the lower limit of the dynamic temperature threshold range is set to be greater than the ambient temperature for segmentation, and the upper limit is set to be less than the highest effective temperature of the human body surface temperature. This can effectively distinguish the environmental background and the infant's body area, and then segment the infant's body area.

[0128] The lower limit value is lower than the maximum effective temperature of the first target area, the lower limit value is higher than the ambient temperature reference value but lower than the minimum effective temperature of the second target area, the first target area is the head area, and the second target area is the torso area.

[0129] The iteration step size is the amount by which the variable is updated at each iteration. This can be optimized and determined in advance through training or statistical analysis of a large amount of thermal infrared image data. Preset temperature increments obtained through training methods can achieve higher segmentation accuracy than those set based on experience. For example, if the iteration step size obtained through training is 0.1°C, the iteration step size can be set to 0.1°C.

[0130] S222, using the lower limit of the dynamic temperature threshold range as the starting threshold, gradually increasing the segmentation threshold according to the iteration step, and iteratively segmenting the multiple frames of preliminary segmented images until the upper limit of the dynamic temperature threshold range is reached;

[0131] Among them, the starting threshold can be understood as the step of starting to segment multiple frames of preliminary segmented images, that is, the first step. For example, the lower limit value is 27°C, the upper limit value is 35°C, and the iteration step is 0.1°C. Then, 27°C is used as the starting threshold to segment the multiple frames of preliminary segmented images, and the segmentation threshold is gradually increased by 0.1°C. For example, the second time is 27+0.1°C, and the current segmentation threshold of 27+0.1°C is used to segment the multiple frames of preliminary segmented images until it reaches 35°C.

[0132] S223, calculating the feature ratio of the first connected domain to the second connected domain in each iteration;

[0133] In one embodiment, when the first target area is the head area and the second target area is the torso area, the first connected domain may be the second target area and the second connected domain may be the first target area.

[0134] In one achievable embodiment, the first connected domain may be the largest connected domain, and the second connected domain may be the second largest connected domain. Specifically, the feature ratio may be an area ratio, and in this embodiment, it is the area ratio of the largest connected domain to the second largest connected domain. Connected domains include contours. Specifically, in each iteration, the number of contours in the image may be counted to determine the largest contour and the second largest contour, and then the feature ratio of the largest contour to the second largest contour may be calculated. Further, the area ratio of the largest contour to the second largest contour may be calculated.

[0135] When the largest connected domain is the trunk region of an infant and the second largest connected domain is the head region of the infant, the characteristic ratio of the largest connected domain to the second largest connected domain is the area ratio of the trunk region to the head region of the infant.

[0136] It should be noted that the characteristic ratio of the maximum connected domain and the second largest connected domain is calculated by: calculating the characteristic ratio of the maximum connected domain and the second largest connected domain of all multiple frames of thermal infrared images for each segmentation threshold; or calculating the characteristic ratio of the maximum connected domain and the second largest connected domain of each frame of thermal infrared image for each segmentation threshold.

[0137] In one feasible method, it specifically includes: calculating the total area of ​​the torso region of all multiple frames of thermal infrared images for each segmentation threshold, then calculating the total area of ​​the head region, and then calculating the ratio of the two total contour areas; or, calculating the area of ​​the torso region of each frame of thermal infrared image for each segmentation threshold and the area of ​​the head region of the corresponding frame, and then calculating the area ratio of the torso region and the head region contours to obtain the area ratio of the torso region and the head region of each frame.

[0138] The extraction of the contour area of ​​the region and calculation of the area ratio of each dynamic temperature threshold (i.e., segmentation threshold) may include a contour extraction step: binarizing the multiple frames of preliminary segmented images using the current dynamic temperature threshold to obtain multiple frames of binary images; performing contour extraction on the multiple frames of binary images to determine the largest connected domain and the second largest connected domain; and calculating the areas of the largest connected domain and the second largest connected domain, as well as the area ratio of the largest connected domain to the second largest connected domain. Assuming that S1 represents the area of ​​the largest connected domain and S2 represents the area of ​​the second largest connected domain, the area ratio of the largest connected domain to the second largest connected domain is S1 / S2.

[0139] S224, when the feature ratio reaches a preset threshold, determining the segmentation threshold;

[0140] The preset threshold value may be the minimum value of the feature ratio, and the feature ratio may specifically be the area ratio, when the area ratio is the minimum.

[0141] It should be noted that the determined segmentation threshold may be an optimal segmentation threshold, which is determined according to the actual scenario.

[0142] It is worth mentioning that the determined segmentation threshold can be the current segmentation threshold, that is, the current dynamic temperature threshold, the optimal segmentation threshold for segmenting multiple frames of preliminary segmented images, that is, all frames share the same segmentation threshold; it can also be the segmentation threshold that meets the preset conditions for each frame of the preliminary segmented image in the current segmentation threshold, that is, each frame calculates its own segmentation threshold.

[0143] It should be noted that all frames share the same segmentation threshold and only one threshold calculation is needed to process all frames. The segmentation efficiency is fast and the fluctuation between frames is small. Each frame corresponds to an optimal segmentation threshold, which can segment the preliminary segmented image of each frame and improve environmental robustness.

[0144] It is worth mentioning that the method of the present invention may further include the step of dynamically adjusting the dynamic temperature threshold range, which may specifically include:

[0145] Acquire a preset number of frames of the preliminary segmented image, extract the feature ratio between the first connected region and the second connected region frame by frame, and generate a feature ratio sequence;

[0146] In a preferred embodiment, the first connected domain is the largest connected domain, and the second connected domain is the second largest connected domain.

[0147] In one embodiment, the preliminary segmented image is an image obtained after preliminary segmentation of the original thermal infrared image based on the first temperature threshold. The number of frames can be set according to the actual acquisition frame rate and recognition accuracy requirements, for example, it can be set to 10 consecutive frames or 20 frames; for each frame of the preliminary segmented image of the preset number of frames, the areas of the largest connected domain and the second largest connected domain are extracted respectively, and their area ratio is calculated to form a set of feature ratio sequences, wherein the feature ratio can specifically be an area feature ratio, and the connected domain can be a contour.

[0148] Calculating the coefficient of variation of the characteristic ratio sequence;

[0149] The coefficient of variation may be used to characterize the stability of the characteristic ratio sequence within the frame segment.

[0150] When the coefficient of variation is greater than a preset stability threshold, the upper limit and lower limit of the dynamic temperature threshold range are adjusted according to the current characteristic ratio sequence; wherein the upper limit of the dynamic temperature threshold range is a preset body surface temperature value that is greater than the ambient temperature reference value, and further, the upper limit may be less than or equal to the highest effective temperature of the human body's effective body surface temperature and greater than the ambient temperature reference value; the lower limit is greater than the ambient temperature reference value.

[0151] Specifically, it is determined whether the coefficient of variation is greater than a preset stability threshold. If so, it is considered that the current ambient temperature or image conditions have fluctuated, and the target area extraction is no longer stable, then the dynamic temperature threshold adjustment operation is triggered, and the upper and lower limits of the dynamic temperature threshold range are adjusted according to the current feature ratio sequence, so that the subsequent connected domain segmentation is more in line with the current environmental conditions.

[0152] Among them, the preset stability threshold can be obtained by statistical or training analysis based on a large amount of thermal infrared image data collected under different environmental conditions by calculating the coefficient of variation of the feature ratio sequence of the largest connected domain and the second largest connected domain.

[0153] Furthermore, the adjustment of the upper limit and the lower limit of the dynamic temperature threshold range may also include:

[0154] A new starting threshold is set between the upper limit and the lower limit, and the preliminary segmented image is iteratively segmented according to the iteration step; the feature ratio of the maximum connected domain and the second largest connected domain is recalculated in each iteration, and when the ratio reaches a minimum value, the current threshold is determined to be the updated optimal segmentation threshold.

[0155] This method uses a time-series variability assessment mechanism based on feature ratios (e.g., area ratios) of connected domains. This feature ratio sequence reflects the differences in thermal signatures between target regions (e.g., the infant's body) and non-target regions (e.g., the background or bed surface) in thermal infrared images. When the temperature distribution in the target region changes due to factors such as the external environment, obstruction by bedding, and changes in body position, the upper and lower temperature thresholds are adjusted based on the feature ratio sequence, enabling rapid response to such changes and ensuring that segmentation remains focused on the true and valid target region.

[0156] S23 . Segment the multiple frames of preliminary segmented images frame by frame based on the segmentation threshold to obtain time series data of the first target area and / or time series data of the second target area.

[0157] In one embodiment, after obtaining the segmentation threshold (such as the optimal segmentation threshold) through the above steps, the multiple frames of preliminary segmented images are segmented frame by frame according to the optimal segmentation threshold to obtain the time series data of the first target area and / or the time series data of the second target area.

[0158] Here, time series data refers to time series data, which is a type of data that records in chronological order and reflects the changes of phenomena over time. In this embodiment, it refers to the time series data of the first target region and / or the time series data of the second target region obtained by segmentation based on the optimal segmentation threshold from the preliminary segmented images of multiple consecutive frames. When the first target region is the head region and the second target region is the torso region, it refers to the time series data of the head region and / or the time series data of the torso region obtained by segmentation based on the optimal segmentation threshold from the preliminary segmented images of multiple consecutive frames.

[0159] It should be noted that, in order to ensure the accuracy of the segmented data and improve the accuracy of identifying the sleeping state of infants and young children, in step S22, the dynamic threshold segmentation of the multiple frames of preliminary segmented images is performed based on the dynamic temperature threshold range and the iteration step, and / or in step S33, the multiple frames of preliminary segmented images are segmented frame by frame based on the optimal segmentation threshold, which may include the steps of confirming the segmented areas and verifying their connectivity. Taking step S23 including the area (such as the second target area of ​​the torso area) and connectivity verification as an example, as shown in FIG. Figure 6 As shown, step S23 may include:

[0160] S231, binarizing the multiple frames of preliminary segmented images frame by frame according to the segmentation threshold, removing the first target region frame by frame, and selecting a connected region that meets a preset area threshold from the remaining regions as a second candidate target region;

[0161] In one embodiment, the connected domain that meets a preset area threshold may be the connected domain with the largest area. Specifically, the first target region is the head region, and the second target region is the torso region. After binarizing the multiple frames of the preliminary segmented images frame by frame based on the optimal segmentation threshold, the head region of each frame is identified and removed. Then, the connected domain (or hotspot) with the largest area in the remaining area of ​​each frame is selected as the candidate torso region. The connected domain may be a contour. If it is a contour, the contour with the largest area in the remaining area is selected frame by frame as the candidate torso region of the infant or child.

[0162] S232, checking the number of adjacent pixels between the first target region and the second candidate target region frame by frame, and identifying the second candidate target region whose number of adjacent pixels is greater than a preset threshold as the second target region;

[0163] The preset threshold refers to a threshold of the preset number of adjacent pixels.

[0164] Specifically, after obtaining the second target candidate area of ​​each frame, such as the torso candidate area, the number of adjacent pixels of the head area and the torso candidate area of ​​each frame image is checked frame by frame, and the torso candidate area with the number of adjacent pixels greater than a preset threshold is confirmed as the torso area of ​​the infant.

[0165] S233 , integrating the first target area and / or the second target area of ​​each frame to obtain time series data of the first target area and / or the time series data of the second target area.

[0166] After the second target region such as the torso region is confirmed, the first target region which is the head region and / or the torso region in each frame image is integrated together to obtain time series data of the head region and / or time series data of the torso region.

[0167] S3. Detecting position changes in the time series data of the first target area and / or the time series data of the second target area of ​​the infant to obtain movement change information of the infant;

[0168] In one embodiment, the motion change information includes a sequence of aspect ratio differences of the first target region, such as Figure 7 As shown, at this time step S3 may include:

[0169] S31, calculating the aspect ratio of the time series data of the first target area frame by frame to obtain an aspect ratio sequence of the first target area;

[0170] S32 : Calculate the aspect ratio difference between every two adjacent frames in the aspect ratio sequence of the first target region to obtain an aspect ratio difference sequence of the first target region.

[0171] In another embodiment, the motion change information may further include a sequence of aspect ratio differences of the second target region. In this case, step S4 may further include:

[0172] S33, calculating the aspect ratio of the time series data of the second target area frame by frame to obtain an aspect ratio sequence of the second target area;

[0173] S34, calculating the aspect ratio difference between every two adjacent frames in the aspect ratio sequence of the second target area to obtain an aspect ratio difference sequence of the second target area;

[0174] When the first target area is the head area of ​​the infant and the second target area is the torso area of ​​the infant, the infant's motion change information includes at least one of the aspect ratio difference sequence of the head area and the aspect ratio difference sequence of the torso area.

[0175] In one conceivable implementation, micro-motion detection can be used to detect positional changes in the time series data of the first target area and / or the time series data of the second target area of ​​the infant to obtain information about the infant's movement changes. Micro-motion detection of a series of body regions captures submillimeter body movements, enabling precise identification of the infant's sleep state and improving sleep status recognition accuracy. It can also monitor respiratory rate and abnormal movements (such as the startle reflex) in real time, reducing the false alarm rate compared to static detection, thus achieving truly contactless, high-precision, and always-reliable intelligent care.

[0176] S4. Identify the infant's sleep state based on the movement change information.

[0177] In one embodiment, the infant's sleep state includes at least one of a falling asleep state, a deep sleeping state, and an awake state. The infant's motion change information includes at least one of a length-to-width ratio difference sequence of a first target area and a length-to-width ratio difference sequence of a second target area. Figure 8 As shown, step S4 may specifically include:

[0178] S41: when the difference value of the first consecutive frames in the aspect ratio difference sequence of the first target area is less than or equal to a first preset difference threshold and shows a decreasing trend from the difference value in the second consecutive frames, identifying the infant's sleeping state as falling asleep;

[0179] In an exemplary embodiment, the first target area is the head area, and the second target area is the torso area. The first preset difference threshold is a preset difference in aspect ratio between two adjacent consecutive frames. The first preset difference threshold can be set to approximately 0 to indicate that the infant's head is essentially motionless. The difference in the first consecutive frame number can be understood as the difference in aspect ratio between the head area of ​​one of the two adjacent consecutive frames. The second consecutive frame number is continuous with the first consecutive frame number, and can be understood as the second consecutive frame number being the first consecutive frame number after the first consecutive frame number. Assuming that there are head areas of ABC frames that are sequentially adjacent in time order, the first consecutive frame number is AB, and the second consecutive frame number is BC. The difference in the first consecutive frame number is the difference between the aspect ratio sequence of the head area A and the aspect ratio sequence of the head area B, and the difference in the second consecutive frame number is the difference between the aspect ratio sequence of the head area B and the aspect ratio sequence of the head area C. When the difference between the first consecutive frame number AB is less than or equal to the first preset difference threshold, and the difference with the second consecutive frame number BC shows a decreasing trend, that is, the difference between AB is greater than the difference between BC, it is recognized that the infant's head is basically not moving at this time and the sleeping state is falling asleep.

[0180] Of course, in other embodiments, it is also possible to determine whether an infant is in a sleeping state by setting the difference in the aspect ratio of the head area of ​​other frames to be detected. For example, the difference in the aspect ratio of the head area of ​​three consecutive frames or four consecutive frames or the like is detected to determine whether the infant is in a sleeping state. The difference in the aspect ratio of multiple consecutive frames may show a decreasing change, or may show a trend of first decreasing and then remaining unchanged, which is not limited here.

[0181] S42: when the difference in the third consecutive frame group in the aspect ratio difference sequence of the first target area is equal to a second preset difference threshold, identifying the sleeping state as a deep sleep state;

[0182] In an exemplary embodiment, the first target area is the head area, and the second target area is the torso area. The second preset difference threshold may be 0, indicating that the infant's head is not moving and the infant is in a deep sleep state.

[0183] It should be noted that, in this embodiment, the third consecutive frame group herein can be defined as a frame group comprising a first consecutive frame number, a second consecutive frame number, and a third consecutive frame number; wherein the third consecutive frame number can be understood as the frame number following the first consecutive frame number and the second consecutive frame number. For example, assuming that the first consecutive frame number is AB and the second consecutive frame number is BC, the third consecutive frame number includes AB, BC, and CD, where ABCD are the head regions of adjacent frames sorted in time. When the difference values ​​of the third consecutive frame group in the head region aspect ratio difference sequence, such as the difference values ​​of the AB, BC, and CD head regions, are equal to 0, that is, when the difference value of the head region is 0 for a long period of time, the infant is identified as being in a deep sleep state.

[0184] Of course, you can set the detection of more continuous frames to determine whether the infant is in a deep sleep state. The specific settings and selections are based on the needs of the actual scene and are not restricted here.

[0185] In one embodiment, the total number of frames of the third continuous frame group may be 15*60*3=2700 frames, and of course it may be other frame values, which are not limited here.

[0186] It should be noted that the third continuous frame group here may also be defined in other embodiments as including not only the first continuous frame number and the second continuous frame number, but also the next multiple continuous frames, which is not limited here.

[0187] S43. When the difference in the fourth consecutive frame group in the aspect ratio difference sequence of the second target area is equal to the second preset difference threshold, and the difference in the fifth consecutive frame group in the aspect ratio difference sequence of the second target area or the aspect ratio difference sequence of the first target area is greater than the second preset difference threshold, the sleeping state is identified as the awake state.

[0188] The fourth continuous frame group in this embodiment may refer to a frame group including four continuous frames, and the fifth continuous frame group refers to a frame group including five continuous frames, which in this embodiment is specifically the next five continuous frames after the four continuous frames. The second preset difference threshold is 0.

[0189] In an exemplary embodiment, the first target area is the head area, and the second target area is the torso area. If the fourth consecutive frame group is the torso area or head area of ​​the four frames AB, BC, CD, and DF of the adjacent consecutive frames ABCDF, and the fifth consecutive frame group is the torso area or head area of ​​the five frames FG, GH, HI, IJ, and JK of the adjacent consecutive frames FGHIJK, then the aspect ratio difference in the fourth consecutive frame group of the torso area aspect ratio difference sequence is equal to the second preset difference threshold, specifically: the aspect ratio difference of the torso area of ​​the four frames AB, BC, CD, and DF is equal to 0, and the difference in the fifth consecutive frame group of the torso area aspect ratio difference sequence or the head area aspect ratio difference sequence is greater than the second preset difference threshold, specifically: the difference of the torso area or head area of ​​the last five consecutive frames FG, GH, HI, IJ, and JK is greater than 0, and both conditions are met, then the infant's sleeping state is identified as the awake state.

[0190] In one implementable manner, the total number of frames in the fourth consecutive frame group may be 2700 frames, and the total number of frames in the fifth consecutive frame group may be 900 frames.

[0191] The present invention provides a method for identifying the sleeping state of infants and young children based on thermal infrared. The method: first, by continuously acquiring multiple frames of thermal infrared images, compared with traditional images, the thermal infrared images include temperature information of the body areas of infants and young children, and can reflect information such as the temperature conditions of various body areas, rather than simply taking pictures; wherein the body areas include a first target area such as the head area and a second target area such as the torso area; secondly, the acquired multiple frames of thermal infrared images are segmented frame by frame to obtain the time series data of the first target area of ​​the infant and / or the time series data of the second target area of ​​the infant that changes with time, and specifically adopt a dynamic segmentation method that combines static thresholds and dynamic thresholds, and dynamically segment the infant and young children according to the temperature conditions of the body areas. The time series data of the body area can dynamically adjust the threshold value, identify the temperature changes of the infant, and improve the accuracy of segmentation; furthermore, by detecting the time series data of the body area obtained, the movement change information of the infant is obtained, and the movement changes of the infant at the preset time can be detected. Specifically, micro-motion detection can be used for detection, and micro-motion detection can improve the accuracy of recognition compared with non-micro-motion detection; finally, based on the movement change information of the infant, the sleeping state of the infant is identified; combining thermal infrared, segmentation, detection and recognition methods, it is possible to accurately identify the different sleeping states of the infant, track the sleeping state of the infant in real time, provide technical support for infant sleep monitoring, improve the safety and comfort of the infant's sleep, and realize intelligent care.

[0192] In addition, this method also includes preprocessing. By preprocessing the acquired multiple frames of thermal infrared images to obtain multiple frames of initial thermal infrared images, data deviation can be reduced and the images can be made more consistent with the analysis logic. In this method, the accuracy of identifying the sleeping state of infants and young children can be further improved, realizing intelligent care.

[0193] Second, see Figure 9 The embodiment of the present invention provides a device for identifying infant sleep status based on thermal infrared, the device 200 comprising:

[0194] A thermal infrared image acquisition module 201 is configured to continuously acquire multiple frames of thermal infrared images, wherein the thermal infrared images include the first target area and / or the second target area of ​​the infant;

[0195] A segmentation module 202 is configured to segment the plurality of thermal infrared images frame by frame to obtain time series data of a first target area and / or time series data of a second target area;

[0196] A detection module 203 is configured to detect position changes in the time series data of the first target area and / or the time series data of the second target area of ​​the infant, and obtain movement change information of the infant;

[0197] The sleeping state identification module 204 is configured to identify the sleeping state of the infant based on the movement change information.

[0198] It should be noted that the modules in the second aspect of the infant sleep state recognition device based on thermal infrared correspond one-to-one to the steps in the aforementioned first aspect of the infant sleep state recognition method based on thermal infrared. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned infant sleep state recognition method based on thermal infrared, and will not be repeated here.

[0199] The present invention provides a device for identifying the sleeping state of infants and young children based on thermal infrared. The method: first, a thermal infrared image module is used to continuously acquire multiple frames of thermal infrared images. Compared with traditional images, thermal infrared images include temperature information of the body area of ​​the infant and young children, which can reflect information such as the temperature conditions of various body areas, and do not simply serve the purpose of shooting; wherein the body area includes a first target area such as the head area and a second target area such as the torso area; secondly, a segmentation module is used to segment the acquired multiple frames of thermal infrared images frame by frame to obtain time series data of the first target area of ​​the infant and young children and / or time series data of the second target area of ​​the infant and young children that change with time, specifically, the segmentation module is used to segment the acquired multiple frames of thermal infrared images frame by frame to obtain time series data of the first target area of ​​the infant and young children that change with time and / or time series data of the second target area that change with time. A dynamic segmentation method combining static thresholds and dynamic thresholds is used to dynamically segment the infant's body area according to the temperature of the infant's body area, and the threshold can be dynamically adjusted to identify the temperature change of the infant, thereby improving the accuracy of segmentation. Furthermore, the detection module detects the time series data of the infant's body area to obtain the infant's movement change information, and can detect the infant's movement change at a preset time. Specifically, micro-motion detection can be used for detection, and micro-motion detection can improve the accuracy of recognition compared to non-micro-motion detection. Finally, the sleep state recognition module recognizes the infant's sleep state based on the infant's movement change information. The infant sleep state recognition device based on thermal infrared combines thermal infrared, segmentation, detection and recognition methods, and can accurately identify the different sleep states of infants, track the infant's sleep state in real time, provide technical support for infant sleep monitoring, improve the safety and comfort of infant sleep, and realize intelligent care.

[0200] In addition, the device may also include a preprocessing module, which preprocesses the acquired multiple frames of thermal infrared images to obtain multiple frames of initial thermal infrared images, thereby reducing data deviation and making the images more consistent with the analysis logic. In this method, the accuracy of identifying the sleeping state of infants and young children can be further improved, thereby realizing intelligent care.

[0201] In the third aspect, the present invention provides an infant care robot, see Figure 10 The hardware structure diagram of an infant care robot provided by the present invention is shown, and the care robot includes: at least one processor 301, at least one memory 302, and computer program instructions stored in the memory 302. When the computer program instructions are executed by the processor 301, the infant sleep state recognition method based on thermal infrared as described in the first aspect above is implemented.

[0202] Specifically, the processor may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits for implementing the embodiments of the present invention.

[0203] The memory may include a large capacity memory for data or instructions. By way of example and not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory may include a removable or non-removable (or fixed) medium. Where appropriate, the memory may be inside or outside the data processing device. In a specific embodiment, the memory is a non-volatile solid-state memory. In a specific embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0204] The processor reads and executes computer program instructions stored in the memory to implement any one of the thermal infrared-based infant sleep state recognition methods in the above embodiments.

[0205] The infant care robot may further include a communication interface 303 and a bus 310. Figure 10 As shown, the processor 301 , the memory 302 , and the communication interface 303 are connected via a bus 310 and communicate with each other.

[0206] The communication interface is mainly used to implement communication between the modules, devices, units and / or equipment in the embodiments of the present invention.

[0207] Bus comprises hardware, software or both, couples the parts of electronic equipment to each other.For example, and not limitation, bus can comprise accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations.In suitable cases, bus can comprise one or more buses.Although the embodiment of the present invention describes and shows specific bus, the present invention considers any suitable bus or interconnection.

[0208] In a fourth aspect, in conjunction with the thermal infrared-based infant sleep state recognition method provided in the first aspect, embodiments of the present invention may provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when executed by a processor, the computer program instructions implement any of the thermal infrared-based infant sleep state recognition methods described in the aforementioned embodiments.

[0209] In summary, the embodiments of the present invention provide a method, device, and robot for identifying the sleeping state of infants and young children based on thermal infrared.

[0210] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0211] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in unit, a function card or the like. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0212] It should also be noted that the exemplary embodiments described herein describe methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the steps described above. In other words, the steps may be performed in the order described in the embodiments, or in a different order, or several steps may be performed simultaneously.

[0213] The above description is only a specific embodiment of the present invention. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention.

Claims

1. A method for identifying infant sleep status based on thermal infrared, characterized in that: The method comprises: Continuously acquiring multiple frames of thermal infrared images, wherein the thermal infrared images include the first target area and / or the second target area of ​​the infant; Segmenting the multiple frames of thermal infrared images frame by frame to obtain time series data of the first target area and / or time series data of the second target area; detecting position changes of the time series data of the first target area and / or the time series data of the second target area of ​​the infant to obtain movement change information of the infant; The sleeping state of the infant is identified based on the movement change information.

2. The method for identifying infant sleep status based on thermal infrared according to claim 1, characterized in that: The step of segmenting the multiple frames of thermal infrared images frame by frame to obtain time series data of the first target area and / or time series data of the second target area includes: Performing preliminary segmentation on the multiple frames of thermal infrared images according to a first temperature threshold to obtain multiple frames of preliminary segmented images, wherein the preliminary segmented images include the first target area and / or the second target area of ​​the infant; Performing dynamic threshold segmentation on the multiple frames of preliminary segmented images according to a dynamic temperature threshold range and an iteration step, and determining a segmentation threshold that meets preset conditions by analyzing changes in connected domain features; The multiple frames of preliminary segmented images are segmented frame by frame according to the segmentation threshold to obtain time series data of the first target area and / or time series data of the second target area.

3. The method for identifying infant sleep status based on thermal infrared according to claim 2, characterized in that: The preliminarily segmenting the multiple frames of thermal infrared images according to the first temperature threshold to obtain multiple frames of preliminarily segmented images, wherein the preliminarily segmented images include the first target area and / or the second target area of ​​the infant, including: Acquiring the multiple frames of thermal infrared images; Setting a first temperature threshold, wherein the first temperature threshold is lower than the effective surface temperature of a human body; The pixel values ​​in the multiple frames of thermal infrared images that are less than the first temperature threshold are set as background values ​​frame by frame, and the remaining pixel values ​​remain unchanged, and the multiple frames of preliminary segmented images are output, where the preliminary segmented images include the first target area and / or the second target area of ​​the infant.

4. The method for identifying infant sleep status based on thermal infrared according to claim 2, characterized in that: The method of performing dynamic threshold segmentation on the multiple frames of preliminary segmented images based on the dynamic temperature threshold range and the iteration step size, and determining a segmentation threshold meeting a preset condition by analyzing connected domain features, includes: Setting the dynamic temperature threshold range and the iteration step size; Taking the lower limit of the dynamic temperature threshold range as the starting threshold, gradually increasing the segmentation threshold according to the iterative step size, and iteratively segmenting the multiple frames of preliminary segmented images until the upper limit of the dynamic temperature threshold range is reached; Calculate the feature ratio of the first connected domain and the second connected domain in each iteration; When the feature ratio reaches a preset threshold, determining the segmentation threshold; The upper limit value is a preset body surface temperature value that is greater than an ambient temperature reference value, and the lower limit value is greater than the ambient temperature reference value.

5. The method for identifying infant sleep status based on thermal infrared according to claim 2, characterized in that: The step of segmenting the preliminary segmented image frame by frame based on the segmentation threshold to obtain the time series data of the first target area and / or the time series data of the second target area includes: Binarizing the multiple frames of preliminary segmented images frame by frame according to the segmentation threshold, removing the first target region frame by frame, and selecting a connected region that meets a preset area threshold from the remaining regions as a second candidate target region; checking the number of adjacent pixels between the first target region and the second candidate target region frame by frame, and confirming the second candidate target region whose number of adjacent pixels is greater than a preset threshold as the second target region; The first target area and / or the second target area of ​​each frame are integrated to obtain time series data of the first target area and / or time series data of the second target area.

6. The method for identifying infant sleep status based on thermal infrared according to claim 2, characterized in that: The method further includes: dynamically adjusting the dynamic temperature threshold range: Acquire a preset number of frames of the preliminary segmented image, extract the feature ratio between the first connected region and the second connected region frame by frame, and generate a feature ratio sequence; Calculating the coefficient of variation of the characteristic ratio sequence; When the coefficient of variation is greater than a preset stability threshold, adjusting the upper limit and the lower limit of the dynamic temperature threshold range according to the current characteristic ratio sequence; The upper limit of the dynamic temperature threshold range is a preset body surface temperature value that is greater than an ambient temperature reference value, and the lower limit is greater than the ambient temperature reference value.

7. The method for identifying infant sleep status based on thermal infrared according to any one of claims 1 to 6, characterized in that: The detecting position changes of the time series data of the first target area and / or the time series data of the second target area of ​​the infant to obtain movement change information of the infant includes: Calculating the aspect ratio of the time series data of the first target area frame by frame to obtain an aspect ratio sequence of the first target area; Calculating the aspect ratio difference between every two adjacent frames in the aspect ratio sequence of the first target area to obtain an aspect ratio difference sequence of the first target area; and / or, Calculating the aspect ratio of the time series data of the second target area frame by frame to obtain an aspect ratio sequence of the second target area; calculating the aspect ratio difference between every two adjacent frames in the aspect ratio sequence of the second target area to obtain an aspect ratio difference sequence of the second target area; The movement change information of the infant includes a length-to-width ratio difference sequence of the first target area and / or a length-to-width ratio difference sequence of the second target area.

8. The method for identifying infant sleep status based on thermal infrared according to claim 7, characterized in that: The identifying the sleeping state of the infant based on the movement change information includes: When the difference value of the first consecutive frames in the aspect ratio difference sequence of the first target area is less than or equal to a first preset difference threshold and shows a decreasing trend from the difference value in the second consecutive frames, the sleeping state of the infant is identified as falling asleep; When a difference value in a third consecutive frame group in the aspect ratio difference value sequence of the first target area is equal to a second preset difference threshold, identifying the sleeping state as a deep sleep state; When the difference in the fourth consecutive frame group in the aspect ratio difference sequence of the second target area is equal to the second preset difference threshold, and the difference in the fifth consecutive frame group in the aspect ratio difference sequence of the second target area or the aspect ratio difference sequence of the first target area is greater than the second preset difference threshold, the sleeping state is identified as the waking state.

9. A device for identifying infant sleep status based on thermal infrared, characterized in that: The device comprises: a thermal infrared image acquisition module, configured to continuously acquire multiple frames of thermal infrared images, wherein the thermal infrared images include the first target area and / or the second target area of ​​the infant; a segmentation module, configured to segment the plurality of thermal infrared images frame by frame to obtain time series data of the first target area and / or time series data of the second target area; a detection module, configured to detect position changes in the time series data of the first target area and / or the time series data of the second target area of ​​the infant, and obtain movement change information of the infant; The sleeping state recognition module is used to recognize the sleeping state of the infant based on the movement change information.

10. An infant care robot, characterized in that: include: At least one processor, at least one memory, and computer program instructions stored in the memory, which implement the method according to any one of claims 1 to 8 when the computer program instructions are executed by the processor.

Images