Sleep determination system, sleep determination method, and sleep determination program

Abstract

To provide a sleep determination system, sleep determination method, and sleep determination program capable of determining correctly and in real time whether an infant is in a sleeping state or in an awakening state.SOLUTION: A sleep determination system 1 according to the present invention determines whether an infant is in a sleeping state or an awakening state. The system includes a central processing section 12 and a storage section 13. The storage section 13 stores index information including a face motion index of the infant which indicates a state transition between the sleeping state and the awakening state. The central processing section 12 detects a face motion of the infant based on a face image including an eyelid opening / closing motion acquired by an imaging device 21, and compares the face motion with the index information stored in the storage section 13, so as to determine whether the infant is in the sleeping state or the awakening state.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a sleep determination system, a sleep determination method, and a sleep determination program, and more particularly to a sleep determination system, a sleep determination method, and a sleep determination program for determining whether an infant is in a sleeping state or an awake state. [Background technology]

[0002] Systems for detecting the posture and body movements of infants have been known as tools for monitoring (watching over) infants. Such systems include systems that detect the posture and body movements of infants based on, for example, images captured by an imaging device, and systems that detect the posture and body movements based on data transmitted from a body movement sensor attached to the infant's clothing (e.g., diaper).

[0003] In recent years, various technologies have been developed for such systems that determine whether an infant is asleep based on the infant's posture and body movements. These systems are convenient because they can grasp not only the infant's posture and body movements but also whether the infant is asleep, even from a distance.

[0004] However, in systems that detect the posture and body movements of an infant based on images captured by an imaging device, depending on the installation position of the imaging device, it is difficult to adequately detect the posture and body movements of an infant at the back of the imaging direction, and if the infant is covered with a blanket or the like, this cannot be detected in the first place, so in such cases it is not possible to accurately determine whether the infant is asleep or not.

[0005] On the other hand, in systems that detect an infant's posture and body movements based on data detected by a body movement sensor, there is a high risk that the system will mistakenly determine that the infant is not asleep if the infant makes even a slight movement, even though the infant is actually asleep.As a result, it is difficult to determine that the infant is asleep as early as possible (in real time) after falling asleep.

[0006] Taking these factors into consideration, it can be said that in order to accurately determine whether an infant is asleep, it is more desirable to base the determination primarily on information other than that related to the infant's posture and body movements.

[0007] Generally, when a person falls asleep or wakes up, the main body part that necessarily moves is the eyelids, while the most common body part that is likely to be exposed and not covered by a blanket or the like when an infant is sleeping is the face. Furthermore, unlike adults who are conscious of their surroundings, infants do not refrain from falling asleep or pretend to be asleep. Considering these points, it is efficient to accurately and in real time determine whether an infant is asleep or awake based on the opening and closing of the eyelids.

[0008] For example, Patent Documents 1 and 2 disclose techniques for determining a sleep state based on the opening and closing of eyelids.

[0009] The technology described in Patent Document 1 includes a feature point detection means for detecting eyebrows, upper eyelids, and lower eyelids from a face image, an eye opening detection means for detecting the eye opening degree based on the ratio A of the distance between the upper and lower eyelids to the distance between the eyebrows and the lower eyelid, and an eye opening / closing determination means for determining whether the eyes are open or closed based on the eye opening degree. This technology can reduce false detection of the eyebrow position, making it possible to accurately detect the onset of drowsiness.

[0010] On the other hand, the technology described in Patent Document 2 includes a means for detecting the eyelid opening / closing state of a subject, a means for sequentially detecting the time interval of the transition between the open and closed states of the eyelids, a means for determining an eyelid opening / closing feature calculation time width suitable for the subject from data on the maximum eyelid opening time when the subject is awake, a means for calculating an eyelid opening / closing feature from time-series data on the time interval of the transition between the open and closed states of the eyelids over the eyelid opening / closing feature calculation time width, and a drowsiness determination means for determining that the subject is feeling drowsy when the eyelid opening / closing feature falls below a first threshold or exceeds a second threshold higher than the first threshold. According to this technology, it is determined that the subject is feeling drowsy when the eyelid opening / closing movement measured over the time interval of the transition between the open and closed states of the eyelids is different from a state in which the subject is not feeling drowsy, thereby making it possible to accurately detect the onset of drowsiness. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-219555 [Patent Document 2] Japanese Patent Application Publication No. 2017-209262 Summary of the Invention [Problem to be solved by the invention]

[0012] The technologies described in Patent Documents 1 and 2 above both relate to drowsiness detection devices that detect drowsiness based on the opening and closing of eyelids, and are primarily aimed at preventing drivers from falling asleep while driving a vehicle. That is, the technologies described in Patent Documents 1 and 2 are configured to be applicable to adults and other people who open and close their eyelids differently when drowsy, which inevitably tends to make the program configuration complex.

[0013] Moreover, as described above, the technologies described in Patent Documents 1 and 2 are specialized for detecting drowsiness, and therefore it is difficult to say that they are capable of accurately and in real time determining the transition from a sleep state to an awake state.

[0014] The present invention relates to providing a sleep determination system, a sleep determination method, and a sleep determination program that can overcome the drawbacks of the conventional techniques described above. [Means for solving the problem]

[0015] The present invention relates to a sleep determination system for determining whether an infant is in a sleeping state or a waking state. The sleep determination system includes a facial movement detection means for detecting the facial movement of the infant based on a facial image including the opening and closing movement of the eyelids acquired by an imaging device, a storage means for storing index information including an infant's facial movement index indicating a state transition between the sleeping state and the awake state, and a determination means for comparing the facial movement detected by the facial movement detection means with the index information stored in the storage means to determine whether the infant is in the sleeping state or the awake state.

[0016] The present invention relates to a sleep determination method for determining whether an infant is in a sleeping state or a waking state using a sleep determination system. The sleep determination method includes a facial movement detection step of detecting facial movements of an infant based on a facial image including eyelid opening and closing movements acquired by an imaging device, and a determination step of comparing the facial movements detected by the facial movement detection step with index information including an infant facial movement index indicating a state transition between the sleeping state and the awake state to determine whether the infant is in the sleeping state or the awake state.

[0017] The present invention relates to a sleep determination program for determining whether an infant is in a sleeping state or a waking state using a sleep determination system. The sleep determination system includes a memory means for storing index information including an infant's facial movement index indicating a state transition between the sleeping state and the awake state, and the sleep determination program causes a computer to execute a facial movement detection step for detecting the infant's facial movement based on a facial image including eyelid opening and closing movements acquired by an imaging device, and a determination step for comparing the facial movement detected by the facial movement detection step with the index information stored in the memory means to determine whether the infant is in the sleeping state or the awake state. [Effects of the Invention]

[0018] The sleep determination system, sleep determination method, and sleep determination program of the present invention can accurately determine whether an infant is asleep or awake in real time, even though they have a simple configuration. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is an explanatory diagram for explaining a usage situation of the sleep determination system according to this embodiment. [Figure 2] FIG. 2 is a block diagram showing an outline of the sleep determination system of FIG. [Figure 3] FIG. 3 is a flowchart showing the control contents in the control device of FIG. [Figure 4] FIG. 4 is a flowchart showing the contents of the sleep / wake determination process of FIG. [Figure 5] FIG. 5 is an explanatory diagram showing an example of information stored in the index information storage area of ​​the storage device of FIG. [Figure 6] FIG. 6 is an explanatory diagram showing an example of information stored in the sleep-wake information storage area of ​​the storage device of FIG. [Figure 7] FIG. 7 is an explanatory diagram for explaining the degree of eyelid opening of an infant. DETAILED DESCRIPTION OF THE INVENTION

[0020] A sleep determination system, a sleep determination method, and a sleep determination program according to a preferred embodiment of the present invention will be described below with reference to the drawings. Fig. 1 shows how a sleep determination system 1 according to this embodiment determines whether an infant B is in a "sleeping state" or an "awake state," and Fig. 2 shows an outline of the system configuration of the sleep determination system.

[0021] [Configuration of sleep determination system 1] 1 and 2, the sleep determination system 1 according to this embodiment includes a management server 10 that determines whether infant B is in a "sleeping state" or an "awake state," an imaging device 21 that captures a facial image of infant B, a body movement sensor 22 that detects the posture and body movement of infant B, and a user terminal 30 owned by a user. In this embodiment, the user of the sleep determination system 1 is assumed to be an adult who monitors (watches over) infant B, such as the guardian of infant B or a staff member of a facility that raises (looks after) infant B.

[0022] The management server 10 according to this embodiment is installed in a location away from the location of the user who uses the sleep determination system 1 and the location of infant B. As will be described in detail later, the management server 10 is configured to determine whether infant B is in a "sleeping state" or an "awake state" (hereinafter referred to as "sleep / wake determination") based on facial image data of infant B captured by an imaging device 21 and data related to infant B's posture and body movement acquired by a body movement sensor 22 (hereinafter also referred to as "posture / body movement data"), and to transmit (notify) the determination result to a user terminal 30.

[0023] [Administration Server 10] First, the management server 10 will be described with reference to FIGS. As shown in Figure 2, the management server 10 includes a control device 11, an input unit 14, a display unit 15, and a communication unit 16, and is electrically connected (wired or wireless) to an imaging device 21, a body movement sensor 22, and a user terminal 30 via a public communication line 2.

[0024] [Control device 11] The control device 11 is, for example, a known personal computer system, and includes a central processing unit (CPU) 12 and a storage unit 13.

[0025] [Central Processing Unit 12] The central processing unit 12 reads various programs stored in the storage unit 13, executes predetermined arithmetic processing, and controls the display unit 15 to display predetermined images, etc. In this embodiment, the management server 10, the image capture device 21, and the body movement sensor 22 are configured to be online on the condition that basic information such as an "infant ID" has been input (logged in) from the user terminal 30. This enables the central processing unit 12 to determine whether infant B is asleep or awake based on the facial image data of infant B (see FIG. 1) captured by the image capture device 21 and the posture and body movement data of infant B acquired by the body movement sensor 22. Note that in this embodiment, the central processing unit 12 is configured to, in principle, continuously monitor infant B (determine whether infant B is asleep or awake) while the management server 10, the image capture device 21, and the body movement sensor 22 are online.

[0026] [Storage section 13] The storage unit 13 is made of a semiconductor memory such as a ROM (Read Only Memory), and has storage areas such as a program storage area for storing various programs, an index information storage area 13A, and a sleep / wake information storage area 13B. The program storage area stores a basic operation program that controls the basic operation of the sleep determination system 1, a sleep / wake determination program, an image recognition program, and the like.

[0027] The image recognition program is a program for recognizing the opening and closing of eyelids from facial image data of an infant (see FIG. 1) captured by the imaging device 21. Such image recognition technology for recognizing partial facial movements from facial image data is already known, and therefore a detailed description thereof will be omitted. However, this embodiment is also configured to recognize the opening and closing of eyelids of infant B from facial image data using known image recognition technology. The sleep determination program will be described later (see FIG. 4).

[0028] [Indicator information storage area 13A] As shown in FIG. 5, the index information storage area 13A is an area for storing index information that serves as a criterion when the central processing unit 12 determines whether the infant B (see FIG. 1) is asleep or awake. In the index information storage area 13A, for example, as index information indicating a state transition from an "awake state" to a "sleeping state", if infant B's "eyelid opening and closing movement: present" and "body movement: present or absent" then "judgment result: awake" is stored, while as index information indicating a state transition from a "sleeping state" to an "awake state", if infant B's "eyelid opening and closing movement: present multiple times in 30 seconds" and "body movement: present or absent" then "judgment result: awake" is stored.

[0029] 5 is merely an example, and the content can be appropriately changed (overwritten, etc.), for example, as index information indicating a state transition from an "awake state" to a "sleeping state," if infant B's "eyelid opening / closing movement: no for 30 seconds or more" and "body movement: present" (see "···" at the top of the figure), "judgment result: asleep" may be displayed, or if "eyelid opening / closing movement: no for 30 seconds or more" and "body movement: present for X minutes or more consecutively" (see "···" at the bottom of the figure), "judgment result: asleep." In the following, the explanation will be given on the assumption that the "···" stored in the index information storage area 13A of FIG. 5 stores the above-mentioned exemplary content.

[0030] [Sleep / wake information storage area 13B] As shown in FIG. 6, the sleep / wake information storage area 13B is an area for storing the "basic information" and "sleep / wake determination information" of the infant. Specifically, the area of ​​sleep / wake information storage area 13B that stores "basic information" stores basic information about infant B (see FIG. 1) for which sleep / wake determination is performed, such as "infant ID," "infant name," "birthday," and "age." For example, basic information (personal information, etc.) about infant B, such as "infant name," "birthday," and "age," is information (registration information) acquired when the user uses sleep determination system 1 for the first time, and "infant ID" is information automatically generated in sleep determination system 1 when a registration procedure, etc. is performed.

[0031] In the area of ​​the sleep-wake information storage area 13B that stores "sleep-wake determination information," information on sleep-wake determination made by the sleep determination system 1 for each "infant ID" (infant B, see Figure 1), such as the "determination date," "time when the state was determined to be asleep," and "time when the state was determined to be awake," is stored in chronological order. Storing such information in the sleep-wake information storage area 13B enables the user to check the sleep cycle of infant B after the fact.

[0032] In addition, the area of ​​the sleep / wake information storage area 13B that stores the ``sleep / wake determination information'' can also store information regarding the ``eyelid opening degree'' of infant B (see Figure 1) when the central processing unit 12 determines that the infant is in a ``sleeping state'' in the sleep / wake determination (hereinafter also referred to as ``opening degree information''), as shown in Figure 6.

[0033] Here, a method for calculating the "eyelid opening degree" in a sleeping state will be explained with reference to Fig. 7. Fig. 7(a) shows the state of the eyes of infant B when he is in an "awake state," and Fig. 7(b) shows the state of the eyes of infant B when he is sleeping with his eyes half-closed. As shown in Figures 7(a) and (b), the "eyelid opening degree" in the "sleeping state" can be calculated by the following steps: (1) determining the distance C1 between the highest point of the upper eyelid Lt and the lowest point of the lower eyelid Lb in the "awake state," (2) similarly determining the distance C2 between the highest point of the upper eyelid Lt and the lowest point of the lower eyelid Lb in the "sleeping state," and (3) substituting the determined distances C1 and C2 into the "calculation formula: distance C2 / distance C1."

[0034] Such distances C1 and C2 can be easily determined by applying known image recognition technology to facial image data around the eye E of infant B (see FIG. 1) captured by the imaging device 21. In this case, facial image data in the "awake state" (see FIG. 7(a)) can be used to determine distance C1, and facial image data in the "sleeping state" (see FIG. 7(b)) can be used to determine distance C2. By storing such eyelid opening information in the "sleeping state" in the sleep-wake information storage area 13B, it becomes possible to grasp the state of infant B's eyes ("eyelid opening") in the "sleeping state."

[0035] [Input unit 14, display unit 15 and communication unit 16] Next, the input unit 14, the display unit 15, and the communication unit 16 that constitute the management server 10 will be described with reference to FIG. As shown in Fig. 2, the input unit 14 is, for example, a known keyboard or touch panel, and is a device for inputting data when adding, changing, or deleting various data stored in the memory unit 13. The display unit 15 is, for example, a liquid crystal display (LCD), and is a device for displaying a predetermined image (for example, the contents stored in the memory unit 13 (see Figs. 5 and 6)) based on a command from the central processing unit 12. The communication unit 16 is an interface capable of communicating with communication devices such as the imaging device 21, the body movement sensor 22, and the user terminal 30 via the public communication line 2.

[0036] [Imaging device 21] Next, the imaging device 21 will be described with reference to FIGS. 1 and 2, the imaging device 21 is made up of a camera capable of capturing moving images of the infant B, and may include, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Devices) sensor. The imaging device 21 is preferably of the so-called auto-tracking type that automatically tracks the face, etc., and a tabletop type (see FIG. 2) or a fixed type can be used as long as it can capture an image of the face of the infant B. Note that FIG. 1 shows an auto-tracking tabletop type imaging device 21.

[0037] The imaging device 21 of this embodiment is electrically connected (wired or wireless) to the management server 10 and the user terminal 30, and is configured to capture a facial image of infant B upon receiving a command signal from the central processing unit 12 and / or the user terminal 30, and transmit the facial image data to the central processing unit 12 and the user terminal 30.

[0038] [Body movement sensor 22] Like the imaging device 21, the body movement sensor 22 is electrically connected (wired or wireless) to the management server 10 and the user terminal 30 and is attached to infant B's clothing (e.g., diaper). The body movement sensor 22 is a so-called acceleration sensor (e.g., a so-called triaxial sensor) and is configured to sequentially output acceleration signals (posture and body movement data) to the central processing unit 12 (management server 10) upon detecting infant B's body movement. Upon receiving the acceleration signal, the central processing unit 12 converts it into a voltage signal, digitally converts it, removes noise, and performs processing such as combining triaxial accelerations or extracting gravity direction components to generate acceleration data. This allows the central processing unit 12 to constantly monitor (understand) infant B's detailed body movements and posture based on the generated acceleration data.

[0039] Next, the user terminal 30 will be described with reference to FIG. 2, the user terminal 30 is a terminal owned by a user that can communicate with other communication devices (e.g., the management server 10, the imaging device 21, and the body movement sensor 22) via the public communication line 2. As the user terminal 30, for example, a well-known personal computer 30A or a mobile terminal (e.g., a so-called smartphone) 30B can be adopted.

[0040] In this embodiment, when a user uses the sleep determination system 1, for example, the user accesses the management server 10 from the user terminal 30, and then enters (logs in) "basic information" (e.g., "infant ID" in FIG. 6) of infant B (see FIG. 1), etc., so that the management server 10, the image capture device 21, and the body movement sensor 22 go online. When these are online, face image data captured by the image capture device 21 and posture and body movement data detected by the body movement sensor 22 are sequentially transmitted to the management server 10, thereby enabling the central processing unit 12 to determine whether infant B is asleep or awake.

[0041] As will be described in detail later, in this embodiment, the central processing unit 12 is configured to transmit information indicating the details of the following cases to the user terminal 30: (1) when it is determined that infant B has transitioned from an "awake state" to a "sleeping state" (see "step S105" in FIG. 4); (2) when it is determined that infant B has transitioned from a "sleeping state" to an "awake state" (see "step S110" in FIG. 4); or (3) when it is determined that neither facial image data from the imaging device 21 nor posture / body movement data from the body movement sensor 22 has been input (see "step S600" in FIG. 3). This allows the user to grasp infant B's state ("awake state," "sleeping state," and "abnormal state") even when the user is located far from infant B. Note that the user terminal 30 according to this embodiment is configured to allow the user to remotely control the imaging device 21 by performing a predetermined operation. This allows the user to check on infant B's condition at a desired timing even when the user is located far from infant B.

[0042] [Control process in sleep determination system 1] Next, the control process (hereinafter referred to as "infant monitoring process") executed in the sleep determination system 1 will be described with reference to Figures 1 to 4. For convenience of explanation, the following description will be made on the assumption that (1) the imaging device 21 is in a state where it can acquire a facial image of infant B, (2) the body movement sensor 22 is in a state where it can detect the posture and body movement of infant B, (3) the management server 10, the imaging device 21, and the body movement sensor 22 are online, and (4) the management server 10 and the user terminal 30 are in a state where they can communicate with each other.

[0043] (Step S100) As shown in FIG. 3, the infant monitoring process according to this embodiment is mainly performed by the central processing unit 12 in accordance with the sleep / wake determination program stored in the memory unit 13, and begins with processing of step S100.

[0044] The central processing unit 12 performs sleep / wake determination processing in step S100. Specifically, the central processing unit 12 executes a subroutine shown in FIG. 4 when performing the sleep / wake determination processing in step S100. After performing the sleep / wake determination process, the central processing unit 12 moves the process to step S200.

[0045] (Step S101) Next, before describing the processing of step S200 by the central processing unit 12, the subroutine (processing of "steps S101" to S113") of the sleep / wake determination processing (step S100) will be described with reference to FIG. As shown in FIGS. 2 to 4, the subroutine for the sleep / wake determination process starts with the execution of the process in step S101.

[0046] In step S101, the central processing unit 12 performs a process of determining whether or not an image of the eyelids of infant B is included in the face image data transmitted by the imaging device 21. The recognition of the image of the eyelids of infant B is performed in accordance with an image recognition program stored in the storage unit 13. If the central processing unit 12 determines that the facial image data includes an image of the eyelids, it proceeds to step S102, and if it determines that the facial image data does not include an image of the eyelids, it proceeds to step S112. Note that the facial image data may not include an image of the eyelids when, for example, infant B moves its body while sleeping, infant B is lying face down, or infant B's eyelids are covered by a blanket or the like.

[0047] (Step S102) In step S102, the central processing unit 12 performs a process of determining whether or not there has been no eyelid opening / closing movement for a specified time or longer. The content of "there has been no eyelid opening / closing movement for a specified time or longer" corresponds to the index information ("no eyelid opening / closing movement for 30 seconds or longer") stored in the index information storage area 13A of the storage unit 13. The above-mentioned specified time is "30 seconds" in accordance with the information stored in the index information storage area 13A. This specified time is not limited to "30 seconds" and can be changed as appropriate, as described above. If the central processing unit 12 determines that the eyelids have not been opened or closed for a specified time or longer, it proceeds to step S103, and if it determines that the eyelids have been opened or closed within the specified time, it proceeds to step S107.

[0048] In the process of step S102, the determination is made based solely on the "eyelid opening and closing movement," but it is also possible to make the determination by taking into account the above-mentioned "eyelid opening degree" (see FIG. 6). This point will be explained below using an example of infant B, who normally sleeps with his eyes facing away from the camera (eye opening: 20%). Infant B sleeping with eyes half-opened like this maintains "eyelid opening: 20%" for more than a specified time, and is therefore determined to be in a "sleeping state" because "there has been no eyelid opening / closing movement for more than a specified time," just like infant B sleeping with "eye opening: 0%" (see "Step S103"). However, if infant B sleeping with "eyelid opening: 0%" opens and closes his / her eyelids (multiple times) within the range of "eye opening: 0% to 20%," this will be determined to be "there has been eyelid opening / closing movement within the specified time," and there is a risk that infant B will be determined (misjudged) to be in a "wakeful state" despite being asleep (see "Step S108").

[0049] Therefore, in the process of step S102, it is preferable to determine whether or not the eyelids have not been opened or closed for a specified period of time, taking into consideration information about the "eyelid opening degree," and, for example, in the case of infant B sleeping with eyes half-opened as exemplified above, the opening and closing of the eyelids within the range of "eye opening degree: 0% to 20%" is not included in the "eyelid opening and closing movement." This configuration makes it possible to improve the accuracy of sleep determination.

[0050] In this regard, if it is determined in step S102 that "there has been no eyelid opening / closing movement for a specified period of time or longer," it is desirable to determine the "eyelid opening degree" of infant B based on the facial image data transmitted from the imaging device 21 and store (accumulate) this in the memory unit 13 (the "sleep / wake information memory area 13B" in Figure 6).

[0051] In this case, the information (eyelid opening information) stored in the sleep-wake information storage area 13B may be analyzed, and the analyzed information (hereinafter referred to as "analysis result information") may be stored as training data in the storage unit 13 (for example, "sleep-wake information storage area 13B" in FIG. 6). With this configuration, the determination in step S102 may be made taking into account the analysis result information, which is training data, and in this case, the accuracy of sleep determination may be further improved.

[0052] For example, even if the central processing unit 12 determines that the baby B is in an "awake state," when the user actually checks on the baby B, the baby may be asleep with its eyes half-closed. In this case, the user can request the management server 10 to acquire and store such information in order to have the management server 10 learn the "eyelid opening degree" of the infant B who is sleeping with his eyes half-closed.

[0053] Specifically, learning of such "eyelid opening" can be performed by the following procedure: (1) the user transmits information (sleep information) indicating a "sleeping state" (false determination) from the user terminal 30 to the management server 10; (2) the central processing unit 12 then determines the "eyelid opening" based on the received facial image data; and (3) the central processing unit 12 then analyzes the "eyelid opening" of infant B, taking into account the determined opening information, and stores the analysis result information as training data in the storage unit 13 (e.g., the "sleep-wake information storage area 13B" in FIG. 6). With this configuration, the determination in step S102 can be made taking into account the analysis result information that corrects the false determination, thereby significantly improving the accuracy of sleep determination.

[0054] (Step S103) In step S103, the central processing unit 12 performs a process of determining that the infant B is in a "sleeping state" (sleeping state determination process). After performing the sleep state determination, the central processing unit 12 advances the process to step S104.

[0055] (Step S104) In step S104, the central processing unit 12 performs a process of determining whether or not the state has transitioned from the "wakeful state" to the "asleep state." If the central processing unit 12 determines that a state transition has occurred, it moves the process to step S105, and if it determines that a state transition has not occurred, it returns the process to step S101. As described above, in this embodiment, when the use of the sleep determination system 1 is started while infant B is asleep, the process of "Steps S101 to S104" is configured to be repeatedly executed until infant B transitions to the "awake state."

[0056] (Step S105) In step S105, the central processing unit 12 performs a process (notification process) of transmitting information indicating that infant B has transitioned from the "wakeful state" to the "sleeping state" to the user terminal 30. This allows the user to know that infant B has fallen asleep through the user terminal 30 (for example, a displayed image and sound) even when the user is in a location far away from infant B. After performing the notification process, the central processing unit 12 advances the process to step S106.

[0057] (Step S106) In step S106, the central processing unit 12 performs a process of storing information such as the date and time when the sleep state determination was made in step S103 in the sleep / wake information storage area 13B of the memory unit 13 (see ``Sleep / wake determination information'' in Figure 6). After performing the above-mentioned storage process, the central processing unit 12 proceeds to step S101. Note that, if the "eyelid opening degree" as described above is determined in the process of step S102, it is preferable to also store information regarding the "eyelid opening degree" in this step 106 (see "Sleep / wakefulness determination information" in FIG. 6).

[0058] (Step S107) In step S107, the central processing unit 12 performs a process of determining whether or not the eyelids have been opened and closed multiple times within a specified time. The content of "the eyelids have been opened and closed multiple times within a specified time" corresponds to the index information ("eyelids have been opened and closed multiple times in 30 seconds") stored in the index information storage area 13A (see FIG. 6) of the storage unit 13. The specified time is "30 seconds" according to the information stored in the index information storage area 13A. This specified time is not limited to "30 seconds" and can be changed as appropriate, as described above.

[0059] If the central processing unit 12 determines that the eyelids were opened and closed multiple times within the specified time, it proceeds to step S108. If the central processing unit 12 determines that the eyelids were not opened and closed multiple times within the specified time, it returns to step S102. Note that in step S107, as in the above-described modified example performed in the processing of step S102, it is also possible to determine "whether the eyelids were opened and closed multiple times within the specified time" by taking into account the above-described analysis result information. This configuration can further improve the accuracy of awakening determination.

[0060] (Step S108) In step S108, the central processing unit 12 performs a process of determining that the infant B is in an "awake state" (awake state determination process). After making the above-described wakefulness determination, the central processing unit 12 advances the process to step S109.

[0061] (Step S109) In step S109, the central processing unit 12 performs a process of determining whether or not the state has transitioned from the "asleep state" to the "awake state." If the central processing unit 12 determines that a state transition has occurred, it shifts the process to step S110, and if it determines that a state transition has not occurred, it terminates the sleep / wake determination process and shifts the process to step S200 (see FIG. 3).

[0062] (Step S110) In step S110, the central processing unit 12 performs a process (notification process) of transmitting information indicating that infant B has transitioned from the "sleeping state" to the "awake state" to the user terminal 30. This allows the user to know that infant B has woken up through the user terminal 30 (for example, a displayed image and sound). After performing the notification process, the central processing unit 12 moves the process to step S111.

[0063] (Step S111) In step S111, the central processing unit 12 performs a process of storing the date and time when the wakefulness determination process in step S108 was performed in the sleep / wakefulness information storage area 13B of the storage unit 13 (see "Sleep / wakefulness determination information" in Figure 6). After carrying out the above-mentioned storage process, the central processing unit 12 ends the sleep / wake determination process and moves the process to step S200 (see FIG. 3).

[0064] (Step S112) In step S112, the central processing unit 12 performs processing to determine whether or not a body movement of the infant B has been detected based on the posture and body movement data acquired by the body movement sensor 22. If the central processing unit 12 determines that the body movement of the infant B has been detected, it returns to step S101, and if it determines that the body movement of the infant B has not been detected, it moves the process to step S113.

[0065] When the central processing unit 12 determines that body movement of infant B has been detected, it can calculate (for example, by integrating) the orientation and direction of infant B's body based on the posture and body movement data detected by the body movement sensor 22, and determine (estimate) the position of infant B's face, before returning the process to step S101. In such a case, if the image capturing device 21 is of a type whose image capturing direction can be changed by remote control (for example, the above-mentioned automatic tracking type), the image capturing direction of the image capturing device 21 can be moved toward that position by remote control by the central processing unit 12. As a result, the central processing unit 12 can detect (recognize) an image of infant B's eyelids in the subsequent process of step S101 ("Yes" in the process of "step S101").

[0066] (Step S113) In step S113, the central processing unit 12 performs a process of determining that infant B is in an "abnormal state" (abnormality determination). This abnormality determination is made when the image data of infant B's face captured by the imaging device 21 does not include an image of the eyelids (step S101) and no body movement of infant B is detected by the body movement sensor 22 (step S113). Such a state of infant B may be, for example, when the face is covered by a blanket or the like and the body is not moving, i.e., when there is a risk of some abnormality occurring in infant B. After making the above-mentioned abnormality determination, the central processing unit 12 ends the sleep / wake determination process and moves the process to step S200 (see FIG. 3).

[0067] (Step S200) As shown in FIG. 3, in this embodiment, step S200 is executed after the subroutine of step S100 ("sleep / wake determination process" in FIG. 4) is executed. 2 and 3, the central processing unit 12 performs a process of determining whether or not an abnormality determination has been made in step S200. This abnormality determination is performed in the process of step S113 (see FIG. 4) described above. If the central processing unit 12 determines that an abnormality determination has been made, it shifts the process to step S700, and if it determines that an abnormality determination has not been made, it shifts the process to step S200.

[0068] (Step S300) In step S300, the central processing unit 12 performs processing to transmit a command signal to turn off the power to the imaging device 21. As a result, monitoring of the baby B by the imaging device 21 is (temporarily) stopped. After transmitting a command signal to turn off the power to the imaging device 21, the central processing unit 12 proceeds to step S400. Note that even if the imaging device 21 is stopped by performing the processing of step S300, it is preferable that the user be able to view the state of the baby B by remotely operating the imaging device 21 via the user terminal 30.

[0069] (Step S400) In step S400, the central processing unit 12 performs a process of determining whether or not the body movements of infant B are on a decreasing trend. Specifically, the central processing unit 12 determines whether or not the body movements of infant B are on a decreasing trend based on the posture and body movement data transmitted in real time from the body movement sensor 22. If the central processing unit 12 determines that the body movements of infant B are on a decreasing trend, it proceeds to step S500, and if it determines that the body movements of infant B are not on a decreasing trend, it repeatedly executes the processing of step S400 until the body movements begin to decrease.

[0070] (Step S500) In step S500, the central processing unit 12 performs processing to send a command signal to turn on the power to the imaging device 21. As a result, the imaging device 21, which was stopped in the processing of step S300, is activated, and monitoring of infant B by the management server 10 using the imaging device 21 is resumed.

[0071] As described above, in this embodiment, the monitoring of infant B using the image capturing device 21 by the management server 10 is temporarily stopped only when infant B is in an "awake state" (see "steps S108 to S111" in FIG. 4) and body movements are not decreasing (see "step S400" in FIG. 3), that is, only when there is little risk to infant B while sleeping. This not only extends the life of the image capturing device 21, but also reduces the monitoring burden (control burden) on the management server 10. After transmitting a command signal to turn on the power to the imaging device 21, the central processing unit 12 moves the process to step S100 ("step S101" in FIG. 4).

[0072] (Step S600) The central processing unit 12 performs an abnormality determination process in step S600. This abnormality determination process is executed when a positive determination (a determination of "Yes") is made in the process of step S200, that is, when an image of the eyelids of infant B cannot be recognized (see "step S101" in FIG. 4) and when a body movement of infant B cannot be detected ("step S113" in FIG. 4).

[0073] Specifically, the central processing unit 12 performs processing to transmit information indicating the occurrence of an abnormality, such as information such as "Infant B's body movement: no" and "Infant's eyelids: not detected," to the user terminal 30. If infant B's posture (e.g., face down) can be estimated based on the posture and body movement data detected by the body movement sensor 22, it is preferable to also transmit this information to the user terminal 30. This allows the user, even if they are in a location far from infant B, to be informed in real time through the user terminal 30, via voice, a display screen, or the like, that some kind of abnormality has occurred. In this case, the user can remotely control the imaging device 21 using the user terminal 30, thereby viewing the state of infant B on the display unit of the user terminal 30.

[0074] Furthermore, in step S600, the central processing unit 12 can store the details of the abnormality determination, the date and time when the determination was made, etc. in the storage unit 13 (such as the sleep / wake information storage area 13B). With this configuration, the user can later check the date and time when the abnormality occurred in infant B, the details, etc. After performing the abnormality determination process, the central processing unit 12 ends the infant monitoring process. Note that in this embodiment, the infant monitoring process is ended after performing the abnormality determination process, but it is also possible to return to the process of step S100 again without ending the infant monitoring process.

[0075] As described above, in this embodiment, the facial image including the eyelids of infant B captured by the imaging device 21 is compared with index information (see Figure 5) representing the opening and closing movement of the eyelids to determine whether infant B is in a "sleeping state" or an "awake state." Generally, the eyelids are a part of the human body that necessarily move when the state transitions between "asleep" and "wakefulness," and in this embodiment, sleep / wakefulness determination is performed based on the opening and closing of the eyelids. Moreover, the subjects to be determined are infants who, unlike conscious adults, do not, for example, pretend to be asleep (a behavior contrary to physiological phenomena) when in the "wakefulness" state. In this regard, according to this embodiment, it is possible to accurately determine whether infant B is "asleep" or "awake" at the time when the infant B transitions between the "asleep" and "awake" states.

[0076] In addition, in this embodiment, as index information for determining whether the state is "sleeping" or "awake," information such as "sleeping" ("sleeping") if "eyelid opening and closing movements: no for 30 seconds or more" and "awake" ("awake") if "eyelid opening and closing movements: multiple times in 30 seconds" is stored in the memory unit 13 (index information memory area 13A). In other words, in this embodiment, the sleep / wake state is determined based on an index (numerical value) derived from the brain waves, etc., of infant B who is in a "sleeping state" or "wakeful state," so that more accurate determination results can be obtained.

[0077] [Variation 1] In this embodiment, the sleep / wake state determination is made mainly based on the "eyelid opening / closing movement" of infant B (see "step S102" and "step S107" in Figure 4), but such a determination can also be made taking into account posture / body movement data (posture / body movement) detected by the body movement sensor 22. Such a modified example will be described below with reference to Fig. 5. As described above, the above embodiment shows an example in which information such as "Determination result: Asleep" ("Sleeping state") is stored in each of the two "Determination result: ..." sections in the index information storage area 13A shown in Fig. 5.

[0078] In this modification, information such as "Determination result: awake" ("awake state") is stored in each of the two "Determination result: . . . " portions of the index information storage area 13A shown in FIG. With this configuration, for example, if "there is no eyelid opening / closing movement for a specified time (30 minutes) or longer," in the above embodiment, the state is determined to be "asleep" regardless of the presence or absence of body movement of infant B (see FIG. 1) (see "steps S102" and "step S103" in FIG. 4), but in this modified example, a different determination is made, such as "wakefulness" if there is body movement of infant B, and "sleeping" if there is no body movement. In other words, with this configuration, it is possible to reliably determine whether infant B is "asleep" or "awake" by taking into account "body movement" in addition to "the state of opening / closing of eyelids" of infant B, thereby making it possible to obtain more accurate determination results.

[0079] [Variation 2] Furthermore, in the above embodiment and variant examples, the determination of "asleep" or "awake" was made mainly based on the opening and closing of the eyelids of infant B (see Figure 1), but such a determination can also be made based on, for example, the facial expression of infant B. In such a case, a program that can extract and analyze facial expressions for each predefined action unit (hereinafter referred to as "AU") from a facial image, evaluate the action strength of the "AU" and encode it, such as a well-known FACS estimation automatic program ("Affdex" by Affctiva, Inc.), can be used. Examples of the "AU" include "AU5" (raising upper eyelid), "AU7" (tensing eyelid), "AU42" (squinting eyes), "AU43" (closing eyes), and "AU44" (narrowing eyes).

[0080] In other words, by using such a program, the movements of each "AU" can be quantified from the facial image acquired by the imaging device 21, and by storing the numerical values ​​as index information in the memory unit 13, it is possible to determine whether infant B is "asleep" or "awake" based on the facial expression of the infant. For example, the storage unit 13 (for example, the index information storage area 13A in FIG. 5) may store index information such as "judgment result: awake" when "AU43 (eyes closed): 3 or more". With this configuration, it is possible to determine whether the baby B is "asleep" or "awake" based on subtle (minute) facial expressions, thereby making it possible to obtain highly accurate determination results.

[0081] [Variation 3] In the above embodiment and each modification, the state of both the "asleep state" and the "wakeful state" is determined, but it is also possible to determine only one of the states.

[0082] The present invention has been described above based on the preferred embodiments and modifications thereof, but the present invention is not limited to the above-described embodiments and modifications.

[0083] In relation to the above-described embodiments, the present invention further discloses the following apparatus and method. <1> A sleep determination system for determining whether an infant is in a sleeping state or a waking state, comprising: a facial movement detection means for detecting a facial movement of an infant based on a facial image including an eyelid opening and closing movement acquired by an imaging device; a storage means for storing index information including an infant's facial movement index indicating a state transition between the sleeping state and the awake state; a determination means for comparing the facial movement detected by the facial movement detection means with the index information stored in the storage means to determine whether the subject is in the sleeping state or the awake state; A sleep determination system comprising: <2> the index information includes a facial movement index that indicates the awake state when there are multiple eyelid opening and closing movements within a specified time, and the sleeping state when there are no eyelid opening and closing movements within the specified time. <1> The sleep determination system according to claim 1. <3> further comprising an input means for receiving input of sleep information indicating that the infant is in the sleeping state; the facial movement detection means is configured to detect an eyelid opening degree of the infant in response to the input of the sleep information by the input means, the storage means stores, as training data, eyelid opening information indicating an eyelid opening degree of the infant detected by the facial movement detection means; The determining means determines whether the state is the sleeping state or the awake state based on the opening degree information stored in the storage means. <1> The sleep determination system according to claim 1. <4> Further provided is a posture / body movement detection means for detecting the posture / body movement of the infant, The index information includes an action index based on a combination of an eyelid opening / closing action of the infant and a posture and body movement, the determining means determines whether the subject is in the sleeping state or the awake state based on the facial movement detected by the facial movement detecting means, the posture and body movement detected by the posture and body movement detecting means, and the index information stored in the storage means; <1> The sleep determination system according to claim 1. <5> The face movement detection means is configured to operate when the body movement detected by the posture and body movement detection means shows a decreasing tendency. <4> The sleep determination system according to claim 1. <6> The device further comprises a notification means for notifying an abnormality when neither the eyelid opening / closing movement by the face movement detection means nor the body movement by the posture / body movement detection means is detected. <4> or <5> The sleep determination system according to claim 1.

[0084] <7> the facial movement detection means detects facial movements of the infant in real time; the determining means continuously determines whether the subject is in the sleeping state or the awake state based on the facial movement detected by the facial movement detecting means; <1> ~ <6> 10. The sleep determination system according to claim 9, wherein the sleep determination system is a sleep determination system. <8> The sleep determination system further includes an output unit configured to output determination information indicating a determination result determined by the determination unit to an external device. the output means outputs the determination information when the determination result determined by the determination means indicates a state transition between the sleeping state and the awake state; <1> ~ <7> 10. The sleep determination system according to claim 9, wherein the sleep determination system is a sleep determination system. <9> The storage means stores determination information indicating the determination result determined by the determination means and time information indicating the date and time of the determination in association with each other. <1> ~ <8> 10. The sleep determination system according to claim 9, wherein the sleep determination system is a sleep determination system. <10> The storage means includes: As information indicating a state transition from the awake state to the sleep state, when there is an eyelid opening / closing action and when there is the posture / body movement, the action indicator indicating the awake state is stored, and The action indicator that indicates the state transition from the sleeping state to the awake state when the eyelids are opened and closed multiple times within a specified time is stored. <4> ~ <6> 10. The sleep determination system according to claim 9, wherein the sleep determination system is a sleep determination system. <11> the sleep determination system further comprises an eyelid opening calculation means for calculating an eyelid opening degree of the infant in the sleeping state based on the facial movement detected by the facial movement detection means; the eyelid opening degree calculation means calculates the eyelid opening degree of the infant in the sleeping state by dividing the distance between the highest point of the upper eyelid and the lowest point of the lower eyelid in the sleeping state by the distance between the highest point of the upper eyelid and the lowest point of the lower eyelid in the awake state; the storage means stores eyelid opening information indicating the eyelid opening degree of the infant in the sleeping state calculated by the eyelid opening degree calculation means; <1> ~ <10> 10. The sleep determination system according to claim 9, wherein the sleep determination system is a sleep determination system. <12> The sleep determination system further comprises an output means for outputting abnormality information indicating an abnormality to the outside when neither the eyelid opening / closing movement by the face movement detection means nor the body movement by the posture / body movement detection means is detected. <4> or <5> The sleep determination system according to claim 1.

[0085] <13> The sleep determination system further includes an eyelid image determination means for determining whether or not an image of an eyelid is included in the face image acquired by the imaging device, the determining means determines whether the state is the sleeping state or the awake state on condition that the eyelid image determining means determines that the image contains an image of an eyelid. <1> ~ <12> 10. The sleep determination system according to claim 9, wherein the sleep determination system is a sleep determination system. <14> The storage means stores the opening degree information as teacher data. <11> The sleep determination system according to claim 1. <15> The determining means determines whether the state is the sleeping state or the awake state based on the opening degree information stored in the storage means. <11> or <14> The sleep determination system according to claim 1. <16> when the use of the sleep determination system is started when the baby is in the sleeping state, the determination means repeatedly determines the sleeping state until the baby transitions to the awake state, on condition that the facial movement of the baby is detected by the facial movement detection means; <1> ~ <15> 10. The sleep determination system according to claim 9, wherein the sleep determination system is a sleep determination system. <17> The device further comprises a face position estimation means for estimating the position of the face of the infant based on the posture and body movement detected by the posture and body movement detection means. <4> ~ <6> 10. The sleep determination system according to claim 9, wherein the sleep determination system is a sleep determination system. <18> the face position estimation means estimates the position of the face of the infant when the face movement detection means does not detect an eyelid opening / closing movement; <17> The sleep determination system according to claim 1. <19> When the posture of the infant can be estimated based on the past body movement and posture detected by the posture and body movement detection means, the output means outputs the abnormality information and posture information indicating the posture of the infant to the outside. <12> The sleep determination system according to claim 1.

[0086] <20> A sleep determination method for determining whether an infant is in a sleeping state or an awake state using a sleep determination system, comprising: a facial movement detection step of detecting a facial movement of the infant based on a facial image including an eyelid opening and closing movement acquired by an imaging device; a determining step of comparing the facial movement detected by the facial movement detecting step with index information including a facial movement index of the infant indicating a state transition between the sleeping state and the awake state, and determining whether the infant is in the sleeping state or the awake state; A sleep determination method comprising: <21> A sleep determination program for determining whether an infant is in a sleeping state or an awake state using a sleep determination system, comprising: The sleep determination system includes: a storage means for storing index information including an infant's facial movement index indicating a state transition between the sleeping state and the awake state; The sleep determination program a facial movement detection step of detecting a facial movement of the infant based on a facial image including an eyelid opening and closing movement acquired by an imaging device; a determining step of comparing the facial movement detected by the facial movement detecting step with the index information stored in the storage means to determine whether the state is the sleeping state or the awake state; A sleep determination program that causes a computer to execute the above. [Explanation of symbols]

[0087] 1. Sleep detection system 2 Public communication lines 10 Management Server 11 Control device 12 Central Processing Unit 13 Storage section 13A Index information storage area 13B Sleep / wake information storage area 14 Input section 15 Display section 16 Communications Department 21 Imaging device 22 Body movement sensor B. Infants E:Eye C1,C2:Distance Lt: Upper eyelid Lb: lower eyelid

Claims

1. A sleep determination system for determining whether an infant is in a sleeping state or a waking state, comprising: a facial movement detection means for detecting a facial movement of an infant based on a facial image including an eyelid opening and closing movement acquired by an imaging device; a storage means for storing index information including an infant's facial movement index indicating a state transition between the sleeping state and the awake state; a determination means for comparing the facial movement detected by the facial movement detection means with the index information stored in the storage means to determine whether the subject is in the sleeping state or the awake state; A sleep determination system comprising:

2. The sleep determination system according to claim 1 , wherein the index information includes a facial movement index that determines the awake state when there are multiple eyelid opening and closing movements within a specified time period, and the sleep state when there are no eyelid opening and closing movements within the specified time period.

3. further comprising an input means for receiving input of sleep information indicating that the infant is in the sleeping state; the facial movement detection means is configured to detect an eyelid opening degree of the infant in response to the input of the sleep information by the input means, the storage means stores, as training data, eyelid opening information indicating an eyelid opening degree of the infant detected by the facial movement detection means; The sleep determination system according to claim 1 , wherein the determination means determines whether the state is the sleeping state or the awake state based on the opening degree information stored in the storage means.

4. Further provided is a posture / body movement detection means for detecting the posture / body movement of the infant, the index information includes a movement index based on a combination of an eyelid opening / closing movement of the infant and a posture / body movement, 2. The sleep determination system according to claim 1, wherein the determination means determines whether the subject is in the sleeping state or the awake state based on the facial movement detected by the facial movement detection means, the posture and body movement detected by the posture and body movement detection means, and the index information stored in the storage means.

5. 5. The sleep determination system according to claim 4, wherein the facial movement detection means is configured to operate when the body movement detected by the posture / body movement detection means shows a decreasing trend.

6. 6. The sleep determination system according to claim 4, further comprising a notification unit that notifies an abnormality when neither the eyelid opening / closing movement by the facial movement detection unit nor the body movement by the posture / body movement detection unit is detected.

7. the facial movement detection means detects facial movements of the infant in real time; The sleep determination system according to any one of claims 1 to 6, wherein the determination means continuously determines whether the subject is in the sleeping state or the awake state based on the facial movement detected by the facial movement detection means.

8. The sleep determination system further includes an output unit configured to output determination information indicating a determination result determined by the determination unit to an external device. The sleep determination system according to any one of claims 1 to 7, wherein the output means outputs the determination information when the determination result determined by the determination means indicates a state transition between the sleep state and the wakefulness state.

9. The sleep determination system according to any one of claims 1 to 8, wherein the storage means stores determination information indicating the determination result determined by the determination means and time information indicating the determination date and time in association with each other.

10. A sleep determination method for determining whether an infant is in a sleeping state or an awake state using a sleep determination system, comprising: a facial movement detection step of detecting a facial movement of the infant based on a facial image including an eyelid opening and closing movement acquired by an imaging device; a determining step of comparing the facial movement detected by the facial movement detecting step with index information including an infant's facial movement index indicating a state transition between the sleeping state and the awake state, and determining whether the infant is in the sleeping state or the awake state; A sleep determination method comprising:

11. A sleep determination program for determining whether an infant is in a sleeping state or an awake state using a sleep determination system, comprising: The sleep determination system includes: a storage means for storing index information including an infant's facial movement index indicating a state transition between the sleeping state and the awake state; The sleep determination program a facial movement detection step of detecting a facial movement of the infant based on a facial image including an eyelid opening and closing movement acquired by an imaging device; a determining step of comparing the facial movement detected by the facial movement detecting step with the index information stored in the storage means to determine whether the state is the sleeping state or the awake state; A sleep determination program that causes a computer to execute the above.

Images