Information processing apparatus, information processing method and program

Abstract

To provide an information processing device which outputs an authentication result by performing face authentication of a user with high accuracy in real time without deteriorating a processing capacity when a mask wearer exists in persons accompanying the user of the room.SOLUTION: An information processing apparatus 1 comprises: a sensor 1a which detects entry to a room; an imaging device 1b which obtains image data obtained by imaging an entering person when the entry is detected; a determination unit 1c; a face authentication unit 1d; and an output unit 1e which outputs a result of face authentication processing to a notification destination. The determination unit 1c temporarily determines whether or not the entering person is an authentication object person as a user of the room by determining the presence / absence of mask wearing of the entering person on the basis of image data obtained by the imaging device 1b. The face authentication unit 1d executes face authentication processing by comparing face image data for the authentication object person with feature data indicating the feature of the face that has been stored in advance for each authentication object person and each accompanying person that may enter the room accompanying the authentication object person when the entering person is the authentication object person based on the result of temporary determination.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Caregivers at nursing facilities and homes monitor the condition of those receiving care (those requiring care) to understand their condition and provide assistance. This type of monitoring work places a heavy burden on caregivers, and support is needed to reduce this burden. One way to provide support is to automate the monitoring process as much as possible, but in order to automate this, it is necessary to at least identify the care recipient.

[0003] Regarding toilets, Patent Document 1 describes a technology that collects excretion information indicating the contents of excrement excreted into the toilet bowl without the need to ask the toilet user, and prevents foreign objects other than excrement from being flushed even if they are found in the toilet bowl. In the technology described in Patent Document 1, a server receives face image data captured by a camera, performs face recognition processing by comparing the features of the data with pre-stored authentication data, and obtains identification data associated with the matching authentication data, thereby identifying the toilet user. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2021 / 024584 Summary of the Invention [Problem to be solved by the invention]

[0005] As described above, caregivers monitor and assist the condition of those in need of care, and the burden on caregivers can be reduced by introducing a system in which a server performs face recognition processing based on image data to identify toilet users, as in the technology described in Patent Document 1. However, in this case, face image data acquired by the camera is stored on the server. Since information about toilet use is sensitive information, leaving the face image data on the server is not appropriate because it places a heavy mental burden on users and caregivers. Therefore, it is desirable to use a small device (toilet sensor) that has face recognition, notification, and excrement analysis functions and is installed in the toilet, so that face image data does not leak onto a network such as an external server.

[0006] When such a configuration is adopted, the person who enters the restroom is identified by a device with a face recognition function installed in the restroom, and the caregiver is notified of the user's entry. In this case, the face image data used for face recognition is not stored externally, which reduces the mental burden on the user and caregiver, but face recognition using a toilet sensor causes the following major problems (1) and (2).

[0007] (1) Processing power issues arise due to the need to perform multiple facial recognition tests simultaneously. (2) The problem of increased rates of misidentification due to caregivers wearing masks.

[0008] Regarding (1) above, toilet space is generated when there is a user who cannot enter or leave the toilet without the assistance of a caregiver. When using the toilet, if one or more caregivers enter the toilet to accompany the user, multiple face recognitions must be performed simultaneously when both the user and the caregiver enter the imaging range of the face recognition camera. Therefore, in such cases, the accuracy of identifying the user decreases and the processing time increases.

[0009] Regarding (2) above, in addition to the problem of (1) above, in nursing facilities, users do not wear masks, and although caregivers usually do not wear masks either, they may wear masks during epidemic periods to prevent infection. In this case, when assisting with excretion, the user and the caregiver wearing a mask will be within the imaging range of the face recognition camera. Recognition of a face wearing a mask cannot use the facial information of the part of the face covered by the mask, and matching is performed from a small amount of feature information, which is likely to cause erroneous recognition, and this causes deterioration of the recognition accuracy. As a result, a problem may occur in which a face that is actually a caregiver is mistakenly recognized as that of the user.

[0010] Furthermore, since personal identification through facial recognition needs to be linked with a function to notify caregivers, such as for toilet assistance or rushing to prevent accidents, not only accuracy but also real-time performance is required, so it is desirable to improve the processing power and accuracy of facial recognition processing. On the other hand, if such processing is to be performed without transmitting facial image data over a network, it is desirable to realize the toilet sensor as an edge device incorporating a space-saving and power-saving CPU (Central Processing Unit). Therefore, it is desirable to improve the processing power and accuracy of facial recognition processing by methods other than improving the performance of CPUs.

[0011] As described above, while improvements have been made to install toilet sensors to reduce the burden of excretion management in caregiving, there have been problems with the accuracy and processing power when identifying users who require caregiver assistance through face recognition. In addition, such problems can also occur when managing entry to various rooms, not just toilets, and when entry may occur with an attendant.

[0012] An object of the present disclosure is to provide an information processing device, an information processing method, and a program that solve the above-mentioned problem. The above-mentioned problem is to make it possible to perform face authentication of a user in real time with high accuracy and without reducing processing power when a person accompanying a user of a room is wearing a mask, and to output the authentication result. [Means for solving the problem]

[0013] An information processing device according to a first aspect of the present disclosure includes a sensor that detects the entry of a person into a room, and an imaging device that captures an image of the person who has entered the room and obtains image data when the sensor detects the entry of the person. The information processing device includes a determination unit that determines whether the person who has entered the room is wearing a mask based on the image data obtained by the imaging device, thereby provisionally determining whether the person who has entered the room is a person to be authenticated as a user of the room or not. The information processing device includes a face authentication unit that performs face authentication processing based on face image data of the person to be authenticated in the image data when the person is a person to be authenticated as a result of the provisional determination by the determination unit, and an output unit that outputs the result of the face authentication processing to a notification destination. The face authentication unit performs the face authentication processing by comparing the face image data with feature data indicating face features stored in advance for each person to be authenticated and for each attendant who may enter the room accompanying the person to be authenticated.

[0014] In an information processing method according to a second aspect of the present disclosure, a sensor detects the entry of a person into a room, and when the sensor detects the entry of a person, an imaging device captures an image of the person who entered the room to obtain image data. The information processing method provisionally determines whether the person who entered the room is a person to be authenticated as a user of the room or is the person to be authenticated by determining whether the person is wearing a mask based on the image data obtained by the imaging device. When the provisional determination result indicates that the person is the person to be authenticated, the information processing method executes a face authentication process based on face image data of the person to be authenticated in the image data, and outputs the result of the face authentication process to a notification destination. The face authentication process is executed by comparing the face image data with feature data indicating face features stored in advance for each person to be authenticated and for each attendant who may enter the room accompanying the person to be authenticated.

[0015] A program according to a third aspect of the present disclosure is a program for causing a computer to execute information processing. The information processing inputs a result of detection of a person entering a room by a sensor, and when the sensor detects the entry of a person, causes an imaging device to capture an image of the person who entered the room to obtain image data. The information processing determines whether the person who entered the room is wearing a mask based on the image data obtained by the imaging device, thereby provisionally determining whether the person who entered the room is a person to be authenticated as a user of the room or is the person to be authenticated. When the result of the provisional determination indicates that the person is the person to be authenticated, the information processing executes a face authentication process based on face image data of the person to be authenticated in the image data, and outputs the result of the face authentication process to a notification destination. The face authentication process is executed by comparing the face image data with feature data indicating face features stored in advance for each person to be authenticated and for each attendant who may enter the room accompanying the person to be authenticated. Effect of the Invention

[0016] The present disclosure makes it possible to provide an information processing device, an information processing method, and a program that, when a person accompanying a user of a room is wearing a mask, can perform facial authentication of the user in real time and with high accuracy without reducing processing power, and output the authentication results. [Brief description of the drawings]

[0017] [Figure 1] 1 is a block diagram showing an example of the configuration of an information processing device according to a first embodiment. [Diagram 2] FIG. 11 is a diagram illustrating an example of a configuration of an information processing system including an information processing device according to a second embodiment. [Diagram 3] 3 is a side view showing an example of installation of an information processing device in the information processing system of FIG. 2. [Figure 4] 3 is a block diagram showing an example of a configuration of an information processing device in the information processing system of FIG. 2. [Diagram 5] 3 is a schematic diagram for explaining an example of a processing flow for generating and registering face model data in the information processing system of FIG. 2. [Figure 6] FIG. 5 is a schematic diagram for explaining an example of face authentication processing in the information processing device of FIG. 4 when a caregiver is not wearing a mask. [Figure 7] FIG. 5 is a schematic diagram for explaining an example of face authentication processing in the information processing device of FIG. 4 when a caregiver is wearing a mask. [Figure 8] 3 is a diagram for explaining an example of a notification in a terminal device in the information processing system of FIG. 2. [Figure 9] 5 is a flow diagram for explaining an example of face authentication processing in the information processing device of FIG. 4. FIG. [Figure 10] FIG. 11 is a schematic diagram showing a configuration example of an information processing device according to a third embodiment. [Figure 11] FIG. 2 illustrates an example of a hardware configuration of the apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment will be described with reference to the drawings. In the embodiments, the same or equivalent elements may be denoted by the same reference numerals, and duplicated descriptions will be omitted as appropriate.

[0019] Hereinafter, an embodiment will be described with reference to the drawings. In the embodiments, the same or equivalent elements may be denoted by the same reference numerals, and duplicated descriptions will be omitted as appropriate.

[0020] <Embodiment 1> FIG. 1 is a block diagram showing an example of the configuration of an information processing device according to the first embodiment. As shown in FIG. 1, an information processing device 1 according to this embodiment can include a sensor 1a, an imaging device 1b, a determination unit 1c, a face authentication unit 1d, and an output unit 1e.

[0021] The sensor 1a is a sensor that detects the entry of a person into a room that is a target of entry monitoring, and can be a human presence sensor provided at a position where it can detect the entry. The room that is a target of entry monitoring can be various rooms such as a toilet cubicle or a bathroom.

[0022] In the following, an example will be described in which the authentication subject, who is the person to be monitored for entry, is a care recipient, that is, an example in which the room to be monitored for entry is a room used by the care recipient, but other types of authentication subjects can also be the subject of entry monitoring. Here, the authentication subject refers to a person for whom face authentication is performed by the face authentication unit 1d and the results are output. A care recipient can also be called a person receiving care, a person requiring care, or a person requiring assistance. A caregiver can be a caregiver, a doctor, or the like, and can also be called a caregiver.

[0023] When the authentication subject is a care recipient, the rooms to be monitored include the above-mentioned toilet, bathroom, etc. The information processing device 1 can be used in places where it is necessary to watch over care recipients, such as elderly care facilities, private homes, and hospitals.

[0024] Examples of the sensor 1a include an infrared sensor and an optical sensor. The sensor 1a is not limited to infrared, and may be a sensor using any method, such as image recognition or ultrasonic waves. It is preferable that the sensor 1a is also capable of detecting a person leaving a room after entering it. In a simple example, the sensor 1a can detect that a person has left the room when a person is detected after entering the room.

[0025] When the sensor 1a detects the entry of a person into the room, the imaging device 1b captures the person and obtains image data. The imaging device 1b can be, for example, a camera that captures still images and / or videos, and can be called a face authentication camera because the captured image data is used by the face authentication unit 1d in the subsequent stage. The imaging device 1b is not limited to a visible light camera, and may be an infrared light camera, etc.

[0026] The determination unit 1c determines whether the person who has entered the room is wearing a mask based on the image data obtained by the imaging device 1b, and provisionally determines whether the person who has entered the room is a person to be authenticated as a user of the room (a care recipient in this example) or is a person to be authenticated. The determination unit 1c provisionally determining that the person is not a care recipient means that the person is provisionally determined to be a care recipient. The determination unit 1c can also be configured to determine whether the person who has left the room is a care recipient.

[0027] If the result of the provisional determination by the determination unit 1c is that the person is a person to be authenticated (a person receiving care in this example), that is, if the person is not wearing a mask, the face authentication unit 1d executes face authentication processing based on the face image data of the person to be authenticated in the image data. If the image data includes face image areas of multiple people, the face authentication processing may be executed based on the data of the face image areas of the person to be authenticated among all the people.

[0028] Here, the face authentication unit 1d executes the face authentication process by comparing the face image data with feature data indicating facial features stored in advance for each person to be authenticated (a person receiving care in this example) and for each attendant (a caregiver in this example) who may accompany the person to be authenticated and enter the room. Therefore, if the result of the provisional determination by the determination unit 1c indicates that the person is an attendant (a caregiver in this example), that is, a person wearing a mask, the face authentication unit 1d does not execute the face authentication process and ends the subsequent processes.

[0029] The feature data may be, for example, data showing facial feature points. This comparison may also be performed by inputting facial image data of the care recipient who has entered the room into a learning model (trained model) that has been machine-learned using learning data including the feature data and the image data (facial image data) that is the source of the feature data.

[0030] Then, by such face recognition processing, it is possible to specify the person who entered the target room from the face image data of the person who became the authentication target by the provisional judgment, that is, to specify (identify) the individual who entered the target room. The person specified here is the result of the above-mentioned comparison, and may be an attendant by the provisional judgment. The face recognition unit 1d can also be called an identification unit because it specifies the individual in this way.

[0031] The determination unit 1c and the face authentication unit 1d may have some common components. For example, the determination unit 1c may use intermediate data in the recognition process in the face authentication unit 1d. In other words, the determination unit 1c may be included as part of the functions of the face authentication unit 1d.

[0032] In addition, it is possible that the users of the room may be people who do not fall into the category of either a caregiver or a care recipient (such as family members visiting a patient), but the facial recognition process for such people will fail and they will be determined to be unknown people.

[0033] The output unit 1e outputs the result of the face recognition process to the notification destination. The output unit 1e can be, for example, a communication unit configured with a wired or wireless communication interface. The output unit 1e can transmit, for example, information indicating the result of the face recognition process to a terminal device (not shown) of the notification destination via the communication unit. The output unit 1e can transmit this information to an external server (not shown) via the communication unit, for example, so that the server can transfer the notification information to the terminal device of the notification destination. The notification destination can be set in advance, and it is preferable that the notification destination is changeable.

[0034] Furthermore, the output unit 1e can output the result of the face authentication process to the notification destination regardless of the result of the face authentication process. However, when the output unit 1e authenticates an attendant (a caregiver in this example) in the face authentication process, the output unit 1e can also output the result of the face authentication process to the notification destination, excluding the result of the face authentication process for the attendant. In other words, the output unit 1e can output the result to the notification destination when the person to be authenticated (the person receiving care in this example) is authenticated, and can not output the result when the attendant is authenticated. Furthermore, in the case of an unknown person, the output unit 1e can either notify that an unknown person has entered the room or not notify, and it is sufficient to determine in advance which process to execute.

[0035] The information processing device 1 may also have a control unit (not shown), which may include the above-mentioned determination unit 1c, face authentication unit 1d, and output unit 1e (or a part thereof). The control unit may be realized, for example, by a CPU (Central Processing Unit), a working memory, and a non-volatile storage device storing a program. The program may be a program for causing the CPU to execute a process of inputting a detection result from the sensor 1a, a process of inputting image data captured by the imaging device 1b, and the processes of the determination unit 1c, face authentication unit 1d, and output unit 1e. The control unit provided in the information processing device 1 may also be realized, for example, by an integrated circuit. The above-mentioned storage device may also store a notification destination, and the output unit 1e may output the result of the face authentication process by referring to the notification destination.

[0036] In addition, the information processing device 1 according to the present embodiment is assumed to be configured as a single information processing device, but it can also be configured as multiple devices with distributed functions. In the latter case, each device is provided with a control unit, a communication unit, and a storage unit, etc., as necessary, and these multiple devices are connected by wireless or wired communication as necessary to cooperate and realize the functions of the information processing device 1.

[0037] Here, the information processing method according to the present embodiment will be briefly described. This information processing method performs the following information processing. In the above information processing, the sensor 1a detects the entry of a person into a room, and when the sensor 1a detects the entry of a person, the imaging device 1b captures the person who entered the room to obtain image data. Furthermore, the above information processing determines whether the person who entered the room is wearing a mask based on the image data obtained by the imaging device 1b, thereby provisionally determining whether the person who entered the room is a person to be authenticated as a user of the room or not. Furthermore, in the above information processing, if the person is a person to be authenticated as a result of the provisional determination, a face authentication process is performed based on face image data of the person to be authenticated in the image data, and the result of the face authentication process is output to the notification destination. The face authentication process is performed by comparing the face image data with feature data indicating face features stored in advance for each person to be authenticated and each attendant. Here, when an attendant is authenticated as a result of the face authentication process, the output to the notification destination can also be performed by excluding the result of the face authentication process for the attendant. Furthermore, the above-mentioned program can be a program for causing a computer to execute the above-mentioned information processing.

[0038] As described above, in this embodiment, when a person accompanying a user of a room (e.g., an attendant such as a caregiver) is wearing a mask, facial authentication of the user can be performed in real time with high accuracy without reducing processing power, and the authentication result can be output.

[0039] For example, in this embodiment, the system has a function of performing face recognition processing by preregistering facial feature data of the caregiver and a mask detection function, so that a highly accurate face recognition processing can be realized even with a space-saving, power-saving CPU built into an information processing device as an edge device. To give a more specific example, in this embodiment, when a caregiver wearing a mask enters the toilet for cleaning the toilet, the caregiver can be excluded from the face recognition target by simply performing a mask determination process, which is a light processing compared with the face recognition process, by using these functions. Therefore, in this embodiment, even when multiple caregivers are assisting one user, the face recognition processing can be performed without significantly impairing performance. In addition, in this embodiment, a configuration can be adopted in which a notification is not issued even when a caregiver not wearing a mask enters the toilet.

[0040] <Embodiment 2> 2 to 9, the second embodiment will be described focusing on the differences from the first embodiment, but the various examples described in the first embodiment can be applied. Fig. 2 is a diagram showing a configuration example of an information processing system including an information processing device according to the second embodiment, and Fig. 3 is a side view showing an example of installation of the information processing device in the information processing system of Fig. 2.

[0041] An information processing system according to this embodiment (hereinafter, this system) can be a system in which a part is installed in a toilet and which notifies a notification destination (for example, a caregiver, etc.), and will be specifically described below.

[0042] As shown in Fig. 2, the present system can include an information processing device 60 attached to the toilet bowl 10, a server 40 wirelessly connected to the information processing device 60, and a terminal device 50 wirelessly connected to the server 40. These connections can be made, for example, within one wireless LAN (Local Area Network). For example, the present system can also be configured so that the information processing device 60 can directly transmit information to the terminal device 50. In addition, the following description will be given using an example in which the information processing device 60 is installed in the toilet bowl 10 of a toilet and also acquires excretion information, but the excretion information can be acquired as long as at least a part of it, such as a camera for capturing image data for obtaining the excretion information, is installed in the toilet bowl 10.

[0043] The toilet bowl 10 can have an edge 10c for forming a water-retaining area between itself and the side surface 10a, and can also have a flat surface 10b continuing from the edge 10c. The flat surface 10b can be provided with a toilet seat 11 including, for example, a toilet seat body 11a equipped with a warm water cleaning function for the user to wash, and a toilet seat cover 11b for covering the toilet seat body 11a.

[0044] The information processing device 60 can be a device installed in the toilet (a toilet installed type in this example), and notifies the terminal device 50 of detection events such as the detection of the entry of a care recipient, either directly or via the server 40. Here, an example will be described in which the information processing device 60 is an excretion information collecting device with a function of acquiring excretion information. In this case, excretion-related events such as the start and end of excretion can also be notified as detection events. Although not described in detail, the information processing device 60 can also be configured to record excretion, present excretion information (notification, etc.), and predict excretion in cooperation with the server 40 and the terminal device 50. The information processing device 60 and the toilet 10 can, for example, constitute a toilet with a function of outputting excretion information.

[0045] 2, the shape of the information processing device 60 may be configured such that all or part of its functions are embedded in the toilet seat body 11a, etc. Also, part of the functions of the information processing device 60 may be provided on the toilet seat body 11a side.

[0046] 1, and has the functions of each unit of the information processing device 1. Before describing a detailed example of the information processing device 60, a brief description will be given of main components, including components corresponding to each unit of the information processing device 1 in FIG.

[0047] As shown in Figures 2 and 3, the information processing device 60 of this embodiment may include a human presence sensor 24, an imaging device 25, an information collection unit 26, a storage housing 61, a bridge unit 62, an inner housing 63, a control unit 67, a separate housing 68, and an outer housing 69.

[0048] The human presence sensor 24 is an example of the sensor 1a, and is a sensor that detects an entry into a toilet cubicle, such as a user (customer) of the toilet bowl 10, and may be, for example, an infrared sensor or an optical sensor.

[0049] The imaging device 25 is an example of the imaging device 1b, and is a device that captures the face of a person entering a toilet stall, such as a toilet user, and can be, for example, a camera that captures still images and / or videos (a face authentication camera). The imaging device 25 is not limited to a visible light camera, and may be an infrared light camera, etc.

[0050] The information collecting unit 26 is disposed in the inner housing 63. However, the information collecting unit 26 is disposed so that information can be collected from the accumulation portion (area where excrement is excreted) of the toilet bowl 10, for example, by leaving the information collecting surface of the information collecting unit 26 exposed from an opening of the inner housing 63. The information collecting surface corresponds to, for example, a lens surface if the information collecting unit 26 is a camera, or a detection surface if the information collecting unit 26 is a sensor.

[0051] The information collecting unit 26 is a part that collects information on the excrement in the toilet bowl 10, and can be, for example, an imaging device that collects the contents of the excrement, such as the shape and color of the excrement, through an optical lens, or a distance sensor that optically measures distance. The excrement information can be information indicating the contents of the excrement, and in a simpler example, information indicating whether the excrement is stool (feces) or urination (urine). The excrement information can also include other information, such as information indicating the color of the excrement, and if it is a solid object, its shape. The information collecting unit 26 may simply be configured to detect the presence or absence of excrement and collect the information on the excrement.

[0052] The information collecting unit 26 can set the area including the water-filled portion of the toilet bowl (excretion area) as the collection area, and this excretion area can also be called the planned excretion area. By installing the information collecting unit 26 so that such an excretion area is included in the imaging area, the captured imaging data will include excrement and the like as subjects. Of course, it is preferable that the excretion area is set to an area where the user is not captured, and it is also preferable that the information collecting unit 26 is installed so that the lens and the like are not visible to the user.

[0053] The information collecting unit 26, the human presence sensor 24, and the imaging device 25 are connected to the control unit 67 by wired or wireless communication. Note that Fig. 2 shows an example in which the information collecting unit 26 is connected to the control unit 67 by a cable K1, and the human presence sensor 24 and the imaging device 25 are connected to the control unit 67 by a cable K2.

[0054] The control unit 67 controls the human sensor 24, the imaging device 25, and the information collection unit 26. For example, the control unit 67 can receive the detection result of the human sensor 24, can instruct the imaging device 25 to capture an image and receive the captured image, can instruct the information collection unit 26 to collect information on excrement and receive the collected information. The control unit 67 can also be configured to execute the functions of the determination unit 1c, the face authentication unit 1d, and the output unit 1e. The control unit 67 can be realized, for example, by a CPU, a working memory, and a non-volatile storage device that stores a program. The program can be, for example, a program that causes the CPU to execute the above-mentioned processing. The control unit 67 can also be realized, for example, by an integrated circuit.

[0055] The storage housing 61 is a housing that is placed on the outside of the toilet bowl 10 and stores the control unit 67, and will be described below as the control box 61. The bridge 62 is a portion that bridges the inner housing 63 and the outer housing 69. In other words, the bridge 62 is a mechanism that connects the inner housing 63 and the outer housing 69, and is installed on the edge 10c of the toilet bowl 10. The outer housing 69 is a housing that includes a clamp mechanism for fixing the inner housing 63 and the toilet bowl 10. The outer housing 69 is a housing that is placed on the outside of the toilet bowl 10, and its shape does not matter as long as it does not interfere with excretion.

[0056] The above-mentioned clamp mechanism (fixing clamp mechanism) can be a mechanism that fixes the inner housing 63 and the outer housing 69 to the edge portion 10c, for example, by adjusting the distance between the inner side of the edge portion 10c via the bridge 62 and clamping the inner housing 63 and the outer housing 69. This allows the inner housing 63 of the information processing device 60 to be reliably fixed and installed regardless of the shape of the toilet bowl 10, such as the shape of the edge portion 10c. Furthermore, as long as it is such a mechanism, the shape and components of the fixing clamp mechanism are not important. The main components of the fixing clamp mechanism can be provided in the outer housing 69.

[0057] The inner housing 63 is a housing in which the information collecting unit 26 is disposed, and will hereinafter be described as the sensor box 63. The sensor box 63 is equipped with, for example, an optical sensor for collecting information about excrement, and is disposed inside the toilet bowl 10.

[0058] The separate housing 68 is a housing in which the human sensor 24 and the imaging device 25 are disposed. The separate housing 68 is one in which the human sensor 24 for human detection and the imaging device (camera) 25 for face authentication are disposed inside or partly outside, and will be described below as the human detection / identification box 68. The human detection / identification box 68 may be provided with a camera adjustment mechanism for adjusting the direction of the camera in the horizontal direction. The imaging device 25 may have a lens exposed from the human detection / identification box 68 so as to be able to capture an image of the face of the toilet user, but it is sufficient if it can capture a face. The human sensor 24 may have a detection surface exposed from the human detection / identification box 68 so as to be able to detect the toilet user, but it is sufficient if it can detect a person. If the human sensor 24 and the imaging device 25 are connected to the control unit 67 by wire or wireless communication, information can be transmitted and received.

[0059] The human detection / identification box 68 can be disposed at a position separated from the control box 61 and the toilet 10, as exemplified in FIG. 2 and FIG. 3. FIG. 2 and FIG. 3 show an example in which the human detection / identification box 68 is installed on the wall W of the toilet in which the toilet 10 is installed. Of course, the human detection / identification box 68 can be installed on a place other than the wall W, such as the ceiling of the room or a shelf installed in the room. The human detection / identification box 68 can adopt a wired connection with the control box 61 via a cable K2 or a wireless connection, and in either case, it can be used at a position separated from the control box 61. The human detection / identification box 68 rotates horizontally, and a mechanism for fixing the orientation of the rotating box can be provided.

[0060] In order to facilitate detection and imaging of people, the human presence sensor 24 and the imaging device 25 are preferably disposed so that the detection surface and imaging lens are located at a position at least higher than the bridge 62. It is more preferable to provide the detection surface and imaging lens at a position a predetermined height higher than the height of the toilet seat 11.

[0061] In addition, in the configuration of the information processing device 60 illustrated in Fig. 2, the control box 61 having the human detection / identification box 68 and the control unit 67, and the sensor box 63 having an excretion information collection function are connected by cables K1 and K2. However, the human detection / identification box 68 is disposed on the wall W of the toilet, and the box having the control unit 67 is disposed on the side of the toilet bowl 10, and the installation work is also easy. The control box 61 can also be disposed on the back side of the toilet bowl 10, and the installation work in this case is also easy. In addition, it is preferable that both cables K1 and K2 include a power supply line that supplies power from the control box 61. Note that power is supplied to the control box 61 from a power cable not shown.

[0062] 2, it is assumed that the cable K1 passes through the inside of the bridge 62, but the cable K1 can also be connected to the information collecting unit 26, led out from the sensor box 62, and directly connected to the control box 61. In that case, the cable K1 passes through the upper surface side of the edge 10c of the toilet bowl 10 (passes between the upper surface of the edge 10c and the toilet seat body 11a).

[0063] The server 40 may include a control unit 41 for controlling the whole server 40, and a storage unit 42 for storing, for example, notification conditions and various acquired information (and information generated based on the notification conditions). The control unit 41 may be configured to execute transfer of notification to the terminal device 50 as a part of the functions of the output unit 1e in FIG. 1. For example, the storage unit 42 may store, as notification conditions, a notification destination for each detected event and for each assisted person as a user. This allows the server 40, which has a part of the functions of the output unit 1e, to output to the notification destination in response to the occurrence of a detected event according to the notification conditions. Note that the number of notification destinations to be notified of a certain detected event may be one or more. In addition, the server 40 is preferably provided with a setting unit for setting notification conditions from an external device such as the terminal device 50, which allows the notification conditions to be set according to the operation mode.

[0064] The control unit 41 can be realized, for example, by a CPU, a working memory, and a non-volatile storage device that stores a program. The control unit 41 can also be realized, for example, by an integrated circuit. This storage device can also be used as the storage unit 42, and the program can be a program for causing the CPU to realize the functions of the server 40.

[0065] The terminal device 50 is a terminal device held by a caregiver who cares for (will care for) a care recipient as a toilet user, and can be a portable information processing device, but may also be a stationary device. In the former case, the terminal device 50 can be a smart device such as a mobile phone (including what is called a smartphone), a tablet, or a mobile PC, but can also be an electronic bulletin board, etc. Although not shown, the terminal device 50 can also include a control unit that controls the entire device and a storage unit, and this control unit can be realized by, for example, a CPU, a working memory, a storage unit, etc., like the control unit 41. In addition, the program stored in this storage unit can be a program for causing the CPU to realize the functions of the terminal device 50.

[0066] Next, a detailed example of the information processing device 60 will be described with reference to FIG. 4. FIG. 4 is a block diagram showing an example of the configuration of an information processing device in the information processing system of FIG. 2. The information processing device 60 can be composed of two devices, for example, as shown in FIG. 2 and FIG. 3. The information processing device 60 can also be called a "toilet sensor" or a "toilet sensor device" since it has various sensors installed in the toilet. The toilet sensor corresponds to a so-called edge in this system that monitors the use of the toilet by the care recipient via a communication network.

[0067] The control box 61 can house, for example, the CPU 27a, the connector 27b, the USB I / Fs 27c and 27d, and the WiFi module 27e. Note that USB is an abbreviation for Universal Serial Bus, and both USB and WiFi are registered trademarks (hereinafter the same). Note that the control box 61 may not be provided with various I / Fs and connectors, and may be directly connected to the CPU 27a.

[0068] 4, the control box 61 and the sensor box 63 are connected by an interface exemplified by the connector 27b and the USB I / F 27c, and the connection line is housed in the cable K1. A part of the cable K1 passes through the inside of the bridge 62 (and the outer housing 69). The control box 61 and the human detection / identification box 68 are connected by an interface exemplified by the connector 27b and the USB I / F 27d, and the connection line is housed in the cable K2.

[0069] In this way, the information processing device 60 is provided with boards and cables connecting them in each box and bridge 62. By adopting the divided box configuration as described above and optimizing the orientation of the connectors connecting the cables, the information processing device 60 can reduce the space required for the extra cable length and reduce the size of the board and mounted components. Also, in this embodiment, the placement locations of each box are separated, making it possible to reduce the width of each structure.

[0070] The sensor box 63 can house, for example, a distance sensor 26a that functions as a seating sensor that detects when the toilet seat 11 is sat on, and a first camera 26b that photographs excrement.

[0071] Next, the components within each box will be described. The CPU 27a is an example of a main control unit of the information processing device 60, and controls the entire information processing device 60. The connector 27b connects the human presence sensor 24 and the distance sensor 26a to the CPU 27a. The USB I / F 27c connects the first camera 26b to the CPU 27a, and the USB I / F 27d connects the second camera 25 to the CPU 27a.

[0072] Distance sensor 26a is a sensor that measures the distance to an object (the buttocks of the user of toilet bowl 10) and detects that the user has sat on toilet seat 11, and detects that the object has sat on toilet seat 11 when a certain period of time has elapsed after the threshold value is exceeded. Distance sensor 26a also detects that the user has left the toilet seat 11 when the distance to the object changes after sitting down.

[0073] The distance sensor 26a may be, for example, an infrared sensor, an ultrasonic sensor, an optical sensor, or the like. When the distance sensor 26a is an optical sensor, a transmitting / receiving element may be disposed so as to transmit and receive light (not limited to visible light) through a hole provided in the sensor box 63. The transmitting / receiving element may be configured as a transmitting element and a receiving element separately or may be integrated. The distance sensor 26a is connected to the CPU 27a via the connector 27b, and is capable of transmitting the detection result to the CPU 27a. Based on the detection result, the CPU 27a can obtain seating / death data indicating seating / death information of a person, and can also transmit the data to the server 40 via the WiFi module 27e.

[0074] First camera 26b is an example of a camera that captures imaging data that is the basis for acquiring excretion information, and may be an optical camera with a lens portion disposed in a hole provided in sensor box 63. First camera 26b is installed so that the imaging range includes the excretion range in toilet bowl 10. First camera 26b is connected to CPU 27a via USB I / F 27c, and transmits imaging data to CPU 27a.

[0075] The human presence sensor 24 is an example of the sensor 1a, and is a sensor that detects the presence of a person (entry or exit of a person) in a specific area (measurement area range of the human presence sensor 24) that is a part of the toilet room, and can be called an entrance and exit sensor. The human presence sensor 24 can adopt, for example, an infrared sensor, an ultrasonic sensor, an optical sensor, etc., regardless of the detection method. The human presence sensor 24 is connected to the CPU 27a via the connector 27b, and when it detects a person in the specific area, it transmits the detection result to the CPU 27a. This detection result can also be transmitted to the server 40 by the CPU 27a via the WiFi module 27e.

[0076] The second camera 25 is an example of the imaging device 1b, and may be an optical camera, and is an example of a face recognition camera that captures a face image of a user or the like to identify the user of the toilet and obtains face image data. The second camera 25 may be installed in the toilet room (inside the toilet) in which the toilet 10 is installed. The CPU 27a performs face recognition processing using the face image data captured by the second camera 25. In this system, when the human presence sensor 24 detects a person, the second camera 25 can be processed to capture an image of the subject who has entered the toilet, and thus face recognition can be performed only when a person is detected.

[0077] The WiFi module 27e is an example of a communication device that transmits various acquired data to the server 40, and can be replaced with a module that employs other communication standards. Data to be transmitted can be transmitted to the server 40 by the CPU 27a via the WiFi module 27e. Note that face image data acquired by the second camera 25 is not basically transmitted in order to reduce the amount of data and to consider privacy.

[0078] CPU 27a provisionally determines whether the person who has entered the room is a care recipient as a user of the room or not by determining whether the person is wearing a mask based on image data obtained by second camera 25. A provisional determination that the person is not a care recipient means that the person is provisionally determined to be a caregiver.

[0079] Furthermore, when the result of the provisional determination is that the person is a care recipient, that is, that the person is not wearing a mask, the CPU 27a executes face recognition processing based on the face image data of the care recipient in the image data. When the image data includes face image areas of multiple people, the face recognition processing may be executed based on the data of the face image area of ​​the care recipient among all the people. Here, the face recognition processing is executed by comparing the face image data with feature data indicating face features stored in advance for each care recipient and each caregiver who may enter the room accompanying the care recipient. Therefore, in the face recognition processing, when the result of the provisional determination is that the person is a caregiver, that is, that the person is wearing a mask, the face recognition processing is not executed, and the subsequent processing is terminated.

[0080] The feature data may be data indicating facial feature points, for example. The feature data may be data generated from facial image data acquired by a terminal device such as the terminal device 50 and stored in an internal storage device of the information processing device 60. Although not shown, this internal storage device may be connected to the CPU 27a, for example, or may be provided inside the CPU 27a. This comparison may also be performed by inputting facial image data of the care recipient who has entered the room to a learning model (trained model) that has been machine-trained using learning data including the feature data and the image data (facial image data) that is the source of the feature data.

[0081] Then, by such face recognition processing, it is possible to specify the person who entered the target room from the face image data of the person who became the care recipient by the provisional determination, that is, to specify (identify) the individual who entered the target room. Since the person specified here is the result of the above-mentioned comparison, it may be that the person becomes the caregiver by the provisional determination, in which case the face recognition result may not be output. In addition, it is possible that the user of the room may be a person who does not correspond to either the care recipient or the caregiver (for example, a family member who came to visit), and such a person may be determined as an unknown person as a result of the face recognition processing, and may be notified of the fact. In addition, it is preferable that the face image data captured by the second camera 25 is not stored after the face recognition processing, in consideration of privacy.

[0082] The CPU 27a and the WiFi module 27e can be an example of the output unit 1e in FIG. 1. The CPU 27a outputs the result of the face recognition process to the notification destination via the WiFi module 27e. If the CPU 27a recognizes a caregiver in the face recognition process, the CPU 27a can also output the result of the face recognition process to the notification destination, excluding the result of the face recognition process for the caregiver. If the result of the face recognition process is an unknown person, the CPU 27a may or may not notify that an unknown person has entered the room, and it is only necessary to determine in advance which process to execute.

[0083] In this way, the CPU 27a can be responsible for the main processing of detecting the detection event indicated by the result of the face recognition processing, the excretion information, and the seating / leaving information. The notification destination can be the terminal device 50 held by one or more of the caregivers, and the caregiver to be the notification destination can be included in the notification conditions. The caregiver can be notified by finally performing display and / or audio output on the terminal device 50. In addition, the terminal device 50 is described on the assumption that it is a smart device, but it may be a device that only emits a sound when a notification is received, or an electronic bulletin board that displays the entry of a room, a personal name, etc. In this way, the notification destination may be, for example, a notification device of a nurse call system, another terminal device (for example, a PHS: Personal Handy-phone System) held by the caregiver other than the terminal device 50, an intercom, etc. In addition, details of the notification of the excretion information and the seating / leaving information are omitted, but both can be treated as one of the detection events.

[0084] Further, the CPU 27a, the USB I / F 27c, the WiFi module 27e, and the server 40 obtain the above-mentioned excretion information based on the imaging data captured by the first camera 26b. In this case, the main process of obtaining the excretion information from the imaging data can be performed by the server 40. However, the CPU 27a can also be configured to perform this process. In order to obtain the excretion information indicating the content of the excretion based on the imaging data, the server 40 or the CPU 27a includes an analysis unit that analyzes the excretion information based on the imaging data obtained by the first camera 26b. This analysis unit executes the analysis of the excretion information when the care recipient can be authenticated as a result of the face recognition process, that is, when the entrant can be identified and is the care recipient. The first camera 26b can also perform imaging only when such a result of the face recognition process is obtained.

[0085] The server 40 in the example of FIG. 2 inputs the imaging data to the trained model, for example, to obtain excretion information. Preferably, the server 40 obtains information indicating whether the imaging data includes a foreign object other than feces and urine as a subject other than the toilet bowl and its cleaning liquid, as part of the excretion information. The foreign object may be called a "other object" and may be liquid or solid other than feces and urine, and may include, for example, one or more of vomit, bloody stool, vomiting of blood (vomited blood), diapers, urine pads, and toilet paper tubes. In addition, the server 40 may be configured to obtain the shape, color, and amount of the excretion as the excretion information when the excretion is not a foreign object but an excretion. The CPU 27a and the WiFi module 27e, or the server 40, may also notify the notification destination of the analysis result in the analysis unit. In this case, the notification destination may be different from the notification destination of the result of the face recognition process, and the notification destination may be stored, for example, as a notification condition, in the storage device or the storage unit 42 in the information processing device 60.

[0086] Also, the server 40 may store information such as excretion information and information on leaving and sitting, in the storage unit 42, and may output the information in response to access from the terminal device 50. In particular, it is preferable that the server 40 has a function for generating an excretion diary. In this case, if the generated excretion diary is stored in the storage unit 42, the caregiver who is the user of the terminal device 50 can view the excretion diary from the terminal device 50 whenever desired. The excretion diary includes the date and time of the excretion act, and this date and time can be obtained from any of the following: the image capture date and time of the image data of the excretion, the identification date and time of the subject, the entry and exit date and time, the sitting date and time, the leaving date and time, the intermediate date and time between sitting and leaving, and the like.

[0087] Next, a processing example of generating and registering facial feature data will be described with reference to Fig. 5. Fig. 5 is a schematic diagram for explaining an example of the flow of processing of generating and registering facial model data in the information processing system of Fig. 2.

[0088] As shown in FIG. 5, the registration of facial feature data (hereinafter, face model data) is performed by generating face model data and transmitting the face model data to the information processing device 60. First, the face photo used to generate the face model data can be imaging data obtained by a caregiver or the care recipient taking a picture of himself or herself with a terminal such as the terminal device 50. This terminal transmits the obtained imaging data to a server 40 on the cloud, and the server 40 generates face model data for face authentication from the imaging data. Regardless of the type of face model data, it is sufficient that the authentication accuracy in the face authentication process is high, but it is preferable to use face model data that has a light processing load in the face authentication process. The face model data can be data including information extracted from a face photo, such as the eyes, mouth, nose, and contour. In addition, the care recipient can take a face photo of himself or herself with a terminal equivalent to the terminal device 50, transmit the imaging data to the server 40, and the server 40 can generate the face model data.

[0089] Thereafter, the server 40 transmits the generated face model data to the information processing device 60, and the information processing device 60 registers, i.e. installs, the face model data so that the face model data can be referenced during face recognition processing, and holds the face model data. When transmitting data from the server 40 to the information processing device 60, it is possible to transmit only the modeled data in this manner, without transmitting image data itself such as a face photograph.

[0090] The generation of the facial model data can also be executed by a server other than the server 40. In addition, when the facial model data can be generated by a terminal such as the terminal device 50 that has taken the facial photograph, the data generated by the terminal can also be sent directly to the information processing device 60 without using a server such as the server 40.

[0091] Next, functions corresponding to the determination unit 1c and face authentication unit 1d, that is, face detection, face identification (face authentication), and mask detection, will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a schematic diagram for explaining an example of face authentication processing in the information processing device 60 when the caregiver is not wearing a mask. Fig. 7 is a schematic diagram for explaining an example of face authentication processing in the information processing device 60 when the caregiver is wearing a mask.

[0092] The processes of face detection, face identification (face authentication), and mask detection can be performed using a neural network-based algorithm by machine learning. FIG. 6 shows an overview of the face authentication function when the caregiver is not wearing a mask. As shown in FIG. 6, first, when the CPU 27a receives a detection result from the human sensor 24 and detects entry into the toilet, it starts the second camera 25 for face authentication, scans the imaging range of the second camera 25, and when a face is detected, surrounds the detected face with a rectangle. As a result, as shown in the image FD in FIG. 6, the facial areas of the care recipient and the caregiver are surrounded by rectangles fa1 and fa2, respectively, in the imaging data.

[0093] Next, the CPU 27a performs face recognition processing by collating the detected face areas fa1 and fa2 with face model data as shown in the image FR1 of FIG. 6. If the face is determined to be a caregiver as a result of the face recognition processing (in this example, the face area fa2 is determined to be the face of the caregiver), the CPU 27a excludes the face area fa2 from the target of face recognition as shown in the image FR2 of FIG. 6, even if the caregiver is within the imaging range of the second camera 25. The determination that the person is a caregiver can be made, for example, by first detecting a certain feature point in the face model data of the caregiver. This feature point can be registered in advance as a feature point such as a hat, a hairpin, or a hairstyle. Finally, the CPU 27a can identify the face as the care recipient A by performing face recognition processing on the face area fa1 in the image FR2.

[0094] As shown in Fig. 7, when the CPU 27a detects a toilet entry based on the detection result from the motion sensor 24, it starts the second camera 25 for face recognition, scans the imaging range of the second camera 25, and when a face is detected, surrounds the detected face with a rectangle. As a result, as shown in the image FD in Fig. 7, the facial areas of the care recipient and the caregiver are surrounded by the rectangle fa1 and the rectangle fa2, respectively, in the imaging data.

[0095] When there is a mask in the rectangular face areas fa1, fa2, the CPU 27a determines that the mask is being worn by surrounding the mask with a rectangle (rectangular area ma2 in this case) as shown in image MD in Fig. 7. When the CPU 27a determines that the person is wearing a mask, that is, a caregiver, as a result of the mask wearing determination, even if the caregiver is within the imaging range of the second camera 25, the CPU 27a excludes the face area fa2 from the target of face recognition as shown in image FR in Fig. 7, and does not perform face matching. The CPU 27a executes face recognition processing for matching the face area fa1 not wearing a mask in the image FR in Fig. 7 with face model data, and can ultimately identify this face as the care recipient A.

[0096] 6 and 7, a rectangular area is actually enclosed, but it is sufficient if a rectangular area is detected and the area can be specified. Also, the shape of the face can be other than rectangular, such as an ellipse. Also, the mask can be other than rectangular, such as a polygon with five or more sides, or an ellipse.

[0097] Next, a display example of notification information on the terminal device 50 will be described with reference to Fig. 8. Fig. 8 is a diagram for explaining a notification example on the terminal device in the information processing system of Fig. 2.

[0098] 8, the terminal device 50 can notify notification information 52 to 54 in that order on the display screen 51. Notification information 52 is an example of notification information notifying that a care recipient has entered the toilet when a person is detected by the human presence sensor 24 and the person is provisionally determined to be a care recipient. As in this example, a notification may also be sent to the notification destination when the entry of a care recipient, that is, the entry of a person not wearing a mask, is detected as a result of the provisional determination.

[0099] After notification of notification information 52, notification information 53 is notified as a result of face recognition processing. This is an example of notification information notifying that the person who entered the room is the user (in this example, the care recipient PA) as a result of identifying the user by face recognition processing. Note that it is possible to know that the person is PA if a name is associated with the face model data.

[0100] Furthermore, when it is detected that PA has left the room, it is possible to configure so that notification information 54 indicating this is displayed. Furthermore, even if, after notification information 52 is notified, face recognition processing detects that the person entering the room is a caregiver, the name of the individual caregiver (for example, C) can be notified, but it is also possible for no notification to be made. As an alternative process, in this case, a notification can be made that excludes the results of face recognition processing, that is, a notification can be made simply that the person entering the room was a caregiver.

[0101] Next, a flow of face recognition processing and notification in this system triggered by the entry of a person requiring care into a room will be described with reference to Fig. 9. Fig. 9 is a flow diagram for explaining an example of face recognition processing in the information processing device 60. Note that the processing in the information processing device 60 is mainly executed by the CPU 27a.

[0102] First, the information processing device 60 monitors whether or not there is a reaction from the human sensor 24 functioning as an entrance sensor (step S1), and when there is a reaction (step S1 is YES), it starts up the second camera 25, which is an optical camera (step S2). If the result is NO in step S1, that is, when there is no reaction from the human sensor 24, it waits until there is a reaction.

[0103] In this way, when a person requiring care enters the toilet, step S1 becomes YES and second camera 25 is activated. The activated second camera captures an image of the person who has entered, and CPU 27a or second camera 25 performs a process to detect a face within the imaging range and determines whether or not a face has been detected (step S3). In step S3, as illustrated in Figs. 6 and 7, when second camera 25 detects a face, the detected face is surrounded by a rectangle. The determination in step S3 is performed while changing the imaging range of second camera 25 as necessary and capturing images until a face is detected. Step S3 is executed until a face is detected, but the process can also be terminated after a predetermined time has elapsed, for example.

[0104] The CPU 27a performs mask detection by determining whether or not a mask is worn for the face area detected in step S3 (step S4). If step S4 is YES, that is, if it is detected that the person is wearing a mask, the CPU 27a determines that the person is a caregiver, ends the face recognition process, and does not perform any further recognition process on the face image of the person. On the other hand, if step S4 is NO, that is, if the person is not wearing a mask, the CPU 27a compares the face area (image data of the face area) with face model data (step S5). When comparing, it is also possible to generate face model data for the image data of the face area in the same way as when generating the face model data, and compare the two.

[0105] As a result of the comparison in step S5, the CPU 27a determines whether the image data of the face area matches or corresponds to registered face model data (also simply referred to as face data) (step S6). If the answer is YES in step S6, the CPU 27a determines whether the user is a care recipient or a caregiver (step S7). If the CPU 27a determines in step S7 that the user is a caregiver, the process ends. On the other hand, if the CPU 27a determines in step S7 that the user is a user, the CPU 27a cooperates with the notification function and excrement analysis function to notify the user of an entry and analyze the excrement to notify the excrement information (step S8), and then ends the process. The cooperation in step S8 may include cooperation with the server 40.

[0106] Also, if step S6 is NO, that is, if the image data of the face area is unregistered or registered but cannot be matched, the CPU 27a marks the entrant as unknown (step S9), cooperates with the notification function and excrement analysis function (step S8), and ends the process. Note that after the process of step S9, the process can also be ended without such cooperation. In this way, in the information processing device 60, when the identified entrant is a user, it can cooperate with processes such as analysis and notification of excrement indicating the user's toilet usage status (analysis of the user's status and notification to the caregiver).

[0107] The above description of this system is based on the assumption that there is only one toilet 10 (only one toilet room). However, in this system, it is preferable to obtain face recognition results for detecting detection events for multiple toilets. This makes it possible to apply this system to cases where a certain person being assisted may use two or more toilets.

[0108] In addition, the server 40 can be installed in a facility such as a hospital, or in a private home or apartment building for personal use. In either case, the server 40 can be a cloud server as described above.

[0109] The program of the terminal device 50 can be executable and embedded in the terminal device 50 as nursing software including a notification function of notification information. In addition to the notification information, the terminal device 50 can also directly (or directly and automatically) receive and store information obtained on the toilet side such as the information processing device 60, and can also receive and store various information recorded in the server 40.

[0110] Of course, it is also possible to configure the information processing device 60 by itself, without using the server 40, by providing the functions described as the functions of the server 40 on the information processing device 60 side. In other words, it is also possible to adopt a configuration in which the information detected by the information processing device 60 is directly notified to the terminal device 50 held by the caregiver. In this case, the notification can be made directly using, for example, Bluetooth (registered trademark) or ZigBee (registered trademark), or can be made using a communication network using, for example, Wi-Fi or LTE.

[0111] As described above, in this system, in a care facility or a house, face recognition processing is performed based on information acquired by the information processing device 60, which corresponds to a so-called edge of a communication network, and a person requiring care can be monitored. For example, the information processing device 60 is installed in a toilet where entry detection is desired. As a result, the information processing device 60 can detect entry using the human presence sensor 24, perform mask determination processing and face recognition processing, and notify the terminal device 50 of information indicating the entry and the care recipient identified as a result of face recognition if the person is a care recipient.

[0112] Here, in this system, the face model data used for face recognition is stored in a server on the cloud or the like, and the information processing device 60 acquires and holds the face model data in advance by a communication function. The information processing device 60 detects people with the human presence sensor 24, which serves as an entrance / exit sensor, and detects entry and exit into the restroom. After the human presence sensor 24 detects a person, the information processing device 60 performs face recognition processing to identify the person who has entered the restroom by photographing the person who has entered the restroom with the second camera 25 for face recognition and comparing the photograph with the face model data held in the device and extracting the person who most closely resembles the person. The photographed photograph data is not stored in consideration of privacy. The information processing device 60 can also be configured to acquire information such as the location of the entered restroom, the user's name, the time of entry and exit, and whether the user has sat on or left the toilet seat, using the distance sensor 26a, the first camera 26b, the human presence sensor 24, and the second camera 25 for face recognition. In this case, the information processing device 60 can transmit the acquired information to the server 40 via the WiFi module 27e, and the information can be notified to the terminal device 50 held by the caregiver via the server 40. At this time, image data captured by the first camera 26b for excrement and the second camera 25 for face recognition can be not transmitted to the server 40.

[0113] In this system, with the above-mentioned configuration, the two problems specifically described as (1) and (2) can be solved as follows.

[0114] First, the problem (1) above can be solved as follows. In this embodiment, face model data of the user and the caregiver are pre-installed in the information processing device 60. Then, the information processing device 60 compares the face model data with the face captured by the face recognition camera, and if the person who entered the room is identified as a caregiver, that person is excluded from face recognition from then on if he or she is within the imaging range of the face recognition camera. This allows only the user to be the subject of face recognition processing, making it possible to improve the accuracy of person identification without reducing the processing power of face recognition.

[0115] However, even if face model data of caregivers is incorporated, if one of the caregivers wears a mask, the recognition accuracy of face recognition deteriorates as described as the problem (2) above. Therefore, the problem (2) above is solved as follows. In this embodiment, the fact that the caregiver may wear a mask, whereas the user does not wear a mask inside the facility, is utilized, and erroneous recognition of face recognition is reduced by adding a mask detection process to the face recognition process. Specifically, when the information processing device 60 detects that a person has entered the restroom, it performs a face detection process of the person and a mask detection process on the detected face. By performing the mask detection process, it becomes possible to exclude the caregiver who wears a mask from the target of the face recognition process, and face recognition can be performed only on the face not wearing a mask.

[0116] Even if machine learning is used, the determination of whether a mask is worn is faster than face matching, so it is possible to improve real-time performance even in edge devices with space-saving and power-saving CPUs. In particular, identification of users by face recognition in a toilet requires coordination with a notification function to a caregiver, such as toilet assistance for the user or rushing to prevent accidents, so not only identification accuracy but also real-time performance is required, but this system enables real-time processing. In fact, the recognition process of a face wearing a mask is prone to misrecognition because it matches with little facial feature information other than the mask. In contrast, in this system, if the mask is detected by the mask detection process, it can be excluded from the face recognition process, so it is possible to prevent a decrease in the performance of the face recognition process and misrecognition of the user even if the user enters the room accompanied by a caregiver.

[0117] In this embodiment, by incorporating the face model data of the caregiver and mask detection processing as described above, even if the user and the caregiver are within the imaging range of the face recognition camera, face recognition is performed only on the user, so that the accuracy of person identification can be improved without reducing the processing capacity of face recognition. In other words, even if the CPU 27a is a space-saving and power-saving CPU and the information processing device 60 is weak, face recognition of the user is possible without reducing the processing performance and authentication accuracy. As a result, in this embodiment, it is possible to reduce the burden on the caregiver and provide generous support to the user. In addition, in this embodiment, since the caregiver can be excluded from the face recognition, it is also possible to exclude cases such as toilet cleaning by the caregiver. In addition, for excretion management, which is a heavy burden on the caregiver, the burden of the care work can be reduced by linking the toilet entry and exit management using face recognition processing with the excretion analysis function.

[0118] As described above, this system enables real-time user identification with high accuracy and reduced performance degradation, reducing the burden on caregivers and providing thorough support to users. That is, this system achieves the effects described in the first embodiment, and in particular the following effects.

[0119] The first effect is that by registering the caregiver in advance, it is possible to reduce the chance of mistakenly detecting the caregiver as the user, and by incorporating a mask detection process, it is possible to obtain an even greater improvement in accuracy if the caregiver is wearing a mask.

[0120] The second effect is that since mask detection processing is a lighter process than facial recognition processing, even when multiple caregivers are assisting one user, the user can be identified through facial recognition without significantly compromising performance.

[0121] A third effect is that since the mask detection process excludes caregivers wearing masks from the targets of face recognition, the caregivers can be excluded from targets of using the toilet when cleaning the toilet, for example.

[0122] <Embodiment 3> In the second embodiment, a toilet is described as an example of a room, but other types of rooms such as a bathroom can also be used. In this embodiment, an example of use in a bathroom will be described with reference to Fig. 10, but the various examples described in the first and second embodiments can be applied. Fig. 10 is a schematic diagram showing an example of the configuration of an information processing device according to the third embodiment.

[0123] As shown in Fig. 10, the information processing device according to this embodiment does not require the components related to excretion information in the information processing device 60 in embodiment 2. The information processing device according to this embodiment includes a control box 61 and a human detection / identification box 68 connected to the control box 61 via a cable K2. Note that the server 40 and the terminal device 50 are omitted in Fig. 10.

[0124] The human detection / identification box 68 can be installed inside the bathroom, or placed outside the bathroom in a position where it can detect a person entering through the door 71. Figure 10 shows an example in which the human detection / identification box 68 is installed on the far side of the bathtub 70 in the bathroom, facing away from the door 71. The control box 61 can also be installed inside the bathroom, but it can also be installed outside the bathroom.

[0125] In this embodiment, the information processing device, which corresponds to the so-called edge of the communication network, can also monitor the person requiring care in a care facility or a home. In this embodiment, when the information processing device is installed in a bathroom where entry detection and face recognition are desired, it is possible to detect entry by the human sensor 24 and execute face recognition processing based on image data acquired by the second camera 25.

[0126] Although the bathroom has been described as an example, the human detection / identification box 68 can be installed in rooms other than the toilet and bathroom, such as the entrance to the facility, bedrooms, and dining rooms, to monitor the care recipients. In addition to nursing facilities, the box can also be used in hospitals, food companies, kindergartens, schools, and other facilities where many people wear masks and there is a concern about leaking information to networks. For example, by installing a face recognition camera in a dining room, a visitor to the dining room can be identified by face recognition processing, and meals can be prepared according to the user. Alternatively, by installing a face recognition camera at the entrance to the facility, it is possible to prevent the authentication subject from going out alone by notifying other people or issuing an alarm on the terminal of the authentication subject. In any of the installation examples, the identification of the facility user can be linked to the record of other tasks such as nursing care.

[0127] <Other embodiments> [a] In each embodiment, the functions of the information processing system and each device included in the system have been described, but each device is not limited to the configuration example shown in the figure, and it is sufficient for each device to be able to realize these functions, and additional functions may also be provided.

[0128] [b] Each of the devices described in the first to third embodiments may have the following hardware configuration. Fig. 11 is a diagram showing an example of the hardware configuration of the device. The same applies to the other embodiment [a] described above.

[0129] The device 100 shown in FIG. 11 may have a processor 101, a memory 102, and a communication interface (I / F) 103. The processor 101 may be, for example, a microprocessor, a microprocessor unit (MPU), or a CPU. The processor 101 may include a plurality of processors. The memory 102 is, for example, configured by a combination of a volatile memory and a non-volatile memory. The functions of each device described in the first to third embodiments are realized by the processor 101 reading and executing a program stored in the memory 102. At this time, information can be exchanged with other devices via the communication interface 103 or an input / output interface (not shown).

[0130] The program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0131] The present disclosure is not limited to the above-described embodiment, and may be modified as appropriate without departing from the spirit and scope of the present disclosure. In addition, the present disclosure may be implemented by appropriately combining the respective embodiments.

[0132] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes.

[0133] (Appendix 1) A sensor that detects a person entering the room; an imaging device that captures an image of a person entering the room and obtains image data when the sensor detects the person entering the room; a determination unit that determines whether or not a person who has entered the room is wearing a mask based on the image data obtained by the imaging device, and thereby provisionally determines whether or not the person who has entered the room is a person to be authenticated as a user of the room; a face authentication unit that executes face authentication processing based on face image data of the person to be authenticated in the image data when the result of the provisional determination by the determination unit indicates that the person is the person to be authenticated; An output unit that outputs a result of the face recognition processing to a notification destination; Equipped with The face authentication unit executes the face authentication process by comparing the face image data with feature data indicating facial features stored in advance for each of the authentication subjects and for each of the attendants who may accompany the authentication subjects and enter the room. Information processing device. (Appendix 2) When the attendant is authenticated as a result of the face authentication processing, the output unit outputs the result of the face authentication processing excluding the result of the attendant. 2. An information processing device according to claim 1. (Appendix 3) The feature data is generated from face image data acquired by a terminal device and stored in the information processing device. 3. The information processing device according to claim 1 or 2. (Appendix 4) the room is a toilet, and the imaging device is installed inside the toilet; The information processing device includes an analysis unit that analyzes excretion information indicating the content of excretion based on imaging data captured by another imaging device that is installed so as to include an excretion range in the toilet bowl of the toilet within an imaging range, the analysis unit performs an analysis of the excretion information when the person to be authenticated is authenticated as a result of the face authentication process; The output unit outputs the analysis result of the analysis unit to a notification destination. 4. The information processing device according to claim 1, (Appendix 5) The sensor detects when someone enters the room, When the sensor detects the entry of a person into the room, an imaging device captures an image of the person who entered the room to obtain image data; determining whether the person who has entered the room is wearing a mask based on the image data obtained by the imaging device, and provisionally determining whether the person who has entered the room is a person to be authenticated as a user of the room or is the person to be authenticated; If the result of the provisional determination is that the person is the person to be authenticated, a face authentication process is executed by comparing the face image data of the person to be authenticated in the image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may accompany the person to be authenticated and enter a room, based on the face image data of the person to be authenticated in the image data; outputting a result of the face recognition processing to a notification destination; Information processing methods. (Appendix 6) When the attendant is authenticated as a result of the face authentication process, the output to the notification destination is executed by excluding the result of the face authentication process for the attendant. 6. An information processing method as described in Appendix 5. (Appendix 7) The feature data is data generated from face image data acquired by a terminal device and stored. 7. An information processing method according to claim 5 or 6. (Appendix 8) the room is a toilet, and the imaging device is installed inside the toilet; If the authentication subject is authenticated as a result of the face authentication process, an analysis process is executed to analyze excretion information indicating the content of the excretion based on image data captured by another image capture device installed so as to include an excretion range in the toilet bowl of the toilet in an image capture range, outputting the analysis result of the analysis process to a notification destination; An information processing method according to any one of appendix 5 to 7. (Appendix 9) On the computer, The sensor detects when someone enters the room, and inputs the result. When the sensor detects the entry of a person into the room, an image capturing device captures an image of the person who entered the room to obtain image data; determining whether the person who has entered the room is wearing a mask based on the image data obtained by the imaging device, and provisionally determining whether the person who has entered the room is a person to be authenticated as a user of the room or is the person to be authenticated; If the result of the provisional determination is that the person is the person to be authenticated, a face authentication process is executed by comparing the face image data of the person to be authenticated in the image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may accompany the person to be authenticated and enter a room, based on the face image data of the person to be authenticated in the image data; outputting a result of the face recognition processing to a notification destination; A program for executing information processing. (Appendix 10) When the attendant is authenticated as a result of the face authentication process, the output to the notification destination is executed by excluding the result of the face authentication process for the attendant. 9. The program described in Appendix 9. (Appendix 11) The feature data is generated from face image data acquired by a terminal device and stored in the computer. 11. The program according to claim 9 or 10. (Appendix 12) the room is a toilet, and the imaging device is installed inside the toilet; The information processing includes: If the authentication subject is authenticated as a result of the face authentication process, an analysis process is executed to analyze excretion information indicating the content of the excretion based on imaging data captured by another imaging device installed so as to include in its imaging range the excretion range in the toilet bowl of the toilet; outputting the analysis result of the analysis process to a notification destination; The program according to any one of appendix 9 to 11. [Explanation of symbols]

[0134] 1, 60 Information processing device 1a Sensor 1b Imaging device 1c Judgment part 1d Face recognition unit 1e Output section 10 Toilet 10a Side 10b Plane part 10c edge 11 Toilet seat 11a Toilet seat body 11b Toilet seat cover 24 Human Sensor 25 Imaging device (second camera) 26 Information Gathering Department 26a Distance sensor 26b 1st Camera 27a CPU 27b Connector 27c, 27d USB I / F 27e WiFi module 40 Servers 41 Control section 42 Storage section 50 Terminal Equipment 51 Display screen 52, 53, 54 Notification information 61 Storage box (control box) 62 Bridge section 63 Inner housing (sensor box) 67 Control Unit 68 Separate housing (human detection / identification box) 69 Outer housing 70 Bathtub 71 Door 100 devices 101 Processor 102 Memory 103 Communication Interface

Claims

1. A sensor that detects a person entering the room; an imaging device that captures images of the plurality of people who have entered the room and obtains image data when the sensor detects the plurality of people entering the room; a determination unit that determines whether each of a plurality of people who have entered the room is wearing a mask based on the image data obtained by the imaging device, and thereby provisionally determines whether the person who has entered the room is a person to be authenticated as a user of the room; a face authentication unit that executes face authentication processing based on face image data of the person to be authenticated in the image data only when the person to be authenticated is determined to be the person to be authenticated as a result of the provisional determination by the determination unit; An output unit that outputs a result of the face recognition processing to a notification destination; Equipped with the face authentication unit does not execute the face authentication process for the person wearing the mask and ends the process, and executes the face authentication process only for the person not wearing a mask by comparing the face image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may accompany the person to be authenticated and enter the room; Information processing device.

2. When the attendant is authenticated as a result of the face authentication processing, the output unit outputs the result of the face authentication processing excluding the result of the attendant. The information processing device according to claim 1 .

3. The feature data is generated from face image data acquired by a terminal device and stored in the information processing device.

3. The information processing device according to claim 1 or 2.

4. the room is a toilet, and the imaging device is installed inside the toilet; The information processing device includes an analysis unit that analyzes excretion information indicating the content of excretion based on imaging data captured by another imaging device that is installed so as to include an excretion range in the toilet bowl of the toilet within an imaging range, the analysis unit performs an analysis of the excretion information when the person to be authenticated is authenticated as a result of the face authentication process; The output unit outputs the analysis result of the analysis unit to a notification destination.

4. The information processing device according to claim 1.

5. The sensor detects when someone enters the room, When the sensor detects the entry of a plurality of people into the room, an imaging device captures images of the plurality of people who have entered the room to obtain image data; determining whether each of the people who have entered the room is wearing a mask based on the image data obtained by the imaging device, and provisionally determining whether the people who have entered the room are subjects to be authenticated as users of the room; As a result of the provisional determination, the facial recognition process is terminated without being performed on the person wearing a mask, and only when the person not wearing a mask is the person to be authenticated, the facial recognition process is performed based on the facial image data of the person to be authenticated in the image data by comparing the facial image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may accompany the person to be authenticated and enter the room, outputting a result of the face recognition processing to a notification destination; Information processing methods.

6. When the attendant is authenticated as a result of the face authentication process, the output to the notification destination is executed by excluding the result of the face authentication process for the attendant. The information processing method according to claim 5.

7. the room is a toilet, and the imaging device is installed inside the toilet; If the authentication subject is authenticated as a result of the face authentication process, an analysis process is executed to analyze excretion information indicating the content of the excretion based on image data captured by another image capture device installed so as to include an excretion range in the toilet bowl of the toilet in an image capture range, outputting the analysis result of the analysis process to a notification destination; 7. The information processing method according to claim 5 or 6.

8. On the computer, The sensor detects when someone enters the room, and inputs the result. When the sensor detects the entry of a plurality of people into the room, an imaging device captures images of the plurality of people who have entered the room to obtain image data; determining whether each of the people who have entered the room is wearing a mask based on the image data obtained by the imaging device, and provisionally determining whether the people who have entered the room are subjects to be authenticated as users of the room; As a result of the provisional determination, the facial recognition process is terminated without being performed on the person wearing a mask, and only when the person not wearing a mask is the person to be authenticated, the facial recognition process is performed based on the facial image data of the person to be authenticated in the image data by comparing the facial image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may accompany the person to be authenticated and enter the room, outputting a result of the face recognition processing to a notification destination; A program for executing information processing.

9. When the attendant is authenticated as a result of the face authentication process, the output to the notification destination is executed by excluding the result of the face authentication process for the attendant. The program according to claim 8.

10. the room is a toilet, and the imaging device is installed inside the toilet; The information processing includes: If the authentication subject is authenticated as a result of the face authentication process, an analysis process is executed to analyze excretion information indicating the content of the excretion based on image data captured by another image capture device installed so as to include an excretion range in the toilet bowl of the toilet in an image capture range, outputting the analysis result of the analysis process to a notification destination; 10. The program according to claim 8 or 9.

Images