Information processing device, information processing method, and program

Abstract

To provide an information processing device which outputs an authentication result by performing face authentication of a user with high accuracy in real time without deteriorating a processing capacity when a mask wearer exists in persons accompanying the user of the room.SOLUTION: An information processing apparatus 1 comprises: a sensor 1a which detects entry to a room; an imaging device 1b which obtains image data obtained by imaging an entering person when the entry is detected; a determination unit 1c; a face authentication unit 1d; and an output unit 1e which outputs a result of face authentication processing to a notification destination. The determination unit 1c temporarily determines whether or not the entering person is an authentication object person as a user of the room by determining the presence / absence of mask wearing of the entering person on the basis of image data obtained by the imaging device 1b. The face authentication unit 1d executes face authentication processing by comparing face image data for the authentication object person with feature data indicating the feature of the face that has been stored in advance for each authentication object person and each accompanying person that may enter the room accompanying the authentication object person when the entering person is the authentication object person based on the result of temporary determination.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Caregivers monitor the care recipients (those requiring care) in care facilities, homes, and other care settings to understand their condition and provide assistance. This monitoring work places a heavy burden on caregivers, and support is needed to reduce this burden. One way to achieve this support is to automate the monitoring process as much as possible, but this automation requires at least identifying the care recipient.

[0003] Regarding toilets, Patent Document 1 describes a technology that collects excretion information indicating the contents of excrement excreted into the toilet bowl without the need to interview the toilet user, and prevents foreign objects other than excrement from being flushed even if they are found in the toilet bowl. In the technology described in Patent Document 1, a server receives facial image data captured by a camera, compares its features with pre-stored authentication data, and performs facial recognition processing to obtain identification data associated with the matching authentication data, thereby identifying the toilet user. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2021 / 024584 Summary of the Invention [Problem to be solved by the invention]

[0005] As described above, caregivers monitor and assist the condition of those in need of care. The burden on caregivers can be reduced by implementing a system in which a server performs facial recognition processing based on image data to identify toilet users, as in the technology described in Patent Document 1. However, in this case, facial image data acquired by the camera is stored on the server. Because information about toilet use is sensitive information, leaving such facial image data on the server is not appropriate, as it places a significant mental burden on users and caregivers. Therefore, it is desirable to use a small device (toilet sensor) installed in the toilet that has facial recognition, notification, and excrement analysis functions, and prevents facial image data from leaking onto a network, such as an external server.

[0006] If such a configuration is adopted, a person entering the restroom will be identified by a device with a facial recognition function installed inside the restroom, and only that information will be notified to the caregiver that the user has entered. In this case, the facial image data used for facial recognition is not stored externally, which can reduce the mental burden on the user and caregiver, but facial recognition using a toilet sensor poses the following major problems (1) and (2).

[0007] (1) Processing power issues arising from the need to perform multiple facial recognitions simultaneously. (2) The problem of increased misidentification rates due to caregivers wearing masks.

[0008] Regarding (1) above, toilet space issues arise when a user is unable to enter or exit the toilet without the assistance of a caregiver. When using the toilet, if one or more caregivers accompany the user and enter the imaging range of the facial recognition camera, multiple facial recognitions must be performed simultaneously. Therefore, in such cases, the accuracy of identifying the user decreases and the processing time increases.

[0009] Regarding (2) above, in addition to the problem of (1) above, in nursing care facilities, users do not wear masks, and caregivers, although they usually do not wear masks, may wear masks during infectious disease epidemics to prevent infection while working. In this case, when assisting with excretion, the user and the caregiver, who is wearing a mask, will be within the imaging range of the facial recognition camera. Recognizing a face wearing a mask cannot use the facial information covered by the mask, and matching is performed using limited feature information, which is prone to misrecognition, and this can lead to reduced recognition accuracy. As a result, there is a risk of the face being mistakenly recognized as that of the user, even though it is actually that of a caregiver.

[0010] Furthermore, because facial recognition-based personal identification must be linked to a notification function for caregivers, such as for toilet assistance or emergency response to prevent accidents, not only accuracy but also real-time performance is required, making it desirable to improve the processing power and accuracy of facial recognition processing. On the other hand, if such processing is to be performed without transmitting facial image data over a network, it is desirable for the toilet sensor to be implemented as an edge device incorporating a space-saving, power-saving CPU (Central Processing Unit). Therefore, it is desirable to improve the processing power and accuracy of facial recognition processing by methods other than improving the performance of CPUs.

[0011] As described above, while efforts have been made to install toilet sensors to reduce the burden of excretion management in caregiving, there have been issues with the accuracy and processing power when identifying users who require caregiver assistance through facial recognition. Furthermore, such issues can arise when managing entry to various rooms, not just toilets, and when entering a room with an accompanying person.

[0012] An object of the present disclosure is to provide an information processing device, an information processing method, and a program that solve the above-mentioned problem, which is to enable, when a person accompanying a user of a room is wearing a mask, to perform face authentication of the user in real time with high accuracy without reducing processing power, and to output the authentication result. [Means for solving the problem]

[0013] An information processing device according to a first aspect of the present disclosure includes a sensor that detects a person entering a room, and an imaging device that, when the sensor detects the person entering, captures an image of the person entering the room to obtain image data. The information processing device also includes a determination unit that determines whether the person entering the room is wearing a mask based on the image data obtained by the imaging device, thereby provisionally determining whether the person entering the room is a person to be authenticated as a user of the room. The information processing device also includes a face authentication unit that, when the determination unit determines that the person is a person to be authenticated as a user of the room, performs face authentication processing based on facial image data of the person to be authenticated in the image data, and an output unit that outputs the result of the face authentication processing to a notification destination. The face authentication unit performs the face authentication processing by comparing the facial image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may accompany the person to be authenticated and enter the room.

[0014] An information processing method according to a second aspect of the present disclosure includes: detecting a person entering a room with a sensor; and, when the sensor detects the person entering the room, capturing an image of the person entering the room with an imaging device to obtain image data. The information processing method determines whether the person entering the room is wearing a mask based on the image data obtained by the imaging device, thereby provisionally determining whether the person entering the room is a person to be authenticated as a user of the room. If the provisional determination results in the person being a person to be authenticated, the information processing method performs facial recognition processing based on facial image data of the person to be authenticated in the image data, and outputs the results of the facial recognition processing to a notification destination. The facial recognition processing is performed by comparing the facial image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may accompany the person to be authenticated and enter the room.

[0015] A program according to a third aspect of the present disclosure is a program for causing a computer to execute information processing. The information processing receives input of a result of a sensor detecting entry of a person into a room, and, when the sensor detects entry of a person, causes an imaging device to capture an image of the person who entered the room to obtain image data. The information processing determines whether the person who entered the room is wearing a mask based on the image data obtained by the imaging device, thereby provisionally determining whether the person who entered the room is a person to be authenticated as a user of the room or not. If the provisional determination results in the person being a person to be authenticated, the information processing executes facial recognition processing based on facial image data of the person to be authenticated in the image data, and outputs the result of the facial recognition processing to a notification destination. The facial recognition processing is executed by comparing the facial image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may accompany the person to be authenticated and enter the room. [Effects of the Invention]

[0016] The present disclosure provides an information processing device, information processing method, and program that can perform facial authentication of a user in real time with high accuracy and without reducing processing power when a person accompanying a room user is wearing a mask, and output the authentication results. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a block diagram showing an example of the configuration of an information processing device according to a first embodiment. [Figure 2] FIG. 10 is a diagram illustrating an example of the configuration of an information processing system including an information processing device according to a second embodiment. [Figure 3] 3 is a side view showing an example of installation of an information processing device in the information processing system of FIG. 2. [Figure 4] 3 is a block diagram showing an example of the configuration of an information processing device in the information processing system of FIG. 2. FIG. [Figure 5] 3 is a schematic diagram for explaining an example of a processing flow for generating and registering face model data in the information processing system of FIG. 2. FIG. [Figure 6] 5 is a schematic diagram for explaining an example of face authentication processing in the information processing device of FIG. 4 when the caregiver is not wearing a mask. FIG. [Figure 7] FIG. 5 is a schematic diagram for explaining an example of face authentication processing in the information processing device of FIG. 4 when a caregiver is wearing a mask. [Figure 8] 3 is a diagram for explaining an example of a notification made by a terminal device in the information processing system of FIG. 2. FIG. [Figure 9] 5 is a flowchart illustrating an example of face authentication processing in the information processing device of FIG. 4. FIG. [Figure 10] FIG. 10 is a schematic diagram illustrating an example of the configuration of an information processing device according to a third embodiment. [Figure 11] FIG. 2 illustrates an example of a hardware configuration of the apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments will be described with reference to the drawings. In the embodiments, the same or equivalent elements may be denoted by the same reference numerals, and redundant description will be omitted as appropriate.

[0019] Hereinafter, embodiments will be described with reference to the drawings. In the embodiments, the same or equivalent elements may be denoted by the same reference numerals, and redundant description will be omitted as appropriate.

[0020] <Embodiment 1> FIG. 1 is a block diagram showing an example of the configuration of an information processing device according to the first embodiment. As shown in FIG. 1, an information processing device 1 according to this embodiment can include a sensor 1a, an imaging device 1b, a determination unit 1c, a face authentication unit 1d, and an output unit 1e.

[0021] The sensor 1a is a sensor that detects the entry of a person into a room that is the subject of entry monitoring, and can be a human presence sensor installed in a position where it can detect entry. The room that is the subject of entry monitoring can be a variety of rooms, such as a toilet stall or a bathroom.

[0022] The following description will be given using an example in which the person to be authenticated, who is the person to be entry monitored, is a care recipient, that is, an example in which the room to be entry monitored is a room used by the care recipient. However, other types of authentication targets can also be entry monitored. Here, the authentication target refers to a person for whom face authentication is performed by the face authentication unit 1d and the results are output. A care recipient can also be referred to as a person receiving care, a person requiring care, or a person requiring assistance. A caregiver can be a caregiver, a doctor, or the like, and can also be referred to as an assistant.

[0023] When the person to be authenticated is a care recipient, the rooms to be monitored include the above-mentioned toilet, bathroom, etc. The information processing device 1 can be used in places where it is necessary to watch over people who require care, such as elderly care facilities, private homes, and hospitals.

[0024] Examples of the sensor 1a include an infrared sensor and an optical sensor. The sensor 1a is not limited to infrared sensors, and may use any type of sensor, such as image recognition or ultrasonic waves. It is also preferable that the sensor 1a be able to detect a person leaving a room after entering it. In a simple example, the sensor 1a can detect that a person has left the room when it detects a person entering the room and then detects that person.

[0025] When the sensor 1a detects a person entering the room, the imaging device 1b captures an image of the person entering the room to obtain image data. The imaging device 1b can be, for example, a camera that captures still images and / or videos, and can be called a face authentication camera because the acquired image data is used by the face authentication unit 1d at a subsequent stage. Furthermore, the imaging device 1b is not limited to a visible light camera, and may also be an infrared light camera, etc.

[0026] The determination unit 1c determines whether the person who has entered the room is wearing a mask based on the image data obtained by the imaging device 1b, and thereby provisionally determines whether the person who has entered the room is a person to be authenticated as a user of the room (a care recipient in this example). A provisional determination by the determination unit 1c that the person is not a care recipient means that the person is provisionally determined to be a care recipient. The determination unit 1c can also be configured to determine whether the person who has left the room is a care recipient.

[0027] If the result of the provisional determination by the determination unit 1c is that the person is a person to be authenticated (a care recipient in this example), that is, if the person is not wearing a mask, the face authentication unit 1d executes face authentication processing based on the face image data of the person to be authenticated in the image data. If the image data includes face image areas of multiple people, the face authentication processing can be executed based on the data of the face image area of ​​the person to be authenticated among all the people.

[0028] Here, the face authentication unit 1d performs the face authentication process by comparing the face image data with feature data indicating facial features stored in advance for each person to be authenticated (a care recipient in this example) and for each attendant (a caregiver in this example) who may accompany the person to be authenticated and enter the room. Therefore, if the result of the provisional determination by the determination unit 1c is that the person is an attendant (a caregiver in this example), that is, if the person is wearing a mask, the face authentication unit 1d does not perform the face authentication process and ends the subsequent processes.

[0029] The feature data may be, for example, data indicating facial feature points. This comparison may also be performed by inputting facial image data of the care recipient who has entered the room into a learning model (trained model) that has undergone machine learning using learning data including the feature data and the image data (facial image data) that is the basis of the feature data.

[0030] Then, by this facial recognition process, it is possible to identify the person who entered the target room from the facial image data of the person who has been provisionally determined to be the authentication target, that is, to identify (identify) the individual who entered the target room. The person identified here is the result of the above-mentioned comparison, and may be an attendant based on the provisional determination. The facial recognition unit 1d can also be called an identification unit because it identifies individuals in this way.

[0031] The determination unit 1c and the face authentication unit 1d may have some common components. For example, the determination unit 1c may use intermediate data from the recognition process in the face authentication unit 1d. In other words, the determination unit 1c may be included as part of the functions of the face authentication unit 1d.

[0032] In addition, it is possible that the room users may include people who do not fall into either the category of carer or caregiver (for example, family members visiting), but such people may fail to be recognized as unknown individuals as a result of the facial recognition process.

[0033] The output unit 1e outputs the result of the face authentication processing to the notification destination. The output unit 1e can be, for example, a communication unit configured with a wired or wireless communication interface or the like. The output unit 1e can, for example, transmit information indicating the result of the face authentication processing to a terminal device (not shown) of the notification destination via the communication unit. The output unit 1e can, for example, transmit this information to an external server (not shown) via the communication unit, so that the server can transfer the notification information to the terminal device of the notification destination. The notification destination can be set in advance, and it is preferable that the notification destination be changeable.

[0034] Furthermore, the output unit 1e can output the result of the facial recognition processing to the notification destination regardless of the result of the facial recognition processing. However, if the output unit 1e authenticates an attendant (a caregiver in this example) in the facial recognition processing, the output unit 1e can also output the result of the facial recognition processing to the notification destination, excluding the result of the facial recognition processing for the attendant. In other words, the output unit 1e can output the result to the notification destination when the person to be authenticated (the care recipient in this example) is authenticated, and can not output the result when the attendant is authenticated. Furthermore, in the case of an unknown person, the output unit 1e may or may not notify that an unknown person has entered the room, and it is only necessary to determine in advance which processing to execute.

[0035] The information processing device 1 may also have a control unit (not shown), which may include the above-mentioned determination unit 1c, face authentication unit 1d, and output unit 1e (or parts thereof). The control unit may be realized, for example, by a CPU (Central Processing Unit), a working memory, and a non-volatile storage device storing a program. The program may be a program for causing the CPU to execute a process of inputting a detection result from the sensor 1a, a process of inputting image data captured by the imaging device 1b, and the processes of the determination unit 1c, face authentication unit 1d, and output unit 1e. The control unit provided in the information processing device 1 may also be realized, for example, by an integrated circuit. The storage device may also store a notification destination, and the output unit 1e may output the result of the face authentication process by referring to the notification destination.

[0036] Furthermore, although the information processing device 1 according to the present embodiment is assumed to be configured as a single information processing device, it can also be configured as multiple devices with distributed functions. In the latter case, each device is provided with a control unit, a communication unit, and, if necessary, a storage unit, etc., and these multiple devices can be connected via wireless or wired communication as necessary to cooperate and realize the functions of the information processing device 1.

[0037] Here, a brief description will be given of an information processing method according to this embodiment. This information processing method performs the following information processing. In this information processing, sensor 1a detects a person entering a room. When sensor 1a detects the person entering the room, image capture device 1b captures the person and obtains image data. Furthermore, the information processing determines whether the person entering the room is wearing a mask based on the image data obtained by image capture device 1b, thereby provisionally determining whether the person entering the room is a person to be authenticated as a room user. Furthermore, if the provisional determination results in a person being a person to be authenticated, the information processing performs facial recognition processing based on facial image data of the person to be authenticated in the image data, and outputs the results of the facial recognition processing to a notification destination. The facial recognition processing is performed by comparing the facial image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant. Here, if an attendant is identified as a result of the facial recognition processing, the results of the facial recognition processing for the attendant can be excluded from the output to the notification destination. The above-described program can also be a program for causing a computer to execute the above-described information processing.

[0038] As described above, in this embodiment, if a person accompanying a user of a room (for example, an attendant such as a caregiver) is wearing a mask, facial authentication of the user can be performed in real time with high accuracy without reducing processing power, and the authentication results can be output.

[0039] For example, this embodiment includes a function for pre-registering facial feature data of caregivers and performing face recognition processing, as well as a mask detection function, enabling highly accurate face recognition processing even with a space-saving, power-saving CPU built into an information processing device serving as an edge device. To give a more specific example, this embodiment utilizes these functions to exclude a caregiver wearing a mask from face recognition processing when they enter the restroom for cleaning or other purposes by simply performing a mask determination process, which requires less processing time than face recognition processing. Therefore, this embodiment allows face recognition processing without significantly impairing performance even when multiple caregivers are assisting a single user. Furthermore, this embodiment can also be configured so that a notification is not sent even when a caregiver not wearing a mask enters the restroom.

[0040] <Embodiment 2> 2 to 9, the second embodiment will be described, focusing on the differences from the first embodiment, but the various examples described in the first embodiment can also be applied. Fig. 2 is a diagram showing an example of the configuration of an information processing system including an information processing device according to the second embodiment, and Fig. 3 is a side view showing an example of installation of the information processing device in the information processing system of Fig. 2.

[0041] An information processing system according to this embodiment (hereinafter referred to as the present system) can be a system in which a part of it is installed in a toilet and sends notifications to notification recipients (for example, caregivers, etc.), and will be specifically described below.

[0042] As shown in Figure 2, the present system can include an information processing device 60 attached to the toilet 10, a server 40 wirelessly connected to the information processing device 60, and a terminal device 50 wirelessly connected to the server 40. These connections can be made, for example, within a single wireless LAN (Local Area Network). For example, the present system can also be configured so that the information processing device 60 can send information directly to the terminal device 50. In the following, an example will be described in which the information processing device 60 is installed in the toilet 10 and also acquires excretion information, but excretion information can be acquired if at least a part of it is installed in the toilet 10, such as a camera for capturing image data to obtain the excretion information.

[0043] The toilet bowl 10 can have an edge 10c that forms a water-retaining area between itself and the side surface 10a, and can also have a flat surface 10b that continues from the edge 10c. The flat surface 10b can be provided with a toilet seat 11 that includes, for example, a toilet seat body 11a equipped with a warm water cleaning function for the user to wash, and a toilet seat cover 11b that covers the toilet seat body 11a.

[0044] The information processing device 60 can be a device installed in the toilet (installed in the toilet in this example), and notifies the terminal device 50 of detection events, such as the detection of the entry of a care recipient, directly or via the server 40. Here, an example will be described in which the information processing device 60 is an excretion information collecting device with the function of acquiring excretion information. In this case, excretion-related events, such as the start and end of excretion, can also be notified as detection events. Although not described in detail, the information processing device 60 can also be configured to record excretion, present excretion information (notification, etc.), and predict excretion in cooperation with the server 40 and the terminal device 50. The information processing device 60 and the toilet 10 can, for example, constitute a toilet with a function to output excretion information.

[0045] 2, the shape of the information processing device 60 is not limited to the shape shown in Fig. 2, and it may be configured such that all or part of its functions are embedded in the toilet seat body 11a, etc. Also, part of the functions of the information processing device 60 may be provided on the toilet seat body 11a side.

[0046] 1, and has the functions of each unit of the information processing device 1. Before describing a detailed example of the information processing device 60, a brief description will be given of the main components, including components corresponding to each unit of the information processing device 1 in FIG.

[0047] As shown in Figures 2 and 3, the information processing device 60 of this embodiment can include a human presence sensor 24, an imaging device 25, an information collection unit 26, a storage housing 61, a bridge unit 62, an inner housing 63, a control unit 67, a separate housing 68, and an outer housing 69.

[0048] The human sensor 24 is an example of the sensor 1a, and is a sensor that detects a person entering a toilet stall, such as a user (customer) of the toilet stool 10, and may be, for example, an infrared sensor or an optical sensor.

[0049] Imaging device 25 is an example of imaging device 1b, and is a device that captures the face of a person entering a toilet stall, such as a toilet user, and can be, for example, a camera that captures still images and / or videos (a face authentication camera). Furthermore, imaging device 25 is not limited to a visible light camera, and may also be an infrared light camera, etc.

[0050] The information collecting unit 26 is disposed in the inner housing 63. However, the information collecting unit 26 is disposed so that information can be collected from the collecting portion (area where excrement is excreted) of the toilet bowl 10, for example, by leaving the information collecting surface of the information collecting unit 26 exposed from an opening in the inner housing 63. The information collecting surface corresponds to, for example, a lens surface if the information collecting unit 26 is a camera, or a detection surface if the information collecting unit 26 is a sensor.

[0051] The information collecting unit 26 is a component that collects information about excrement in the toilet 10, and may be, for example, an imaging device that collects information about the excrement, such as its shape and color, through an optical lens, or a distance sensor that optically measures distance. The excrement information may be information indicating the excrement's content, and in a simpler example, information indicating whether the excrement is feces (stool) or urine (urine). The excrement information may also include other information, such as information indicating the excrement's color and, if it is solid, its shape. The information collecting unit 26 may simply be configured to detect and collect the excrement information.

[0052] The information collecting unit 26 can set the area including the water-filled portion of the toilet bowl (excretion area) as the collection area, and this excretion area can also be called the expected excretion area. By installing the information collecting unit 26 so that such excretion area is included in the imaging area, excrement and the like will be included as subjects in the captured imaging data. Of course, it is preferable that the excretion area be set to an area that does not capture the user, and it is also preferable that the information collecting unit 26 be installed so that lenses and the like are not visible to the user.

[0053] The information collection unit 26, the human presence sensor 24, and the imaging device 25 are connected to the control unit 67 by wire or wireless communication. Note that Fig. 2 shows an example in which the information collection unit 26 is connected to the control unit 67 by a cable K1, and the human presence sensor 24 and the imaging device 25 are connected to the control unit 67 by a cable K2.

[0054] The control unit 67 controls the human presence sensor 24, the imaging device 25, and the information collection unit 26. For example, the control unit 67 can receive the detection result of the human presence sensor 24, instruct the imaging device 25 to capture an image and receive the captured image, and instruct the information collection unit 26 to collect excrement information and receive the collected information. The control unit 67 can also be configured to execute the functions of the determination unit 1c, the face authentication unit 1d, and the output unit 1e. The control unit 67 can be realized, for example, by a CPU, a working memory, and a non-volatile storage device storing a program. The program can be, for example, a program for causing the CPU to execute the above-mentioned processes. The control unit 67 can also be realized, for example, by an integrated circuit.

[0055] The storage housing 61 is a housing that is placed on the outside of the toilet bowl 10 and that houses the control unit 67, and will hereinafter be described as the control box 61. The bridge 62 is a part that bridges the inner housing 63 and the outer housing 69. In other words, the bridge 62 is a mechanism that connects the inner housing 63 and the outer housing 69, and is installed on the edge 10c of the toilet bowl 10. The outer housing 69 is a housing that includes a clamp mechanism for fixing the inner housing 63 and the toilet bowl 10. The outer housing 69 is a housing that is placed on the outside of the toilet bowl 10, and its shape does not matter as long as it does not get in the way of excretion.

[0056] The clamp mechanism (fixing clamp mechanism) described above can be a mechanism that fixes the inner housing 63 and the outer housing 69 to the edge 10c, for example, by clamping them while adjusting the distance between them and the inside of the edge 10c via the bridge 62. This allows the inner housing 63 of the information processing device 60 to be reliably fixed and installed regardless of the shape of the toilet bowl 10, such as the shape of the edge 10c. Furthermore, as long as it is such a mechanism, the shape and components of the fixing clamp mechanism are not important. The main components of the fixing clamp mechanism can be provided on the outer housing 69.

[0057] The inner housing 63 is a housing in which the information collection unit 26 is disposed, and will be hereinafter referred to as the sensor box 63. The sensor box 63 is equipped with, for example, an optical sensor for collecting information about excrement, and is placed inside the toilet bowl 10.

[0058] The separate housing 68 is a housing in which the human presence sensor 24 and the imaging device 25 are disposed. The separate housing 68 accommodates the human presence sensor 24 for human detection and the imaging device (camera) 25 for facial authentication either inside or partially outside, and will be described below as the human detection / identification box 68. The human detection / identification box 68 may be provided with a camera adjustment mechanism for adjusting the direction of the camera horizontally. The imaging device 25 may have a lens exposed from the human detection / identification box 68 so as to be able to capture an image of the face of a toilet user, but it is sufficient if it can capture an image of a face. The human presence sensor 24 may have a detection surface exposed from the human detection / identification box 68 so as to be able to detect toilet users, but it is sufficient if it can detect a person. If the human presence sensor 24 and the imaging device 25 are connected to the control unit 67 via wired or wireless communication, they can send and receive information.

[0059] As shown in Figures 2 and 3, the human detection / identification box 68 can be disposed at a position separated from the control box 61 and the toilet 10. Figures 2 and 3 show an example in which the human detection / identification box 68 is installed on the wall W of the toilet where the toilet 10 is installed. Of course, the human detection / identification box 68 can also be installed on a location other than the wall W, such as the ceiling or a shelf installed in the room. The human detection / identification box 68 can be connected to the control box 61 via a wired cable K2 or wirelessly, and in either case, can be used at a position separated from the control box 61. The human detection / identification box 68 rotates horizontally, and a mechanism for fixing the orientation of the rotating box can be provided.

[0060] Furthermore, in order to facilitate the detection and imaging of people, the human presence sensor 24 and the imaging device 25 are preferably disposed so that the detection surface and imaging lens are located at a position at least higher than the bridge 62. It is more preferable to provide the detection surface and imaging lens at a position that is a predetermined height higher than the height of the toilet seat 11.

[0061] 2, the control box 61 having the human detection / identification box 68 and the control unit 67, and the sensor box 63 having the excretion information collection function are connected by cables K1 and K2. However, the human detection / identification box 68 is disposed on the wall W of the toilet, and the box having the control unit 67 is disposed on the side of the toilet 10, making the installation easy. The control box 61 can also be disposed on the back side of the toilet 10, making the installation easy in this case. It is also desirable that both cables K1 and K2 include a power supply line that supplies power from the control box 61. Power is supplied to the control box 61 from a power cable (not shown).

[0062] 2, it is assumed that the cable K1 passes through the inside of the bridge 62, but the cable K1 can also be connected to the information collection unit 26, led out from the sensor box 62, and directly connected to the control box 61. In that case, the cable K1 passes through the upper surface side of the edge 10c of the toilet bowl 10 (passes between the upper surface of the edge 10c and the toilet seat body 11a).

[0063] The server 40 may include a control unit 41 that controls the entire server 40 and a storage unit 42 that stores, for example, notification conditions and various acquired information (and information generated based on the notification conditions). The control unit 41 may be configured to transfer notifications to the terminal device 50 as part of the functions of the output unit 1e in FIG. 1. For example, the storage unit 42 may store, as notification conditions, notification destinations for each detected event and for each user being assisted. This allows the server 40, which has part of the functions of the output unit 1e, to output to the notification destinations in response to the occurrence of a detected event in accordance with the notification conditions. Note that the number of notification destinations for a certain detected event may be one or more. The server 40 also preferably includes a setting unit that sets notification conditions from an external device such as the terminal device 50, thereby enabling the notification conditions to be set according to the operation mode.

[0064] The control unit 41 can be realized, for example, by a CPU, a working memory, and a non-volatile storage device that stores a program. The control unit 41 can also be realized, for example, by an integrated circuit. This storage device can also serve as the storage unit 42, and this program can be a program that causes the CPU to realize the functions of the server 40.

[0065] The terminal device 50 is a terminal device carried by a caregiver who is caring for (or will be caring for) a care recipient who is using the toilet, and can be a portable information processing device, but it can also be a stationary device. In the former case, the terminal device 50 can be a smart device such as a mobile phone (including what is called a smartphone), a tablet, or a mobile PC, or it can also be an electronic bulletin board or the like. Although not shown, the terminal device 50 can also include a control unit that controls the entire device and a storage unit, and this control unit, like the control unit 41, can be realized by, for example, a CPU, a working memory, a storage unit, etc. Furthermore, the program stored in this storage unit can be a program that causes the CPU to realize the functions of the terminal device 50.

[0066] Next, a detailed example of the information processing device 60 will be described with reference to FIG. 4. FIG. 4 is a block diagram showing an example of the configuration of an information processing device in the information processing system of FIG. 2. The information processing device 60 can be composed of two devices, for example, as shown in FIGS. 2 and 3. The information processing device 60 can also be called a "toilet sensor" or "toilet sensor device" because it has various sensors installed in the toilet. The toilet sensor corresponds to a so-called edge in this system, which monitors the use of the toilet by the care recipient via a communication network.

[0067] The control box 61 can house, for example, a CPU 27a, a connector 27b, USB I / Fs 27c and 27d, and a WiFi module 27e. Note that USB is an abbreviation for Universal Serial Bus, and both USB and WiFi are registered trademarks (the same applies below). Note that the control box 61 may not be provided with various I / Fs and connectors and may be directly connected to the CPU 27a.

[0068] 4, the control box 61 and the sensor box 63 are connected by an interface exemplified by the connector 27b and the USB I / F 27c, and the connecting wire is housed in the cable K1. A part of the cable K1 passes through the inside of the bridge 62 (and the outer housing 69). The control box 61 and the human detection / identification box 68 are connected by an interface exemplified by the connector 27b and the USB I / F 27d, and the connecting wire is housed in the cable K2.

[0069] In this way, the information processing device 60 is provided with circuit boards and cables connecting them in each box and bridge 62. By adopting the divided box configuration described above and optimizing the orientation of the connectors that connect the cables, the information processing device 60 can reduce the space required for excess cable length and reduce the size of the circuit board and mounted components. Also, in this embodiment, by separating the placement locations of each box, the width of each structure can be reduced.

[0070] The sensor box 63 can house, for example, a distance sensor 26a that functions as a seating sensor that detects when the toilet seat 11 is sat on, and a first camera 26b that photographs excrement.

[0071] Next, the components in each box will be described. The CPU 27a is an example of a main control unit of the information processing device 60, and controls the entire information processing device 60. The connector 27b connects the human presence sensor 24 and the distance sensor 26a to the CPU 27a. The USB I / F 27c connects the first camera 26b to the CPU 27a, and the USB I / F 27d connects the second camera 25 to the CPU 27a.

[0072] Distance sensor 26a is a sensor that measures the distance to an object (the buttocks of the user of toilet bowl 10) and detects that the user has sat on toilet seat 11, and detects that the object has sat on toilet seat 11 when a certain time has passed since the distance exceeds a threshold value. Furthermore, distance sensor 26a detects that the user has left toilet seat 11 when the distance to the object changes after sitting down.

[0073] The distance sensor 26a may be, for example, an infrared sensor, an ultrasonic sensor, or an optical sensor. If an optical sensor is used for the distance sensor 26a, a transmitting / receiving element may be disposed so as to transmit and receive light (not limited to visible light) through a hole provided in the sensor box 63. The transmitting / receiving element may be configured as a separate transmitting element and a receiving element, or may be integrated. The distance sensor 26a is connected to the CPU 27a via the connector 27b, and is capable of transmitting the detection result to the CPU 27a. Based on the detection result, the CPU 27a can obtain seating / non-seating data indicating information on whether a person is seated or not, and can also transmit the data to the server 40 via the WiFi module 27e.

[0074] First camera 26b is an example of a camera that captures image data that is the basis for acquiring excretion information, and can be an optical camera with a lens portion disposed in a hole provided in sensor box 63. First camera 26b is installed so that the image capture range includes the excrement excretion range on toilet bowl 10. First camera 26b is connected to CPU 27a via USB I / F 27c, and transmits the image capture data to CPU 27a.

[0075] The human presence sensor 24 is an example of the sensor 1a, and is a sensor that detects the presence of a person (entry or exit) in a specific area (measurement area range of the human presence sensor 24) that is part of the toilet room, and can be called an entry / exit sensor. The human presence sensor 24 can employ any detection method, such as an infrared sensor, ultrasonic sensor, or optical sensor. The human presence sensor 24 is connected to the CPU 27a via a connector 27b, and when it detects a person in the specific area, it transmits the detection result to the CPU 27a. The detection result can also be transmitted by the CPU 27a to the server 40 via the WiFi module 27e.

[0076] The second camera 25 is an example of the imaging device 1b, and may be an optical camera, an example of a face recognition camera that captures a facial image of a user or the like to identify the user of the toilet and obtains face image data. The second camera 25 may be installed in the toilet room (inside the toilet) where the toilet 10 is installed. The CPU 27a performs face recognition processing using the face image data captured by the second camera 25. In this system, when the human presence sensor 24 detects a person, the second camera 25 can be processed to capture an image of the subject entering the toilet, thereby enabling face recognition to be performed only when a person is detected.

[0077] WiFi module 27e is an example of a communication device that transmits various acquired data to server 40, and can be replaced with a module that employs other communication standards. Data to be transmitted can be transmitted to server 40 by CPU 27a via WiFi module 27e. Note that facial image data acquired by second camera 25 is not generally transmitted in order to reduce the amount of data and to protect privacy.

[0078] CPU 27a provisionally determines whether the person who has entered the room is a care recipient or not as a user of the room by determining whether the person is wearing a mask based on image data obtained by second camera 25. A provisional determination that the person is not a care recipient means that the person is provisionally determined to be a caregiver.

[0079] Furthermore, if the result of the provisional determination is that the person is a care recipient, i.e., the person is not wearing a mask, the CPU 27a executes face recognition processing based on the facial image data of the care recipient in the image data. If the image data includes facial image regions of multiple people, the face recognition processing can be executed based on the data of the facial image region of the care recipient among all the people. Here, the face recognition processing is executed by comparing the facial image data with feature data indicating facial features stored in advance for each care recipient and each caregiver who may enter the room accompanying the care recipient. Therefore, in the face recognition processing, if the result of the provisional determination is that the person is a caregiver, i.e., the person is wearing a mask, the face recognition processing is not executed and the subsequent processing is terminated.

[0080] The feature data may be, for example, data indicating facial feature points. The feature data may be data generated from facial image data acquired by a terminal device such as the terminal device 50 and stored in an internal storage device of the information processing device 60. This internal storage device may be connected to the CPU 27a (not shown), or may be provided within the CPU 27a. This comparison may also be performed by inputting facial image data of the care recipient who has entered the room into a learning model (trained model) that has undergone machine learning using learning data including the feature data and the image data (facial image data) that is the basis of the feature data.

[0081] This facial recognition process can identify the person who entered the room from the facial image data of the person who has been provisionally determined to be a care recipient, i.e., identify the individual who entered the room. The person identified here is the result of the comparison described above, and may be provisionally determined to be a caregiver. In such cases, the facial recognition result may not be output. It is also possible for the room users to include people who do not qualify as either care recipients or caregivers (e.g., family members visiting a patient). Such people may be identified as unknown individuals through the facial recognition process and may be notified accordingly. In consideration of privacy, it is preferable not to store the facial image data captured by the second camera 25 after the facial recognition process.

[0082] The CPU 27a and the WiFi module 27e can be an example of the output unit 1e in FIG. 1. The CPU 27a outputs the result of the face recognition process to the notification destination via the WiFi module 27e. If the CPU 27a authenticates a caregiver in the face recognition process, it can also output the result of the face recognition process to the notification destination, excluding the result of the face recognition process for the caregiver. If the result of the face recognition process identifies an unknown person, the CPU 27a may or may not notify that an unknown person has entered the room, and it is only necessary to determine in advance which process to execute.

[0083] In this way, the CPU 27a can be responsible for the main processing of detecting detected events indicated by the results of the facial recognition processing, excretion information, and seating / leaving information. The notification destination can be the terminal device 50 held by one or more caregivers, and the caregiver to be notified can be included in the notification conditions. The caregiver can be notified by displaying and / or outputting audio on the terminal device 50. While the terminal device 50 has been described assuming that it is a smart device, it may also be a device that simply emits a sound when a notification is received, or an electronic bulletin board that displays the entry of a room, a person's name, etc. In this way, the notification destination may be, in addition to or instead of the terminal device 50, a notification device of a nurse call system, another terminal device (e.g., a PHS (Personal Handy-phone System)) held by the caregiver other than the terminal device 50, an intercom, etc. While details of the notification of excretion information and seating / leaving information will be omitted, both can be treated as one of the detected events.

[0084] Furthermore, the CPU 27a, the USB I / F 27c, the WiFi module 27e, and the server 40 obtain the above-mentioned excretion information based on the imaging data captured by the first camera 26b. In this case, the server 40 can be mainly responsible for the process of obtaining the excretion information from the imaging data. However, the CPU 27a can also be configured to perform this process. In order to obtain the excretion information indicating the details of the excretion based on the imaging data, the server 40 or the CPU 27a includes an analysis unit that analyzes the excretion information based on the imaging data obtained by the first camera 26b. This analysis unit performs analysis of the excretion information when the care recipient can be authenticated as a result of the face recognition process, that is, when the entrant can be identified and is the care recipient. Imaging by the first camera 26b can also be performed only after the result of such face recognition process is obtained.

[0085] In the example of FIG. 2, the server 40 inputs image data into a trained model to acquire excretion information. Preferably, the server 40 primarily acquires, as part of the excretion information, information indicating whether the image data contains a foreign object, i.e., an object other than feces and urine, as a subject other than the toilet bowl and its cleaning liquid. The foreign object may be referred to as a "other object," and may be liquid or solid as long as it is something other than feces and urine. For example, the foreign object may include one or more of vomit, bloody stool, vomited blood (hematemesis), diapers, urine pads, and toilet paper rolls. Furthermore, if the object is excretion rather than a foreign object, the server 40 may acquire the shape, color, and amount of the excretion as excretion information. The CPU 27a and the WiFi module 27e, or the server 40, may also notify the recipient of the analysis results from the analysis unit. The recipient of the analysis results may be different from the recipient of the facial recognition processing results. The recipient of the analysis results may be stored, for example, as a notification condition in the storage device or the storage unit 42 of the information processing device 60.

[0086] Furthermore, the server 40 can store information such as excretion information and information on leaving and sitting, for example, in the storage unit 42, and can output the information in response to access from the terminal device 50. In particular, it is desirable for the server 40 to have a function for generating an excretion diary. In this case, if the generated excretion diary is stored in the storage unit 42, the caregiver who is the user of the terminal device 50 can view the excretion diary whenever desired from the terminal device 50. The excretion diary will include the date and time of excretion behavior, and this date and time can be obtained from any of the date and time of imaging data of excretion, the date and time of identification of the subject, the date and time of entry and exit, the date and time of sitting, the date and time of leaving, the date and time between sitting and leaving, etc.

[0087] Next, an example of a process for generating and registering facial feature data will be described with reference to Fig. 5. Fig. 5 is a schematic diagram for explaining an example of the flow of a process for generating and registering facial model data in the information processing system of Fig. 2.

[0088] As shown in FIG. 5, facial feature data (hereinafter, referred to as face model data) is registered by generating face model data and transmitting the face model data to the information processing device 60. First, the face photograph used to generate the face model data can be image data obtained by a caregiver or the care recipient using a terminal such as the terminal device 50. This terminal transmits the obtained image data to a server 40 on the cloud, and the server 40 generates face model data for face recognition from the image data. Regardless of the type of face model data, it is sufficient that the recognition accuracy in the face recognition process is high, but face model data that reduces the processing load in the face recognition process is preferable. The face model data can be data including information extracted from a face photograph, such as the eyes, mouth, nose, and contours. Alternatively, the care recipient can take a face photograph of themselves using a terminal similar to the terminal device 50 and transmit the image data to the server 40, which then generates the face model data.

[0089] Thereafter, the server 40 transmits the generated face model data to the information processing device 60, and the information processing device 60 registers, i.e. installs, and holds the face model data so that the face model data can be referenced during face authentication processing. When transmitting data from the server 40 to the information processing device 60, it is possible to transmit only the modeled data in this way, without transmitting image data such as a facial photograph itself.

[0090] The generation of facial model data can also be executed by a server other than the server 40. Furthermore, when facial model data can be generated by a terminal such as the terminal device 50 that took the facial photograph, the data generated by the terminal can be sent directly to the information processing device 60 without using a server such as the server 40.

[0091] Next, functions corresponding to the determination unit 1c and face authentication unit 1d, that is, face detection, face identification (face authentication), and mask detection, will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a schematic diagram for explaining an example of face authentication processing in the information processing device 60 when the caregiver is not wearing a mask. Fig. 7 is a schematic diagram for explaining an example of face authentication processing in the information processing device 60 when the caregiver is wearing a mask.

[0092] The processes of face detection, face identification (face authentication), and mask detection can be performed using neural network-based algorithms based on machine learning. Figure 6 shows an overview of the face authentication function when the caregiver is not wearing a mask. As shown in Figure 6, when the CPU 27a receives the detection result from the motion sensor 24 and detects entry into the restroom, it activates the second camera 25 for face authentication, scans the imaging range of the second camera 25, and when a face is detected, it surrounds the detected face with a rectangle. As a result, as shown in image FD in Figure 6, the facial areas of the care recipient and the caregiver in the imaging data are surrounded by rectangles fa1 and fa2, respectively.

[0093] Next, the CPU 27a performs face recognition processing by comparing the detected face areas fa1 and fa2 with face model data, as shown in image FR1 in FIG. 6. If the face is determined to be a caregiver as a result of the face recognition processing (in this example, face area fa2 is determined to be the face of a caregiver), the CPU 27a excludes face area fa2 from the target of face recognition, as shown in image FR2 in FIG. 6, even if the caregiver is within the imaging range of the second camera 25. The determination that the face is a caregiver can be made, for example, by first detecting certain feature points in the face model data of the caregiver. These feature points can be registered in advance, such as a hat, hairpin, or hairstyle. Finally, the CPU 27a performs face recognition processing on face area fa1 in image FR2, and can identify this face as care recipient A.

[0094] 7 shows an overview of the face recognition function when the caregiver is wearing a mask. First, when CPU 27a receives a detection result from motion sensor 24 and detects entry into the toilet, it activates second camera 25 for face recognition, scans the imaging range of second camera 25, and when a face is detected, surrounds the detected face with a rectangle. As a result, as shown in image FD in FIG. 7, the facial areas of the care recipient and the caregiver in the imaging data are surrounded by rectangles fa1 and fa2, respectively.

[0095] When there are masks in the rectangular facial areas fa1 and fa2, the CPU 27a determines that a mask is being worn by surrounding the mask with a rectangle (in this case, rectangular area ma2) as shown in image MD in Fig. 7. If the CPU 27a determines that the person is wearing a mask, that is, a caregiver, as a result of the mask wearing determination, the CPU 27a excludes the facial area fa2 from the target of face recognition as shown in image FR in Fig. 7, and does not perform face matching, even if the caregiver is within the imaging range of the second camera 25. The CPU 27a executes face recognition processing for matching the face area fa1 not wearing a mask in image FR in Fig. 7 with face model data, and can ultimately identify this face as care recipient A.

[0096] 6 and 7, a rectangular area is actually enclosed, but it is sufficient if a rectangular area can be detected and specified. Furthermore, the shape of the face can be other than rectangular, such as an ellipse. Furthermore, the mask can also be other shapes, such as a polygon with five or more sides or an ellipse, instead of a rectangle.

[0097] Next, a display example of notification information on the terminal device 50 will be described with reference to Fig. 8. Fig. 8 is a diagram for explaining a notification example on the terminal device in the information processing system of Fig. 2.

[0098] 8, terminal device 50 can notify notification information 52 to 54 in that order on its display screen 51. Notification information 52 is an example of notification information that notifies that a care recipient has entered the toilet when a person is detected by human sensor 24 and the person is provisionally determined to be a care recipient. As in this example, a notification may also be sent to the notification destination when the provisional determination detects that a care recipient has entered the toilet, that is, when the entry of a person not wearing a mask has been detected.

[0099] After notification of notification information 52, notification information 53 is notified as a result of the facial recognition process. This is an example of notification information that notifies that the person who entered the room is the user (care recipient PA in this example) as a result of identifying the user through facial recognition process. Note that it is possible to know that the person is PA if a name is associated with the face model data.

[0100] Furthermore, if it is detected that PA has left the room, notification information 54 indicating this can be displayed. Furthermore, even if, after notification information 52 is sent, facial recognition processing detects that the person entering the room is a caregiver, the name of the individual caregiver (for example, Mr. C) can be notified, but no notification need be made. As an alternative process, in this case, a notification can be made that excludes the results of facial recognition processing, that is, a notification can be made simply to the effect that the person entering the room was a caregiver.

[0101] Next, a flow of face authentication processing and notification in this system, triggered by the entry of a care-requiring person into a room, will be described with reference to Fig. 9. Fig. 9 is a flow diagram for explaining an example of face authentication processing in the information processing device 60. Note that the processing in the information processing device 60 is mainly executed by the CPU 27a.

[0102] First, the information processing device 60 monitors whether or not there is a reaction from the human presence sensor 24, which functions as an entrance sensor (step S1), and when there is a reaction (when step S1 is YES), it activates the second camera 25, which is an optical camera (step S2). If step S1 is NO, that is, when there is no reaction from the human presence sensor 24, it waits until there is a reaction.

[0103] In this way, when a person requiring care enters the toilet, the result in step S1 becomes YES, and second camera 25 is activated. The activated second camera captures an image of the person who has entered, and CPU 27a or second camera 25 itself performs a process of detecting a face within the image capturing range and determines whether or not a face has been detected (step S3). In step S3, as illustrated in FIGS. 6 and 7, when second camera 25 detects a face, the detected face is surrounded by a rectangle. The determination in step S3 is performed while changing the image capturing range of second camera 25 as necessary and capturing images until a face is detected. Step S3 is executed until a face is detected, but the process can also be terminated, for example, after a predetermined time has elapsed.

[0104] The CPU 27a performs mask detection by determining whether or not the face area detected in step S3 is wearing a mask (step S4). If step S4 returns YES, that is, if it detects that the person is wearing a mask, the CPU 27a determines that the person is a caregiver, terminates the face authentication process, and does not perform any further authentication process on the face image of the person. On the other hand, if step S4 returns NO, that is, if the person is not wearing a mask, the CPU 27a compares the face area (image data of the face area) with face model data (step S5). During comparison, face model data can also be generated for the image data of the face area in the same way as when the face model data was generated, and the two can be compared.

[0105] As a result of the comparison in step S5, the CPU 27a determines whether the image data of the face area matches or corresponds to registered face model data (also simply referred to as face data) (step S6). If the result is YES in step S6, the CPU 27a determines whether the user is a care recipient or a caregiver (step S7). If the CPU 27a determines that the user is a caregiver in step S7, it ends the process. On the other hand, if the CPU 27a determines that the user is a user in step S7, it cooperates with the notification function and excrement analysis function to notify the user of an entry and analyze the excrement to notify the excretion information (step S8), and ends the process. The cooperation in step S8 can also include cooperation with the server 40.

[0106] Also, if the result in step S6 is NO, that is, if the image data of the face area is unregistered or has been registered but cannot be matched, the CPU 27a will mark the entrant as unknown (step S9), cooperate with the notification function and excrement analysis function (step S8), and end the process. Note that after the process of step S9, the process can also be ended without such cooperation. In this way, in the information processing device 60, if the identified entrant is a user, it can cooperate with processes such as analysis and notification of excrement indicating the user's toilet usage status (analysis of the user's status and notification to the caregiver).

[0107] The above explanation of this system is based on the assumption that there is only one toilet 10 (only one toilet room). However, it is preferable for this system to obtain face recognition results for detecting detection events for multiple toilets. This makes it possible to apply this system even when a person being assisted may use two or more toilets.

[0108] In addition, the server 40 can be installed within a facility such as a hospital, or in a private home or apartment complex for personal use. In either case, the server 40 can be a cloud server as described above.

[0109] Furthermore, the program of the terminal device 50 can be executable and embedded in the terminal device 50 as care software including a notification function of notification information. Furthermore, in addition to notification information, the terminal device 50 can also directly (or directly and automatically) receive and store information obtained on the toilet side by the information processing device 60 or the like, and similarly can receive and store various information recorded by the server 40.

[0110] Of course, it is also possible to configure the information processing device 60 by itself, without using the server 40, by providing the functions described as the functions of the server 40, so that the information processing device 60 notifies the terminal device 50 without using the server 40. In other words, it is also possible to adopt a configuration in which the information detected by the information processing device 60 is directly notified to the terminal device 50 carried by the caregiver. In this case, the notification can be made directly using, for example, Bluetooth (registered trademark) or ZigBee (registered trademark), or can be made using a communication network using, for example, Wi-Fi or LTE.

[0111] As described above, in this system, facial recognition processing can be performed based on information acquired by the information processing device 60, which corresponds to a so-called edge of a communication network, in a nursing facility or a home, and a person requiring care can be monitored. For example, the information processing device 60 is installed in a toilet where entry detection is desired. As a result, the information processing device 60 can detect entry using the human presence sensor 24, perform mask determination processing and facial recognition processing, and, if the person is a care recipient, notify the terminal device 50 of information indicating the entry and the care recipient identified as a result of the facial recognition.

[0112] In this system, the face model data used for face recognition is stored on a server on the cloud or the like, and the information processing device 60 acquires and stores the face model data in advance using a communication function. The information processing device 60 detects people using a human presence sensor 24, which serves as an entrance / exit sensor, to detect their entry and exit from the restroom. After the human presence sensor 24 detects a person, the information processing device 60 performs face recognition processing to identify the person by photographing the person entering the restroom using a second camera 25 for face recognition and comparing the photograph with face model data stored in the device and extracting the person who most closely resembles the person. The photographed photograph data is not stored in consideration of privacy. The information processing device 60 can also be configured to acquire information such as the location of the toilet entered, the user's name, entry and exit times, and whether the person sat on or left the toilet seat using the distance sensor 26a, the first camera 26b, the human presence sensor 24, and the second camera 25 for face recognition. In this case, the information processing device 60 transmits the acquired information to the server 40 via the WiFi module 27e, and the information can be notified to the terminal device 50 carried by the caregiver via the server 40. At this time, image data captured by the first camera 26b for excrement and the second camera 25 for face authentication can be prevented from being transmitted to the server 40.

[0113] In this system, with the above-mentioned configuration, the two problems specifically described as (1) and (2) can be solved as follows.

[0114] First, the problem (1) above can be solved as follows. In this embodiment, facial model data of the user and the caregiver is pre-installed in the information processing device 60. The information processing device 60 then compares the facial model data with the face captured by the facial recognition camera, and if the person who entered the room is identified as a caregiver, that person is subsequently excluded from facial recognition if they are within the imaging range of the facial recognition camera. This allows only the user to be the subject of facial recognition processing, thereby improving the accuracy of person identification without reducing the processing power of facial recognition.

[0115] However, even if caregiver facial model data is incorporated, if one of the caregivers wears a mask, the accuracy of face recognition deteriorates, as explained in the problem (2) above. Therefore, the problem (2) above is solved as follows. In this embodiment, by taking advantage of the fact that caregivers sometimes wear masks while users do not wear masks inside the facility, mask detection processing is added to the face recognition processing to reduce false recognition in face recognition. Specifically, when the information processing device 60 detects that a person has entered the restroom, it performs face detection processing on the person and performs mask detection processing on the detected face. By performing mask detection processing, it is possible to exclude caregivers wearing masks from the target of face recognition processing, and face recognition can be performed only on faces not wearing masks.

[0116] Even when machine learning is used, determining whether a person is wearing a mask is faster than face matching, making it possible to improve real-time performance even on edge devices equipped with space-saving, power-efficient CPUs. In particular, identifying users using face recognition in restrooms requires coordination with a caregiver notification function, which can assist the user with toileting or respond to accidents. This requires not only high accuracy but also real-time performance, and this system enables real-time processing. In fact, face recognition processing for masked faces is prone to misrecognition because it matches only a limited number of facial features other than the mask. In contrast, this system can exclude a mask from face recognition processing when it detects it is being worn during mask detection processing, preventing a decrease in face recognition performance and misrecognition of the user even when the user enters the room accompanied by a caregiver.

[0117] In this embodiment, by incorporating the caregiver's facial model data and performing mask detection processing as described above, facial recognition is performed only on the user, even if both the user and the caregiver are within the imaging range of the facial recognition camera. This improves the accuracy of person identification without sacrificing facial recognition processing power. In other words, even in a low-power information processing device 60 with a space-saving, power-saving CPU 27a, facial recognition of the user is possible without sacrificing processing performance or recognition accuracy. This reduces the burden on the caregiver and provides thorough support to the user. Furthermore, since the caregiver can be excluded from facial recognition in this embodiment, cases such as toilet cleaning by the caregiver can also be excluded. Furthermore, even in the case of excretion management, which is a heavy burden on the caregiver, the burden of assistance work can be reduced by linking toilet entry / exit management using facial recognition processing with the excretion analysis function.

[0118] As explained above, this system enables real-time user identification with high accuracy and minimal performance degradation, reducing the burden on caregivers and providing thorough support to users. That is, this system achieves the effects explained in the first embodiment, and in particular the following effects:

[0119] The first effect is that by registering the caregiver in advance, it is possible to reduce the chance of mistakenly detecting the caregiver as the user, and by adding a mask detection process, it is possible to obtain even greater accuracy improvements if the caregiver is wearing a mask.

[0120] The second advantage is that mask detection processing is lighter than facial recognition processing, so even when multiple caregivers are assisting a single user, it is possible to identify the user through facial recognition without significantly compromising performance.

[0121] The third effect is that the mask detection process excludes caregivers wearing masks from the target of face recognition, so that caregivers can be excluded from the target of toilet use when cleaning the toilet, etc.

[0122] <Embodiment 3> In the second embodiment, a toilet is used as an example of a room, but other types of rooms such as a bathroom can also be used. In this embodiment, an example of use in a bathroom will be described with reference to Fig. 10, but the various examples described in the first and second embodiments can also be applied. Fig. 10 is a schematic diagram showing an example of the configuration of an information processing device according to the third embodiment.

[0123] 10, the information processing device according to this embodiment does not require the components related to excretion information in the information processing device 60 according to embodiment 2. The information processing device according to this embodiment includes a control box 61 and a human detection / identification box 68 connected to the control box 61 via a cable K2. Note that the server 40 and the terminal device 50 are omitted from FIG. 10.

[0124] The human detection / identification box 68 can be installed inside the bathroom, or can be placed outside the bathroom in a position where it can detect a person entering through the door 71. Figure 10 shows an example in which the human detection / identification box 68 is installed on the far side of the bathtub 70 in the bathroom as seen from the door 71. The control box 61 can also be installed inside the bathroom, but it can also be installed outside the bathroom.

[0125] In this embodiment, too, in a nursing facility or a home, monitoring of a person requiring care can be performed by an information processing device that corresponds to a so-called edge of a communication network. In this embodiment, when the information processing device is installed in a bathroom where entry detection and face recognition are desired, it is possible to detect entry by the human presence sensor 24 and perform face recognition processing based on image data acquired by the second camera 25.

[0126] Although a bathroom has been used as an example, the human detection / identification box 68 can also be installed in rooms other than toilets and bathrooms, such as facility entrances, bedrooms, and dining rooms, to monitor care recipients. In addition to nursing homes, the system can also be used in hospitals, food service companies, kindergartens, schools, and other facilities where many people wear masks and there are concerns about leaking information over networks. For example, installing a facial recognition camera in a dining room can identify visitors to the dining room through facial recognition processing, allowing meals to be prepared tailored to the user. Alternatively, installing a facial recognition camera at the entrance to a facility can prevent a person to be authenticated from leaving the facility alone by notifying others or sounding an alarm on the person's device. In any of these installation examples, the identification of facility users can be linked to records of other tasks, such as caregiving.

[0127] <Other embodiments> [a] In each embodiment, the functions of the information processing system and each device included in the system have been described, but each device is not limited to the configuration example shown in the figure, and it is sufficient for each device to be able to realize these functions, and additional functions may also be provided.

[0128] [b] Each of the devices described in the first to third embodiments may have the following hardware configuration. Fig. 11 is a diagram showing an example of the hardware configuration of the device. The same applies to the other embodiment [a] above.

[0129] The device 100 shown in FIG. 11 may have a processor 101, a memory 102, and a communication interface (I / F) 103. The processor 101 may be, for example, a microprocessor, an MPU (Micro Processor Unit), or a CPU. The processor 101 may include multiple processors. The memory 102 is configured, for example, by a combination of a volatile memory and a non-volatile memory. The functions of each device described in embodiments 1 to 3 are realized by the processor 101 reading and executing a program stored in the memory 102. In this case, information can be sent and received with other devices via the communication interface 103 or an input / output interface (not shown).

[0130] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0131] The present disclosure is not limited to the above-described embodiments, and may be modified as appropriate without departing from the spirit and scope of the present disclosure. In addition, the present disclosure may be implemented by appropriately combining the respective embodiments.

[0132] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0133] (Appendix 1) A sensor that detects a person entering the room; an imaging device that, when the sensor detects a person entering the room, captures an image of the person entering the room and obtains image data; a determination unit that determines whether or not a person who has entered the room is wearing a mask based on the image data obtained by the imaging device, and thereby provisionally determines whether or not the person who has entered the room is a person to be authenticated as a user of the room; a face authentication unit that executes face authentication processing based on face image data of the person to be authenticated in the image data when the result of the provisional determination by the determination unit indicates that the person is the person to be authenticated; an output unit that outputs a result of the face authentication processing to a notification destination; Equipped with the face authentication unit performs the face authentication process by comparing the face image data with feature data indicating facial features stored in advance for each of the authentication subjects and each of the attendants who may accompany the authentication subjects and enter the room; Information processing device. (Appendix 2) When the attendant is authenticated as a result of the face authentication processing, the output unit outputs the result of the face authentication processing excluding the result of the face authentication processing for the attendant. 2. The information processing device according to claim 1. (Appendix 3) The feature data is data generated from face image data acquired by a terminal device and stored in the information processing device. 3. The information processing device according to claim 1 or 2. (Appendix 4) the room is a toilet, and the imaging device is installed inside the toilet; the information processing device includes an analysis unit that analyzes excretion information indicating the content of excretion based on image data captured by another image capturing device that is installed so as to include an excretion area in the toilet bowl in an image capturing range, the analysis unit performs an analysis of the excretion information when the person to be authenticated is successfully authenticated as a result of the face authentication process; The output unit outputs the analysis result of the analysis unit to a notification destination. 4. The information processing device according to any one of claims 1 to 3. (Appendix 5) The sensor detects when someone enters the room, When the sensor detects a person entering the room, an imaging device captures an image of the person entering the room to obtain image data; determining whether the person who has entered the room is wearing a mask based on the image data obtained by the imaging device, and provisionally determining whether the person who has entered the room is a person to be authenticated as a user of the room or not; If the result of the provisional determination is that the person is the person to be authenticated, a facial authentication process is performed by comparing the facial image data of the person to be authenticated in the image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may enter the room accompanying the person to be authenticated, outputting the result of the face authentication processing to a notification destination; Information processing methods. (Appendix 6) When the attendant is authenticated as a result of the face authentication processing, the output to the notification destination is executed excluding the result of the face authentication processing for the attendant. 1. The information processing method described in Appendix 5. (Appendix 7) The feature data is data generated from face image data acquired by a terminal device and stored. 7. An information processing method according to claim 5 or 6. (Appendix 8) the room is a toilet, and the imaging device is installed inside the toilet; If the person to be authenticated is successfully authenticated as a result of the face authentication process, an analysis process is performed to analyze excretion information indicating the content of the excretion based on image data captured by another image capture device installed so as to include in its image capture range the excretion range of the toilet bowl of the toilet, outputting the analysis results of the analysis processing to a notification destination; An information processing method according to any one of Supplementary Notes 5 to 7. (Appendix 9) On the computer, The sensor detects when someone enters the room and inputs the results. When the sensor detects a person entering the room, the image capturing device captures an image of the person entering the room to obtain image data; determining whether the person who has entered the room is wearing a mask based on the image data obtained by the imaging device, and provisionally determining whether the person who has entered the room is a person to be authenticated as a user of the room or not; If the result of the provisional determination is that the person is the person to be authenticated, a facial authentication process is performed by comparing the facial image data of the person to be authenticated in the image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may enter the room accompanying the person to be authenticated, outputting the result of the face authentication processing to a notification destination; A program for performing information processing. (Appendix 10) When the attendant is authenticated as a result of the face authentication processing, the output to the notification destination is executed excluding the result of the face authentication processing for the attendant. 10. The program described in Appendix 9. (Appendix 11) The feature data is generated from face image data acquired by a terminal device and stored in the computer. 11. The program according to claim 9 or 10. (Appendix 12) the room is a toilet, and the imaging device is installed inside the toilet; The information processing If the person to be authenticated is successfully authenticated as a result of the face authentication process, an analysis process is performed to analyze excretion information indicating the content of the excretion based on image data captured by another image capture device installed so as to include in its image capture range the excretion range of the toilet bowl of the toilet, outputting the analysis results of the analysis processing to a notification destination; A program according to any one of appendices 9 to 11. [Explanation of symbols]

[0134] 1, 60 Information processing device 1a Sensor 1b Imaging device 1c Judgment part 1d Face recognition unit 1e Output section 10 Toilet 10a Side 10b Plane part 10c edge 11 Toilet seat 11a Toilet seat body 11b Toilet seat cover 24 Human Sensor 25 Imaging device (second camera) 26 Information Gathering Department 26a Distance sensor 26b First Camera 27a CPU 27b connector 27c, 27d USB I / F 27e WiFi module 40 servers 41 Control Unit 42 Storage section 50 Terminal Equipment 51 Display screen 52, 53, 54 Notification Information 61 Storage enclosure (control box) 62 Bridge section 63 Inner housing (sensor box) 67 Control Unit 68 Separate housing (human detection / identification box) 69 Outer housing 70 Bathtub 71 Door 100 devices 101 processors 102 memory 103 Communication Interface

Claims

1. A sensor that detects a person entering the room; an imaging device that, when the sensor detects the entry of a plurality of people into the room, captures images of the plurality of people who have entered the room and obtains image data; a determination unit that determines whether each of a plurality of people who have entered the room is wearing a mask based on the image data obtained by the imaging device, and thereby provisionally determines whether the person who has entered the room is a person to be authenticated as a user of the room; a face authentication unit that executes face authentication processing based on face image data of the person to be authenticated in the image data only when the result of the provisional determination by the determination unit indicates that the person is the person to be authenticated; an output unit that outputs a result of the face authentication processing to a notification destination; Equipped with the face authentication unit terminates the process without performing the face authentication process on the person wearing the mask, and performs the face authentication process only on the person not wearing a mask by comparing the face image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may enter the room accompanying the person to be authenticated; Information processing device.

2. When the attendant is authenticated as a result of the face authentication processing, the output unit outputs the result of the face authentication processing excluding the result of the face authentication processing for the attendant. The information processing device according to claim 1 .

3. The feature data is data generated from face image data acquired by a terminal device and stored in the information processing device.

3. The information processing device according to claim 1 or 2.

4. the room is a toilet, and the imaging device is installed inside the toilet; the information processing device includes an analysis unit that analyzes excretion information indicating the content of excretion based on image data captured by another image capturing device that is installed so as to include an excretion area in the toilet bowl in an image capturing range, the analysis unit performs an analysis of the excretion information when the person to be authenticated is successfully authenticated as a result of the face authentication process; The output unit outputs the analysis result of the analysis unit to a notification destination. The information processing device according to any one of claims 1 to 3.

5. An information processing method executed by a computer, comprising: The computer The sensor detects when someone enters the room, When the sensor detects the entry of a plurality of people into the room, an imaging device captures images of the plurality of people who have entered the room to obtain image data; determining whether each of the people who have entered the room is wearing a mask based on the image data obtained by the imaging device, and provisionally determining whether the people who have entered the room are persons to be authenticated as users of the room; As a result of the provisional determination, the process is terminated without performing face authentication processing on the person wearing the mask, and only when the person not wearing a mask is the person to be authenticated, the process is performed by comparing the face image data of the person to be authenticated in the image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may accompany the person to be authenticated and enter the room, based on the face image data of the person to be authenticated in the image data, outputting the result of the face authentication processing to a notification destination; Information processing methods.

6. The computer When the attendant is authenticated as a result of the face authentication processing, the output to the notification destination is executed excluding the result of the face authentication processing for the attendant. The information processing method according to claim 5 .

7. the room is a toilet, and the imaging device is installed inside the toilet; The computer If the person to be authenticated is successfully authenticated as a result of the face authentication process, an analysis process is performed to analyze excretion information indicating the content of the excretion based on image data captured by another image capture device installed so as to include in its image capture range the excretion range of the toilet bowl of the toilet, outputting the analysis results of the analysis processing to a notification destination; 7. The information processing method according to claim 5 or 6.

8. On the computer, The sensor detects when someone enters the room and inputs the results. When the sensor detects the entry of a plurality of people into the room, an imaging device captures images of the plurality of people who have entered the room to obtain image data; determining whether each of the people who have entered the room is wearing a mask based on the image data obtained by the imaging device, and provisionally determining whether the people who have entered the room are persons to be authenticated as users of the room; As a result of the provisional determination, the process is terminated without performing face authentication processing on the person wearing the mask, and only when the person not wearing a mask is the person to be authenticated, the process is performed by comparing the face image data of the person to be authenticated in the image data with feature data indicating facial features stored in advance for each person to be authenticated and for each attendant who may accompany the person to be authenticated and enter the room, based on the face image data of the person to be authenticated in the image data, outputting the result of the face authentication processing to a notification destination; A program for performing information processing.

9. When the attendant is authenticated as a result of the face authentication processing, the output to the notification destination is executed excluding the result of the face authentication processing for the attendant. The program according to claim 8.

10. the room is a toilet, and the imaging device is installed inside the toilet; The information processing If the person to be authenticated is successfully authenticated as a result of the face authentication process, an analysis process is performed to analyze excretion information indicating the content of the excretion based on image data captured by another image capture device installed so as to include in its image capture range the excretion range of the toilet bowl of the toilet, outputting the analysis results of the analysis processing to a notification destination; The program according to claim 8 or 9.

Images