Information processing device, information processing method, and recording medium

Abstract

This information processing device comprises: an acquisition unit that acquires face image information including at least one of luminance information related to the luminance of a face image obtained by imaging the face of a subject in a predetermined space, covering information related to whether or not a facial area in the face image is covered, and position information related to the position of the facial area in the face image; and a storage unit that, with regard to a face image captured from when the subject enters the predetermined space until the subject leaves the predetermined space, attaches the same identification information to the face image captured and to the face image information corresponding to the face image captured for storage. The information processing device collects information useful for understanding the situation when the face image was captured.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to the technical fields of an information processing device, an information processing method, and a recording medium.

[0002] Patent Document 1 describes a technique for confirming the correctness of camera operation by leaving text information without saving the image.

[0003] JP 2011-077811 A

[0004] An object of the present disclosure is to provide an information processing device, an information processing method, and a recording medium that aim to improve upon the techniques disclosed in prior art documents.

[0005] One aspect of the information processing device includes an acquisition means for acquiring facial image information including at least one of brightness information regarding the brightness of a facial image captured of the face of a subject in a specified space, occlusion information regarding whether a facial area in the facial image is occluded, and position information regarding the position of the facial area in the facial image, and a storage means for storing the facial image information corresponding to facial images captured from the time the subject enters the specified space until the time the subject leaves the space, with the same identification information attached.

[0006] One aspect of the information processing method acquires facial image information including at least one of brightness information regarding the brightness of a facial image captured of the face of a subject in a specified space, occlusion information regarding whether a facial area in the facial image is occluded, and position information regarding the position of the facial area in the facial image, and stores the facial image information corresponding to facial images captured from the time the subject enters the specified space until the time the subject leaves the space, with the same identification information attached.

[0007] One aspect of the recording medium has recorded thereon a computer program for causing a computer to execute an information processing method for acquiring facial image information including at least one of brightness information regarding the brightness of a facial image captured of the face of a subject in a specified space, occlusion information regarding whether a facial area in the facial image is occluded, and position information regarding the position of the facial area in the facial image, and storing the facial image information corresponding to facial images captured from the time the subject enters the specified space until the time the subject leaves the space by attaching the same identification information to the facial image information.

[0008] FIG. 1 is a block diagram showing an example of a configuration of an information processing device according to the present disclosure. FIG. 2 is a conceptual diagram showing an example of a scene to which an information processing device according to the present disclosure is applied. FIG. 3 is a block diagram showing an example of a configuration of an information processing device according to the present disclosure. FIG. 4 is a flowchart showing an example of an information processing operation of an information processing device according to the present disclosure. FIG. 5 is a table showing an example of face image information according to the present disclosure. FIG. 6 is a flowchart showing an example of an inference operation of an information processing device according to the present disclosure. FIG. 7 is a block diagram showing an example of a configuration of an information processing device according to the present disclosure. FIG. 8 is a table showing an example of summary information according to the present disclosure. FIG. 9 is a block diagram showing an example of a configuration of an information processing device according to the present disclosure. FIG. 10 is a conceptual diagram showing an example of a visualization operation of an information processing device according to the present disclosure.

[0009] Hereinafter, embodiments of an information processing device, an information processing method, and a recording medium will be described with reference to the drawings. [1: First Embodiment]

[0010] A first embodiment of an information processing device, an information processing method, and a recording medium will be described. Hereinafter, the first embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 1 according to the present disclosure. [1-1: Configuration of Information Processing Device 1]

[0011] 1 is a block diagram showing the configuration of an information processing device 1 according to the present disclosure. As shown in FIG. 1, the information processing device 1 includes an acquisition unit 11 and a storage unit 12.

[0012] The acquisition unit 11 acquires face image information including at least one of brightness information, occlusion information, and position information. The brightness information is information about the brightness of a face image obtained by capturing the face of a target in a predetermined space. The occlusion information is information about whether a face area in the face image is occluded. The position information is information about the position of the face area in the face image.

[0013] The storage unit 12 stores face image information corresponding to face images captured from the time the target enters the predetermined space until the time the target leaves the predetermined space, with the same identification information attached to the face image information. For example, the storage unit 12 may store face image information corresponding to face images captured from the time the target starts to be detected in the predetermined space until the detection is completed, with the same identification information attached to the face image information. In other words, the storage unit 12 stores face image information corresponding to face images captured while the target is using the predetermined space, with the same identification information attached to the face image information. In other words, the storage unit 12 stores face image information attached with the same identification information for each unit of use of the predetermined space. [1-2: Technical Effects of the Information Processing Device 1]

[0014] The information processing device 1 according to the present disclosure acquires facial image information related to a facial image and stores the facial image information with the same identification information for each utilization unit of a predetermined space, thereby making it possible to collect information useful for understanding the situation when the facial image was captured. Furthermore, since the information processing device 1 does not store the facial image itself but stores facial image information related to the facial image, there is no risk of violating the privacy of the subject. [2: Second embodiment]

[0015] A second embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the second embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 2 according to the present disclosure. [2-1: Example of a situation in which the information processing device 2 is applied]

[0016] FIG. 2 illustrates an example of a scene to which the information processing device 2 is applied. In this embodiment, the predetermined space is a private space 100. FIG. 2 illustrates a toilet as an example of the private space 100. The private space 100 is provided with an entrance 101, a toilet 102, and a camera C. The entrance 101 may be provided so as to be openable and closable so that the subject T can enter and exit the private space 100. The camera C may be provided so as to capture an image of the subject T's face when the subject T enters and exits the private space 100. The camera C may be controlled to capture an image in the imaging range R when the subject T's entry into the private space 100 is detected. The camera C may be installed so as to capture an image of a situation in which the subject T does not mind being seen by others. For example, the camera C may be installed so as not to capture an image of the subject T when the subject T is using the toilet 102. In this embodiment, the information processing device 2 performs facial recognition of the subject T in the private space 100 as a process using a facial image. Note that in the present disclosure, an "image" may be a still image or a video. [2-2: Configuration of information processing device 2]

[0017] 3 is a block diagram showing the configuration of the information processing device 2. As shown in FIG. 3, the information processing device 2 includes a calculation device 21 and a storage device 22. The information processing device 2 may further include a communication device 23, an input device 24, and an output device 25. However, the information processing device 2 does not necessarily have to include at least one of the communication device 23, the input device 24, and the output device 25. The calculation device 21, the storage device 22, the communication device 23, the input device 24, and the output device 25 may be connected via a data bus 26.

[0018] The arithmetic device 21 includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an FPGA (Field Programmable Gate Array). The arithmetic device 21 reads a computer program. For example, the arithmetic device 21 may read a computer program stored in the storage device 22. For example, the arithmetic device 21 may read a computer program stored in a computer-readable, non-transitory recording medium using a recording medium reading device (e.g., an input device 24 described later) not shown in the drawings that is included in the information processing device 2. The arithmetic device 21 may acquire (i.e., download or read) the computer program from a device (not shown) located outside the information processing device 2 via the communication device 23 (or another communication device). The arithmetic device 21 executes the read computer program. As a result, logical functional blocks for executing operations to be performed by the information processing device 2 are realized within the arithmetic device 21. In other words, the arithmetic device 21 can function as a controller for realizing logical functional blocks for executing operations (in other words, processes) to be performed by the information processing device 2. The arithmetic device 21 may output information to devices (not shown), such as other computers or cloud servers, that are provided outside the information processing device 2, via the communication device 23 (or other communication devices).

[0019] 3 shows an example of logical functional blocks implemented within the arithmetic device 21 to execute information processing operations. As shown in FIG. 3 , the arithmetic device 21 includes a facial image information acquisition unit 211, which is a specific example of an "acquisition unit" described in the appendix, a memory unit 212, which is a specific example of a "storage unit" described in the appendix, a facial image acquisition unit 213, a face recognition unit 214, which is a specific example of a "face recognition unit" described in the appendix, an estimation unit 215, which is a specific example of a "estimation unit" described in the appendix, and an output unit 216, which is a specific example of an "output unit" described in the appendix. However, at least one of the facial image acquisition unit 213, the face recognition unit 214, the estimation unit 215, and the output unit 216 does not have to be implemented within the arithmetic device 21. The operations of the facial image information acquisition unit 211, the storage unit 212, the facial image acquisition unit 213, the facial recognition unit 214, the estimation unit 215, and the output unit 216 will be described in detail later with reference to FIGS.

[0020] The storage device 22 can store desired data. For example, the storage device 22 may temporarily store a computer program executed by the arithmetic device 21. The storage device 22 may temporarily store data that the arithmetic device 21 temporarily uses when the arithmetic device 21 is executing a computer program. The storage device 22 may store data that the information processing device 2 stores long-term. The storage device 22 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 22 may include a non-temporary recording medium.

[0021] The communication device 23 is capable of communicating with devices external to the information processing device 2 via a communication network (not shown). The communication device 23 may be a communication interface based on standards such as Ethernet (registered trademark), Wi-Fi (registered trademark), Bluetooth (registered trademark), or USB (Universal Serial Bus).

[0022] The input device 24 is a device that accepts information input to the information processing device 2 from outside the information processing device 2. For example, the input device 24 may include an operation device (e.g., at least one of a keyboard, a mouse, and a touch panel) that can be operated by an operator of the information processing device 2. For example, the input device 24 may include a reading device that can read information recorded as data on a recording medium that can be externally attached to the information processing device 2.

[0023] The output device 25 is a device that outputs information to the outside of the information processing device 2. For example, the output device 25 may output information as an image. That is, the output device 25 may include a display device (a so-called display) that can display an image showing the information to be output. For example, the output device 25 may output information as sound. That is, the output device 25 may include an audio device (a so-called speaker) that can output sound. For example, the output device 25 may output information on paper. That is, the output device 25 may include a printing device (a so-called printer) that can print desired information on paper. [2-3: Information Processing Operation Performed by the Information Processing Device 2]

[0024] The information processing operation performed by the information processing device 2 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of the flow of the information processing operation performed by the information processing device 2.

[0025] 4, the face image acquisition unit 213 determines whether or not the target T has entered the private space 100 (step S20). The face image acquisition unit 213 may determine whether or not the target T has entered based on, for example, whether the entrance 101 is open or closed.

[0026] The face image acquisition unit 213 acquires a face image of the face of the target T captured by the camera C (step S21). The face image acquisition unit 213 may control the camera C to capture an image of the target T who has entered the private space 100. The face image acquisition unit 213 may acquire a still image captured by the camera C, or may acquire one frame of a video.

[0027] The face recognition unit 214 performs face recognition using the face image acquired by the face image acquisition unit 213 (step S22). The face recognition unit 214 may detect a face area from the face image acquired by the face image acquisition unit 213 and compare information about the detected face area with registered information to perform face recognition. The face recognition unit 214 may extract feature points such as the eyes, nose, and mouth from the detected face and compare information about the extracted feature points with registered information to perform face recognition. In this embodiment, the face recognition operation may be an operation equivalent to a face matching operation.

[0028] The facial image information acquisition unit 211 acquires facial image information from the facial image acquired by the facial image acquisition unit 213 (step S23). The facial image information includes at least one of brightness information, occlusion information, and position information. Each of the brightness information, occlusion information, and position information may be referred to as element information.

[0029] The luminance information is information about the luminance of the face image. The luminance information may include information about the luminance of the entire face image. In other words, the luminance information may include information representing the lighting state of the private space. The luminance information may be information indicating a defect in the lighting itself. Furthermore, the luminance information may include information about the luminance of the face region in the face image. The luminance information may include information representing the luminance of the face image and information representing the luminance of the face region of the face image. The luminance information may include information representing how the light hits the face. The luminance information may be information indicating the positional relationship between the light and the face. The luminance information may be information indicating that the light hitting the face is backlit.

[0030] The occlusion information is information relating to whether or not a face region in a face image is occluded. The face region may be occluded by, for example, a mask or sunglasses.

[0031] The position information is information relating to the position of a face region in a face image. The position information may include information indicating the position of the target's face. The position of the face may be, for example, the position of the upper left corner of the face region in the face image. The position information may also include information indicating the positions of parts included in the face. Examples of parts included in the face include the left eye, right eye, nose, and mouth. The positions of the left eye and right eye may be the positions of the centers of the pupils. The position of the nose may be the position of the tip of the nose. The position of the mouth may be the positions of the left corner of the mouth and the right corner of the mouth.

[0032] The storage unit 212 stores the facial image information by assigning a management ID to it (step S24). The management ID is identification information for identifying facial image information for each usage unit of the private space 100. Therefore, the storage unit 212 stores the facial image information corresponding to facial images captured from when the subject T entered the private space 100 until when he left the private space 100 by assigning the same management ID to the facial image information. A facial image information table may be implemented in the storage unit 22. In this case, the storage unit 212 may store a record of facial image information including the management ID in the facial image information table in the storage unit 22.

[0033] The facial image information acquisition unit 211 acquires facial image information in the form of text data. The storage unit 212 stores the facial image information in the text data. FIG. 5 illustrates an example of a facial image information table. As illustrated in FIG. 5, the storage unit 212 may save the facial image information in a comma separated value (CSV) format. FIG. 5 illustrates an example in which each record stored in the facial image information table includes a management ID, an image capture time, a user ID identified by facial recognition, the reliability of the facial recognition result, and multiple pieces of element information. FIG. 5 illustrates an example in which the facial image information includes, as element information, the brightness of the image, the brightness of the face, the X coordinate of the face, the Y coordinate of the face, the width of the face, the height of the face, whether a mask is used, the X coordinate of the left eye, the X coordinate of the right eye, the X coordinate of the nose, the X coordinate of the left side of the mouth, the X coordinate of the right side of the mouth, the Y coordinate of the left eye, the Y coordinate of the right eye, the Y coordinate of the nose, the Y coordinate of the left side of the mouth, and the Y coordinate of the right side of the mouth. 5, one face image may correspond to one record of face image information in the face image information table. The storage unit 212 assigns a different management ID for each use of the private space 100, so that information for each use can be distinguished.

[0034] The face image acquisition unit 213 determines whether the target T has left the private space 100 (step S25). If the target T has not left the private space 100 (step S25: No), the process may return to step S21.

[0035] If the target T has left the private space 100 (step S25: Yes), the estimation unit 215 determines whether or not the face recognition of the target T has been successful (step S26).

[0036] If the facial recognition of the target T fails (step S26: No), the estimation unit 215 estimates the cause of the failure based on the facial image information assigned the same management ID (step S27). The causes of the failure of the facial recognition are often "I: Insufficient illumination of the space," "II: Possibility of the distance between the camera C and the target T being too far," "III: Possibility of the target T being wearing a mask," "IV: Possibility of the image being captured in backlight," "V: Possibility of the face being out of the angle of view," and "VI: Possibility of the image being captured with the face not facing forward." Therefore, it may be determined which of these causes applies in order. The cause of the failure of the facial recognition may be multiple causes.

[0037] 6 is a flowchart showing an example of the flow of the operation for analyzing the cause of failure. As shown in FIG. 6, the estimation unit 215 determines the brightness of the entire face image (step S30). If the entire face image is dark, the estimation unit 215 may estimate that the cause of the face recognition failure (cause I) is insufficient illuminance in the private space 100. In a small space such as a toilet, illuminance may be insufficient.

[0038] The estimation unit 215 determines the size of the face in the face image (step S31). When performing face recognition, if the face image is small, the feature amount may be insufficient. If face matching is performed with an insufficient feature amount, there is a risk of false detection, so it may be desirable to exclude a small face from the target of face recognition. If the face in the face image is small, the estimation unit 215 may estimate that the reason for the failure of face recognition (Cause II) is that the target T is far from the camera C and the information necessary for face matching is insufficient.

[0039] Furthermore, when the target T is receiving assistance from another person, it may be desirable to exclude this person from the target of face recognition. In such a case, the target of face recognition may be limited based on the size of the face. Therefore, limiting the target based on the size of the face may cause the face recognition to fail.

[0040] The estimation unit 215 determines whether or not the target T is wearing a mask (step S32). If the target T is wearing a mask, the estimation unit 215 may estimate that the reason for the failure of the face recognition is that the target T is not a target for face recognition (Cause III).

[0041] Furthermore, when the subject T is receiving assistance from another person, it may be desirable to exclude this person from the target of facial recognition. Generally, in nursing care or medical settings, employees often wear masks. In such cases, the target of facial recognition may be limited based on whether or not the person is wearing a mask. For example, a person wearing a mask may be presumed to be an employee, and the person wearing a mask may be excluded from the target of facial recognition. For this reason, limiting the target based on whether or not the person is wearing a mask may cause facial recognition to fail.

[0042] The estimation unit 215 determines the brightness of the face area of ​​the face image (step S33). If the face area is dark, the estimation unit 215 may estimate that the cause of the face recognition failure (Cause IV) is that the image was captured in a backlit state. Backlighting may occur depending on the position of the lighting.

[0043] The estimation unit 215 determines the coordinates of the face (step S34). If the coordinates of the face are far from the center of the face image, the estimation unit 215 may estimate that the cause of the failure in face recognition (cause V) is that at least a portion of the face area is outside the angle of view. The estimation unit 215 may estimate that the cause of the failure in face recognition is that a portion of the face area necessary for face recognition is outside the angle of view. In private spaces 100 such as restrooms, the angle of view is often limited to the upper body, centered on the face, in consideration of psychological stress. For this reason, if subject T uses a wheelchair, his or her face may be outside the angle of view of the camera.

[0044] The estimation unit 215 determines the orientation of the face in the facial image (step S35). The estimation unit 215 may determine the orientation of the face in the facial image based on the positional relationship of each part included in the face. When performing facial recognition on a moving target T, the angle at which the face is captured may become poor. If the face is significantly tilted, the estimation unit 215 may estimate that the reason for the failure of facial recognition (Cause VI) is that the face is not facing forward.

[0045] The output unit 216 outputs the result of the face recognition (step S28). If the face recognition has failed, the output unit 216 also outputs the cause of the face recognition failure estimated in step S27. The output unit 216 may output at least one of the result of the face recognition and the cause of the face recognition failure to a device (not shown), such as another computer or cloud server, provided outside the information processing device 2, via the communication device 23 (or another communication device). The output unit 216 may control the output device 25 to output at least one of the result of the face recognition and the cause of the face recognition failure. [2-4: Modification]

[0046] The facial image information can be used not only to estimate the cause of failure in processing using facial images for each usage unit of the private space 100, but also for more advanced functions. For example, the information processing device 2 may have a mechanism for deriving an appropriate installation position and angle of the camera C from the facial image information, and may output information regarding changes to the installation position, angle, etc. Furthermore, the information processing device 2 may autonomously change the installation position, angle, etc. of the camera C. Furthermore, the information processing device 2 may be configured to have a mechanism for automatically adjusting the brightness of lighting equipment and the amount of external light taken in, based on the facial image information, depending on the time of day of imaging, etc. The information processing device 2 may also suppress a decrease in the accuracy of facial recognition due to changes in the environment. [2-5: Technical Effects of the Information Processing Device 2]

[0047] In a system that performs processing using facial images captured by a camera, if the processing fails, it is easy to imagine using the captured facial image itself to infer the cause of the failure. However, because facial images are important personal information, it is desirable to operate the system without retaining the facial images themselves in order to protect the privacy of those appearing in the facial images. In particular, when using facial images captured in private spaces, it is desirable to operate the system without retaining the facial images themselves. There is also a demand for processing using facial images, such as facial recognition, to be performed on edge devices with limited computing resources. In this case, too, it is desirable to operate the system without storing the facial images themselves. However, in order to understand the situation at the site where the image was captured without retaining the facial images, a mechanism for understanding the situation is required.

[0048] Meanwhile, causes of failure in processing using face images such as face recognition are often one of the following: "I: Insufficient illumination in the space," "II: Possibility that the distance between camera C and target T is too far," "III: Possibility that target T is wearing a mask," "IV: Possibility that the image was captured against the light," "V: Possibility that the face is out of the angle of view," and "VI: Possibility that the image was captured with the face not facing forward." Therefore, the information processing device 2 acquires and manages information that can identify these main causes, thereby inferring the cause.

[0049] The information processing device 2 according to the present disclosure manages facial image information, which is text data, for each unit of use of the private space 100, and can infer the cause of failure when facial recognition fails based on the facial image information. The information processing device 2 does not store images for analysis, so the facial recognition system can be operated without violating privacy. When facial recognition is not possible, the cause can be inferred from the facial image information, so measures can be taken immediately when an inquiry is made by a user. Since the information processing device 2 stores facial image information, which is text data, it can alleviate the pressure on storage capacity within the device. Furthermore, even in devices with low processing capabilities such as edge devices, it can infer the cause of failure. Furthermore, even when information is stored on a remote server, it can alleviate communication costs and insufficient storage capacity. [3: Third Embodiment]

[0050] A third embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the third embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 3 according to the present disclosure. [3-1: Configuration of Information Processing Device 3]

[0051] An example of the configuration of the information processing device 3 will be described with reference to Fig. 7. Fig. 7 is a block diagram showing an example of the configuration of the information processing device 3.

[0052] As shown in FIG. 7, the information processing device 3 includes a calculation device 21 and a storage device 22, similar to the information processing device 2. Furthermore, the information processing device 3 may include a communication device 23, an input device 24, and an output device 25, similar to the information processing device 2. However, the information processing device 3 does not have to include at least one of the communication device 23, the input device 24, and the output device 25. The information processing device 3 differs from the information processing device 2 in that a generation unit 317 is implemented in the calculation device 21. Other features of the information processing device 3 may be the same as other features of the information processing device 2. Therefore, hereinafter, differences from the embodiments already described will be described in detail, and descriptions of other overlapping parts will be omitted as appropriate. [3-2: Information Processing Operation Performed by Information Processing Device 3]

[0053] The generation unit 317 generates summary information that summarizes the situation from when the target T enters the private space 100 until when he / she leaves the private space 100, based on the facial image information assigned the same management ID. In other words, the generation unit 317 processes the facial image information into summary information. The generation unit 317 may generate summary information that is useful for estimating the cause of failure when processing using a facial image fails, based on the facial image information.

[0054] A summary information table may be implemented in the storage device 22. In this case, the generation unit 317 may store the summary information generated for each usage unit in the summary information table in the storage device 22. That is, the summary information table contains one record corresponding to the management ID.

[0055] 8 illustrates an example of a summary information table. As illustrated in FIG. 8, the summary information table may include a management ID for each usage unit, a time, an authentication result (user ID), a confidence level, image brightness (information regarding cause I), face size (information regarding cause I), whether a mask was worn (information regarding cause III), face brightness (information regarding cause IV), face position (left / right and up / down) (information regarding cause V), face orientation (left / right and up / down) (information regarding cause VI), and the number of captured images. By referring to the summary information, the administrator can infer the cause of the face recognition failure without checking the captured images or videos themselves.

[0056] For example, if the "image brightness" is "dark," it can be inferred that cause I is the cause of the face recognition failure. If the "face size" is "small," it can be inferred that cause II is the cause of the face recognition failure. If the "mask presence" is "present," it can be inferred that cause III is the cause of the face recognition failure. If the "face brightness" is "dark," it can be inferred that cause IV is the cause of the face recognition failure. If the "face position" is not "center," it can be inferred that cause V is the cause of the face recognition failure. If the "face orientation" is not "frontal," it can be inferred that cause VI is the cause of the face recognition failure.

[0057] The information processing device 3 may have a mechanism for directly transmitting summary information generated from facial image information stored in the device to system users. By providing such a function, the number of maintenance steps for the system can be reduced. [3-3: Technical Effects of the Information Processing Device 3]

[0058] The information processing device 3 according to the present disclosure generates summary information summarizing the situation from when the target T enters the private space 100 until when he / she leaves the private space 100 based on the facial image information assigned the same management ID, which is useful for understanding the situation in the private space 100 for each unit of use. [4: Fourth embodiment]

[0059] A fourth embodiment of an information processing device, an information processing method, and a recording medium will be described. Hereinafter, the fourth embodiment of an information processing device, an information processing method, and a recording medium will be described using an information processing device 4 according to the present disclosure. [4-1: Configuration of Information Processing Device 4]

[0060] An example of the configuration of the information processing device 4 will be described with reference to Fig. 9. Fig. 9 is a block diagram showing an example of the configuration of the information processing device 4.

[0061] 9, the information processing device 4 includes a calculation device 21 and a storage device 22, similar to the information processing device 2 and the information processing device 3. Furthermore, the information processing device 4 may include a communication device 23, an input device 24, and an output device 25, similar to the information processing device 2 and the information processing device 3. However, the information processing device 3 does not need to include at least one of the communication device 23, the input device 24, and the output device 25. The information processing device 4 differs from the information processing device 2 and the information processing device 3 in that a visualization unit 418 is implemented within the calculation device 21. Other features of the information processing device 4 may be the same as other features of the information processing device 2. Therefore, hereinafter, differences from the embodiments already described will be described in detail, and descriptions of other overlapping parts will be omitted as appropriate. [4-2: Information Processing Operation Performed by Information Processing Device 4]

[0062] The visualization unit 418 visualizes time-series changes in at least one of the position of the face of the target T and the positions of the facial features based on the facial image information. The visualization unit 418 may draw an image as illustrated in FIG. 10 based on information indicating the position of the face of the target T and information indicating the positions of the facial features included in the facial image information. That is, the visualization unit 418 may draw a facial outline F based on information indicating the X coordinate, Y coordinate, width, and height of the face included in the facial image information. The visualization unit 418 may also draw a left eye LE based on the X coordinate and Y coordinate of the left eye included in the facial image information. The visualization unit 418 may also draw a right eye RE based on the X coordinate and Y coordinate of the right eye included in the facial image information. The visualization unit 418 may also draw a nose N based on the X coordinate and Y coordinate of the nose included in the facial image information. The visualization unit 418 may also draw a mouth M based on the X coordinates and Y coordinates of the left and right sides of the mouth included in the facial image information.

[0063] The visualization unit 418 may draw an image based on the position of the face and the positions of the parts included in the face for each captured face image, and display them side by side. As a result, as illustrated in FIG. 10, the visualization unit 418 can reproduce the time-series changes in the position of the face of the subject T and the positions of the parts included in the face. In other words, by using the information visualized by the visualization unit 418, it is possible to infer the movements of the subject T and the circumstances in which the subject T is using the private space 100 without storing photos or videos. Furthermore, it is possible to analyze the cause of a failure in face recognition from a series of movements. [4-3: Technical Effects of the Information Processing Device 4]

[0064] The information processing device 4 according to the present disclosure visualizes time-series changes in at least one of the position of the face of the target T and the positions of parts included in the face based on facial image information related to the facial image, and is therefore useful for understanding the situation of the target T in the private space 100. Compared to techniques such as skeletal detection and estimation that estimate the movement of a target, the information processing device 4 can easily understand the situation in the private space 100, such as the movement of the target T.

[0065] In nursing care settings, assistance with excretion, for example, places a heavy burden on caregivers. Therefore, support that can reduce the workload is needed. When the information processing device 4 is applied to a nursing care setting, (1) it is possible to identify the care recipient T who uses the private space 100 by facial recognition, and (2) it is possible to grasp the status of the target T within the private space 100, thereby reducing the burden on the caregiver. The output unit 216 may notify the caregiver of the status of the target T within the private space 100.

[0066] In the above embodiment, a case where facial recognition of a target is performed has been described as an example of processing using a facial image. However, processing using a facial image is not limited to identifying an individual using a facial image. For example, emotions, age, etc. can be identified using a facial image. Even when emotions, age, etc. are identified using a facial image, the same identification information may be assigned to each utilization unit of a predetermined space and facial image information may be stored, thereby collecting information useful for understanding the situation when the facial image was captured. Even if identification of emotions, age, etc. using a facial image fails, the cause of the failure may be inferred based on the stored facial image information. [5: Supplementary Note]

[0067] The following supplementary notes are further disclosed with respect to the above-described embodiment. [Supplementary Note 1] An information processing device comprising: an acquisition unit that acquires facial image information including at least one of luminance information regarding the luminance of a facial image captured of the face of a target in a predetermined space, occlusion information regarding whether a facial region in the facial image is occluded, and position information regarding the position of the facial region in the facial image; and a storage unit that stores the facial image information corresponding to facial images captured after the target enters the predetermined space and before exiting the predetermined space, with the same identification information attached. [Supplementary Note 2] The information processing device according to claim 1, further comprising: an estimation unit that estimates a cause of failure when processing using the facial image fails, based on the facial image information to which the same identification information is attached; and an output unit that outputs information indicating the cause of failure. [Supplementary Note 3] The information processing device according to claim 2, further comprising: a face recognition unit that performs face recognition as processing using the facial image, and the output unit outputs at least one of the result of the face recognition and the cause of the failure. [Supplementary Note 4] The information processing device according to claim 1, wherein the facial image information is text data. [Supplementary Note 5] The information processing device according to claim 1, wherein the predetermined space is a private space. [Supplementary Note 6] The information processing device according to claim 1, wherein the position information includes at least one of information indicating the position of the face of the target and information indicating the positions of body parts included in the face. [Supplementary Note 7] The information processing device according to claim 1, further comprising: a generation means for generating summary information summarizing a situation from when the target enters the predetermined space until when the target leaves the predetermined space, based on the face image information to which the same identification information is attached. [Supplementary Note 8] The information processing device according to claim 1, further comprising: a visualization means for visualizing time-series changes in at least one of the position of the face of the target and the positions of body parts included in the face, based on the face image information. [Supplementary Note 9] An information processing method, comprising: acquiring face image information including at least one of brightness information regarding the brightness of a face image captured of the face of a target in a predetermined space, occlusion information regarding whether a face area in the face image is occluded, and position information regarding the position of the face area in the face image; and attaching the same identification information to the face image information corresponding to face images captured from when the target enters the predetermined space until when the target leaves the predetermined space, and storing the same identification information.[Supplementary Note 10] A recording medium having recorded thereon a computer program for causing a computer to execute an information processing method, the method comprising: acquiring facial image information including at least one of brightness information regarding the brightness of a facial image obtained by capturing the face of a target in a predetermined space, occlusion information regarding whether a facial area in the facial image is occluded, and position information regarding the position of the facial area in the facial image; and storing the facial image information corresponding to facial images captured from the time the target enters the predetermined space until the time the target leaves the space, with the same identification information attached thereto.

[0068] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that would be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate. Furthermore, to the extent permitted by law, the disclosures of all documents (e.g., published patent applications) cited in the above disclosure are incorporated by reference as part of the description of this disclosure.

[0069] To the extent permitted by law, this application claims priority to Japanese Patent Application No. 2023-132302, filed August 15, 2023, the disclosure of which is incorporated herein in its entirety by reference.

[0070] 1, 2, 3, 4 Information processing device 11 Acquisition unit 12, 212 Storage unit 211 Facial image information acquisition unit 213 Facial image acquisition unit 214 Face recognition unit 215 Estimation unit 216 Output unit 317 Generation unit 418 Visualization unit

Claims

1. An information processing device comprising: an acquisition means for acquiring facial image information including at least one of luminance information regarding the luminance of a facial image captured of the face of a target in a specified space, occlusion information regarding whether a facial area in the facial image is occluded or not, and position information regarding the position of the facial area in the facial image; and a storage means for storing the facial image information corresponding to facial images captured from the time the target enters the specified space until the time it leaves the space, with the same identification information attached thereto.

2. An information processing device as described in claim 1, comprising: an inference means for inferring a cause of failure when processing using the facial image fails based on the facial image information to which the same identification information is attached; and an output means for outputting information indicating the cause of failure.

3. The information processing device according to claim 2, further comprising a face recognition means for performing face recognition as processing using the face image, wherein the output means outputs at least one of the result of the face recognition and the cause of the failure.

4. The information processing device according to claim 1, wherein the facial image information is text data.

5. The information processing device according to claim 1, wherein the predetermined space is a private space.

6. The information processing device according to claim 1, wherein the position information includes at least one of information indicating the position of the target's face and information indicating the positions of parts included in the face.

7. An information processing device according to claim 1, further comprising a generating means for generating summary information summarizing the situation from when the subject enters the specified space until when the subject exits the specified space, based on the face image information to which the same identification information is attached.

8. The information processing device according to claim 1, further comprising a visualization means for visualizing time series changes in at least one of the position of the target's face and the positions of body parts included in the face, based on the face image information.

9. An information processing method executed by a computer program, which acquires facial image information including at least one of luminance information regarding the luminance of a facial image captured of the face of a subject in a specified space, occlusion information regarding whether a facial area in the facial image is occluded or not, and position information regarding the position of the facial area in the facial image, and stores the facial image information corresponding to facial images captured after the subject enters and leaves the specified space, with the same identification information attached.

10. A recording medium having recorded thereon a computer program for causing a computer to execute an information processing method, which comprises acquiring facial image information including at least one of luminance information regarding the luminance of a facial image captured of the face of a subject in a specified space, occlusion information regarding whether a facial area in the facial image is occluded or not, and position information regarding the position of the facial area in the facial image, and storing the facial image information corresponding to facial images captured after the subject enters the specified space and before exiting it, with the same identification information attached thereto.

Images