Method for acquiring facial image data, information processing device, program, and information processing system

Abstract

A method for acquiring facial image data includes: a first display step of displaying an image including at least a portion of a face of a subject captured by an imaging device on a display device that can be visually recognized by the subject without image obscure; and a second display step in which an image including at least a portion of the face of the subject is displayed on the display device so as to mask the at least a portion of the face of the subject. And a processor configured to determine whether or not a relative position of at least a portion of the face of the subject with respect to the imaging device is appropriate, and to acquire the displayed face image data of the subject when the relative position is appropriate in the determination.

Description

Technical Field

[0001] The present invention relates to a method for obtaining facial image data, an information processing apparatus, a program, and an information processing system. Background Art

[0002] Conventionally, various analysis methods or evaluation methods using a photographing device to photograph the face of a subject and using the photographed facial image data have been known. For example, Patent Document 1 describes a method for evaluating the stress of a subject, the method including: instructing the subject to make a maximum smiling face with the face facing the imaging device, photographing the face of the subject using the imaging device, processing the photographed facial image to obtain a parameter reflecting stress, and using the parameter to calculate a stress index value.

[0003] In the method described in Patent Document 1, when imaging using the imaging device, a monitor (display device) that displays the acquired image in real time is used for alignment. That is, the subject can perform alignment while observing the monitor so that his / her face exists at a given position in the display area of the monitor. At the time of this alignment, the subject sometimes changes the facial expression or the like while looking at the monitor (for example, makes a smiling face although an image of a natural expression needs to be acquired). In this way, appropriate facial image data may sometimes not be obtained, and appropriate evaluation may not be performed. Therefore, in order to prevent the subject from seeing his / her own expression, a scheme of masking at least a part of the face of the subject displayed on the monitor (image masking) has been proposed.

[0004] As described above, by masking at least a part of the face of the subject, the subject cannot see his / her own expression or the like, so that a change in the expression or the like can be prevented. However, due to the image masking, the position of the face itself cannot be grasped, and during the masking period, the position of the face sometimes deviates from the position aligned at the beginning. For example, due to deviation to the right or left with respect to the imaging device, or due to moving away from the imaging device, the position of the face displayed on the monitor sometimes deviates from a given position within the display area, and facial image data cannot be obtained, or even if the facial image data is obtained, the facial image data is not appropriate.

[0005] <Prior Art Documents>

[0006] <Patent Documents>

[0007] Patent Document 1: WO 2019 / 073906 Summary of the Invention

[0008] <Problems to be Solved by the Invention>

[0009] In view of the above, an object of one aspect of the present invention is to obtain appropriate facial image data even when the face is displayed with image masking in a method for obtaining facial image data.

[0010] <Means for Solving the Problem>

[0011] One aspect of the present invention is a method for obtaining facial image data, which includes: a first display step of displaying, on a display device recognizable by the subject, an image including at least a part of the face of the subject captured by a photographing device without image masking; and a second display step of displaying, on the display device, an image including at least a part of the face of the subject with at least a part of the face of the subject masked by an image. In the second display step, it is determined whether the relative position of at least a part of the face of the subject with respect to the photographing device is appropriate. When the relative position is appropriate in the determination, the facial image data of the subject displayed is obtained.

[0012] <Effects of the Invention>

[0013] According to one aspect of the present invention, in a method for obtaining facial image data, appropriate facial image data can be obtained even when the face is displayed with image masking. Description of the Drawings

[0014] Figure 1 It is a flowchart showing the image acquisition method of the present embodiment.

[0015] Figure 2 It is a diagram for explaining the basic steps of image data acquisition.

[0016] Figure 3 It is a diagram showing an example of the overall structure of the present embodiment.

[0017] Figure 4 It is a diagram showing another example of the overall structure of the present embodiment.

[0018] Figure 5 It is a diagram showing still another example of the overall structure of the present embodiment.

[0019] Figure 6 It is a hardware structure diagram of an example of a computer.

[0020] Figure 7 It is a hardware structure diagram of an example of a computer.

[0021] Figure 8 It is a hardware structure diagram of an example of a computer.

[0022] Figure 9It is a functional block diagram of the information processing apparatus according to the present embodiment.

[0023] Figure 10 It is a diagram showing an example of the display in the present embodiment.

[0024] Figure 11 It is a flowchart showing more specific contents of the second display step.

[0025] Figure 12 It is a diagram for explaining a modification example of the display in the process of the present embodiment.

[0026] Figure 13 It is a diagram for explaining a modification example of the display in the process of the present embodiment.

[0027] Figure 14 It is a diagram for explaining a modification example of the display in the process of the present embodiment.

[0028] Figure 15 It is a diagram for explaining a modification example of the display in the process of the present embodiment.

[0029] Figure 16 It is information used in an evaluation method applying the facial image data acquisition method of the present embodiment. Detailed Embodiment

[0030] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In addition, in each drawing, unless otherwise specified, the same or corresponding structures are denoted by the same reference numerals, and the description may be omitted sometimes.

[0031] <Summary>

[0032] Figure 1 It shows a flowchart of the image data acquisition method according to the embodiment of the present invention. As Figure 1 shown, the method for acquiring facial image data includes: a first display step (S10) of displaying, on a display device recognizable by the subject, an image including at least a part of the face of the subject photographed by a photographing device without image masking; and a second display step (S20) of displaying, on the display device, an image including at least a part of the face of the subject with at least a part of the face of the subject masked, and in the second display step, determining whether or not the relative position of at least a part of the face of the subject with respect to the photographing device is appropriate. Further, the method according to the present embodiment includes: a facial image data acquisition step (S30) of acquiring the facial image data of the subject shown in the display when the relative position is appropriate in the determination.

[0033] Figure 2 This is a diagram for explaining the above-described first display step (S10) and second display step (S20). Additionally, Figure 2 the device shown is a computer equipped with a camera and a display, but this form is merely an example for explaining the basic steps.

[0034] As Figure 2 shown in (a) of, in the first display step (S10), the facial image of the subject captured by the camera (imaging device) 250 is displayed on the display (display device) 210. Here, the displayed facial image can be an image including the entire face or an image including only a part of the face. That is, in this specification, the so-called "facial image" refers to an image of the entire face or a part of the face, and the so-called "facial image data" refers to an image of the entire face or a part of the face. In addition, the face can generally be the part above the neck captured by a facing imaging device. For example, it can be set to the range from the hairline of the forehead to the tip of the chin in the vertical direction and to the base of the ear in the horizontal direction. Which range of the face to obtain the image can be determined according to the frontal processing using the facial image data, its use, purpose, etc.

[0035] The above display 210 is a device that the subject can visually observe. In other words, it is configured such that the subject himself / herself can confirm the captured image. Furthermore, the display 210 preferably can display the image of the subject in real time. Thus, in the first display step (S10), the subject can display his / her own face on the display 210 while confirming whether the entire face or a part of the face to be captured is in an appropriate position within the display area, thereby performing face alignment. That is, the relative position with respect to the camera 250 can be made appropriate. As shown in the illustrated example, when the camera 250 is a built-in camera on the same side as the display, the position of the subject's face with respect to the camera 250 is the same as or close to the relative position with respect to the display 210.

[0036] Next, in the second display step (S20), as Figure 2 shown in (b) of, an image masking 215 is applied to the display screen (within the display area). When a person is in a situation where they can observe their own face in real time, even if not necessary, there is a tendency to try to make a good-looking face or a posed face. Therefore, by having the image masking 215, the subject cannot observe their own expression, and unnecessary expression disclosure can be prevented. Thus, for example, even when the required facial image data is facial image data of the subject's natural expression, such desired data can be obtained.

[0037] In this way, by having the image mask 215, it is possible to prevent unnecessary expression disclosure and the like. However, since the face is covered by the image mask 215, it is impossible to confirm or difficult to confirm the position of the face within the display area, and the position of the face may deviate from the position when alignment was performed in the first display step (S10). The longer the time of the second display step (S20) in the state of image masking, the more likely this situation is to occur. In this way, it may be impossible to obtain facial image data, or facial image data may be obtained in a state where alignment was not performed correctly, and subsequent processing may not be appropriately performed. In addition, the time for displaying the face in the state of image masking in the second display step (S20) may be 10 to 60 seconds.

[0038] In response to this, in the present embodiment, when it is determined in the second display step (S20) that the relative position of the face with respect to the camera 250 is appropriate, facial image data is obtained in the facial image data acquisition step (S30) (described in detail later). Thereby, it is possible to prevent the situation of obtaining inappropriate facial image data.

[0039] In addition, in the second display step (S20), information related to the time for obtaining facial image data may also be shown on the display screen of the display 210. As Figure 2 shown in (b) of, for example, the time for image data acquisition may be shown in a countdown manner. Sometimes the subject may be distracted by such time-related information and deviate from the aligned position of the face, but in the method of the present embodiment, it is possible to prevent the position of the face from deviating during the period of image masking, and thus appropriate facial image data can be obtained.

[0040] <System Structure>

[0041] In Figures 3 to 5 the overall structure diagram related to the embodiment is shown. The information processing system related to the present embodiment may be composed of a single information processing device 1 as shown in Figure 3 , or may be composed of one or more subject terminals 2 and a server device 3 connected via a network 5 as shown in Figure 4 . In addition, as shown in Figure 6 , it may also be composed of one or more subject terminals 2, a server device 3, and an examiner terminal 4 connected via a network 5.

[0042] Figure 3 The information processing device 1 shown in determines the propriety of whether the facial image can be obtained as data when displaying the facial image captured by the imaging device in a state of image masking, and can obtain facial image data when appropriate. The information processing device 1 may be a computer such as a personal computer (PC), a smart phone, or a tablet.

[0043] Figure 4 and Figure 5 The subject terminal 2 of the information processing system shown can also be a computer such as a smart phone, a tablet, or a personal computer. In addition, Figure 4 and Figure 5 the server device 3 shown and Figure 5 the examiner terminal 4 shown can be computers such as a personal computer (PC), a smart phone, or a tablet.

[0044] In this way, the information processing system according to this embodiment can be implemented by Figure 3 the single information processing device 1 shown, or can also be implemented by the Figure 4 and Figure 5 client-server mode as shown. In addition, it goes without saying that Figures 3 to 5 the information processing system shown is an example, and there are various system structure examples according to uses and purposes. For example, Figure 4 and Figure 5 the server device 3 can also be configured by being distributed among multiple computers.

[0045] <Hardware Structure>

[0046] The information processing device 1 ( Figure 3 ) and the subject terminal 2 ( Figure 4 and Figure 5 ) can be implemented by computers having the hardware structures shown in Figure 6 and Figure 7 for example. Figure 6 and Figure 7 are hardware structure diagrams of an example of a computer.

[0047] As shown in Figure 6 and Figure 7 , a computer can include an input device 501, a display device 502, an external I / F 503, a RAM 504, a ROM 505, a CPU 506, a communication I / F 507, an HDD 508, a photographing device 509, etc., which are connected to each other via a bus B. In addition, the input device 501, the display device 502, and the photographing device 509 can be of a built-in type or can be used in a manner of being connected as needed. In addition, Figure 6 the computer of Figure 7 has a structure including (built-in) the display device 502 and the photographing device 509, but can also have a structure using the display device 502 and / or the photographing device 509 connected via the external I / F 503. Figure 6 The computer of

[0048] The input device 501 can be a touch panel, operation keys, buttons, keyboard, mouse, etc. used by the subject to input various signals. The display device 502 can be a liquid crystal display, an organic EL display, etc. that display images. The communication I / F 507 is an interface connected to a network 5 such as a local area network (LAN) or the Internet. The computer can communicate via the network 5 using the communication I / F 507. Figure 4 and Figure 5 ).

[0049] The HDD 508 is an example of a non-volatile storage device that stores programs, etc. The programs stored in the HDD 508 include the OS as basic software, applications for acquiring facial image data, and the evaluation or diagnosis of the physical and / or mental state of the subject using the acquired facial image data (such as a stress diagnosis application), etc. In addition, the HDD 508 can also be replaced with a drive device (such as a solid state drive: SSD) or a storage device such as a memory card that uses flash memory as the storage medium. The external I / F 503 is an interface with an external device such as the recording medium 503a. The computer can read and / or write to the recording medium 503a using the external I / F 503.

[0050] Examples of the recording medium 503a include a floppy disk, CD, DVD, SD memory card, USB memory, etc. The ROM 505 is an example of a non-volatile semiconductor memory (storage device) that can retain programs, etc. even when the power is turned off. Programs such as the BIOS executed at startup, OS settings, network settings, etc. are stored in the ROM 505. The RAM 504 is an example of a volatile semiconductor memory (storage device) that temporarily retains programs, etc. The CPU 506 is an arithmetic device that reads programs from storage devices such as the ROM 505 and HDD 508 onto the RAM 504 and executes processing. The imaging device 509 can be a camera (including a 3D camera).

[0051] In the above hardware configuration, the information processing device 1 and the subject terminal 2 of the present embodiment can implement the processing described later by executing a program for acquiring facial image data and a program for additionally evaluating and diagnosing the subject using the facial image data.

[0052] The server device 3 can be implemented, for example, by a computer with the hardware configuration shown in Figure 8 . Figure 8 It is a hardware structure diagram of an example of a computer. In addition, the description of the parts identical to Figure 6 and Figure 7 is omitted.

[0053] Figure 8The computer may include an input device 601, a display device 602, an external I / F 603, a RAM 604, a ROM 605, a CPU 606, a communication I / F 607, an HDD 608, a photographing device 609, etc., which are interconnected via a bus B. In Figure 8 the computer, the display device 602 may be an additional display device for management, and the photographing device 609 may be an additional photographing device for management. In addition, a configuration without one or both of the display device 602 and the photographing device 609 can be adopted. Figure 8 The computer shown communicates via the communication I / F 607 via a network 5.

[0054] <Software Structure>

[0055] As an example, the software structure of the information processing device 1 Figure 6 shown is described. In Figure 9 is shown a functional block diagram of an example of the information processing device according to the present embodiment. The information processing device 1 may include a photographing unit 11, an acquisition unit 12, a display unit 13, a determination unit 14, a correction method generation unit 15, a notification unit 16, and a registration unit 17.

[0056] The photographing unit 11 causes the photographing device 509 to photograph the face of the subject. The photographing may be still image photographing or moving image photographing, and may be color photographing or monochrome photographing.

[0057] The acquisition unit 12 acquires the image of the face of the subject that has been photographed. In addition, the acquisition unit 12 may acquire an image photographed by the information processing device 1, and may also acquire an image photographed by a device other than the information processing device 1. Further, the acquisition unit 12 may acquire data of the subject input through the input device 501, etc.

[0058] The display unit (face image display unit) 13 causes the display device 502 to display the face image acquired by the acquisition unit 12. The display unit 13 can cause the display device 502 to display a marker (such as a frame to be described later) indicating a given position for face alignment. This marker is a marker shown within the display area of the display device 502 in such a way that the relative position of the face of the subject with respect to the photographing device 509 is appropriate.

[0059] The determination unit 14 determines whether the acquired face image is suitable for subsequent processing. Specifically, it determines whether the relative position of the subject's face with respect to the imaging device 509 is appropriate. This determination can be made based on the relationship between the position of the face image displayed on the display device 502 and the above-mentioned markers shown within the display area of the display device 502. In addition, the determination unit 14 can also determine whether the orientation of the subject's face with respect to the imaging device 509 is appropriate, and / or whether the subject's eyelids are closed or not closed.

[0060] When the determination unit 14 determines that the relative position of the subject's face with respect to the imaging device 509 is inappropriate, the correction method generation unit 15 generates a method for correcting (modifying) the position of the face to an appropriate position. The correction method includes the direction in which the position of the subject's face should be moved and the degree (distance) to be moved, etc. In addition, when the determination unit 14 determines whether the orientation of the subject's face with respect to the imaging device 509 is appropriate, and / or whether the subject's eyelids are closed or not closed, and this determination is inappropriate, it includes the direction in which the orientation of the face should be changed and the degree to be changed, and the policy of whether to open or close the eyelids.

[0061] The notification unit 16 notifies the determination result of the determination unit 14 to the subject (to the examiner if there is an examiner, or to the examiner and the subject). In addition, when a correction method is generated by the correction method generation unit 15 when the determination result is inappropriate, the correction method can be notified. The form of notification is not particularly limited, and one or more of vision, hearing, and touch can be used. For example, the notification can be made by causing the display device 502 to display images, texts, etc., or an output device capable of outputting notifications recognizable by means other than vision such as sound and vibration can be provided in the information processing device 1, and this output device can notify the subject or the examiner or both.

[0062] The registration unit 17 registers the face image determined by the determination unit 14 together with the result of the determination unit 14. Since it is necessary to compare two or more face image data in subsequent processing (described later), when the Figure 1 shown face image data acquisition method is repeated, the acquired face image data is also registered each time. In addition, when a correction method is generated by the correction method generation unit 15, the registration unit 17 can also register the content of the generated correction method and the content of the notification made by the notification unit 16 (the form of notification, the number of notifications, and the timing, etc.).

[0063] In addition, the above description of the software structure is based on a single information processing device 1 ( Figure 3 and Figure 6 and Figure 7) Description of an example. However, in the case of a system structure composed of multiple computers, the functional blocks of the above software structure ( Figure 9 and Figure 16 ) can also be distributively included in multiple computers. For example, in the system structure shown in Figure 2 , the functional blocks of the photographing unit 11 and the display unit 13 can be arranged in the subject terminal 2, and the functional blocks of the acquisition unit 12, the determination unit 14, the calibration unit generation unit 15, the notification unit 16, and the registration unit 17 can be arranged in the server device 3.

[0064] <Processing>

[0065] The overall process of acquiring facial image data in this embodiment can be carried out according to the process shown in Figure 1 (referred to as the description of the basic steps above). In the first display step (S10), the display unit 13 causes the display device 502 to display the facial image of the subject photographed by the photographing unit 11 and acquired by the acquisition unit 12 without image masking. Next, in the second display step (S20), the display unit 13 causes the same display device to display the facial image of the subject photographed by the same photographing unit 11 and acquired by the acquisition unit 12 while performing image masking. The second display step (S20) can also be said to be a process of performing image masking so that at least a part of the face is covered in the state where the facial image is displayed in the first display step (S10). And during the second display step (S20), it is determined whether the displayed facial image is suitable for subsequent analysis processing, that is, it is determined whether the relative position of the subject's face with respect to the photographing device 509 is appropriate. If it is determined to be appropriate, through the facial image data acquisition step (S30), the acquisition unit 12 acquires the facial image data displayed in the second display step (S20) (the facial image data in the state where image masking is applied from the perspective of the subject).

[0066] In Figure 10 , specific examples of the display in the first display step (S10) and the second display step (S20) are shown. In Figure 10 , the display 210 is an example of a display device, and the camera (built-in camera) 250 is an example of a photographing device. In the first display step (S10), as shown in Figure 10 (a), the facial image photographed by the camera 250 (photographing device) is displayed on the display 210 (display device) in real time. At this time, it is possible to cause the display 210 to display a frame 212 indicating a given position where the face should be aligned (refer to Figure 2In the marker described above). The subject can, for example, align the subject's own face displayed on the display 210 so that it enters the frame 212 at an appropriate size, thereby adjusting the relative position of the face with respect to the imaging device 509.

[0067] Furthermore, in the second display step (S20), as Figure 10 (b) of FIG. shows, the display unit 13 causes the display 210 to display the face image of the subject in real time and also displays an image mask 215 that covers at least a part of the face of the subject. The image mask 215 may be applied so as to overlap the face image within a range corresponding to the content of subsequent processing using the acquired face image data, its use, purpose, etc. Therefore, the image mask 215 may cover a part of the face image, the whole face, or the entire display area. For example, if features are extracted from the expression of the face (smiling face, serious face, etc.) in the processing using the face image data, as Figure 10 (b) of FIG. shows, it is preferable to perform image masking on the whole face.

[0068] In Figure 11 FIG., an example of the detailed processing of the second display step (S20) is shown. As Figure 11 shown, the second display step (S20) may include: a display step (S21) of displaying at least a part of the face of the subject captured by image masking; a determination step (S22) of determining whether or not the relative position of the face with respect to the imaging device is appropriate in a state where image masking is performed; and a determination result notification step (S23) of notifying the determination result, that is, notifying whether or not the relative position of the face with respect to the imaging device is appropriate. Then, when the relative position of the face is appropriate (S24), the process proceeds to the face image data acquisition step (S30), and when the relative position of the face with respect to the imaging device is not appropriate (S24), the process proceeds to a correction method generation step (S25) of generating a correction method for the relative position of the face, and then a correction method notification step (S26) of notifying the generated correction method to either or both of the subject and the examiner.

[0069] As described above, an example of the display in the second display step (S20) (in the display step (S21)) is as Figure 10As shown in (b) thereof. In the state where the image masking 215 is applied in this way, the determination unit 14 determines whether the facial image acquired by the acquisition unit 12 is an image suitable for subsequent processing. The determination of suitability or unsuitability is a determination of whether the relative position of the subject's face with respect to the camera 250 (imaging device 509) is appropriate. This determination may include, for example, determining whether at least a part of the subject's face displayed on the display 210 (display device 502) is located at a given position in the display area of the display 210, and / or whether it is of a given size. For this purpose, the determination unit 14 can, for example, display a frame 212 in the center of the display 210 through the display unit 13, determine whether the subject's face is contained in this frame 212, and determine whether the area of the subject's face in the display 210 occupies a given proportion or more of the area of the region surrounded by the frame 212. The above-mentioned given proportion of the area of the face in the frame 212 can be determined to be 60% or more, 70% or more, 80% or more, etc. with respect to the area surrounded by the frame 212.

[0070] In addition, the determination using the frame 212 is an example. For example, it is also possible to detect the position and size of the face in the display area of the display 210 (display device 502) in the first display step (S10), the registration unit 17 registers the detection result as the face position in the first display step (S10), and the determination unit 14 compares the face position in the first display step (S10) with the face position in the display step (S21) in the second display step (S20) to determine whether there is a deviation from the face position in the first display step (S10). In this case, as the reference face position in the first display step (S10), the position of the center of the face (e.g., the tip of the nose) and the size of the face can also be registered. Then, the determination unit 14 can also compare the center position (or the position of the tip of the nose) of the face and the size of the face based on the above-mentioned reference face position with the center position (or the position of the tip of the nose) and the size of the displayed face obtained in the display step (S21) of the second display step (S20).

[0071] In addition, the criteria for judging suitability or unsuitability (including the proportion of the area of the face within the above-mentioned frame 212) are determined according to the type, process, use, purpose, etc. of the subsequent processing. For example, in Figure 10In the example shown, since subsequent processing assumes a situation where an image of the entire face of the subject is required (such as the pressure diagnosis processing described later), the determination unit 14 determines whether the relative position of the entire face of the subject with respect to the camera 250 is appropriate. In contrast, in the case where the subsequent processing is a process that only requires an image of a part of the subject's face, the determination unit 14 determines whether the relative position of only a part of the subject's face with respect to the camera 250 is appropriate. For example, the determination unit 14 can determine whether only the part of the face that is the object is included in the frame 212, and whether the area of this part in the display 210 occupies a given ratio or more of the area of the region surrounded by the frame 212.

[0072] Next, in the determination result notification step (S23), the notification unit 16 notifies the result of the determination made by the determination unit 14. The notification can be given to the subject, and in the case where there is an examiner in addition, the notification can be given to the examiner or both the examiner and the subject. The form of the notification is not particularly limited, and can be a form that can be perceived visually, auditorily, or tactilely. For example, the notification can be made by displaying an image, text, etc. on the display 210 (display device 502) as described above, or can be notified from the output device in the information processing device 1 by sound, vibration, etc. If the face is included in the frame 212 in the display 210 and the area of the face with respect to the area of the region surrounded by the frame 212 is a given ratio or more, the determination unit 14 determines that the face image is appropriate, otherwise it determines that the face image is inappropriate. If the notification is in the form of text, it can also be in a manner of overlapping or not overlapping with the image masking 215. For an appropriate face image, for example, text such as "The position of the face is OK." is displayed, and for an inappropriate face image, for example, text such as "The position of the face is deviated!" is displayed. When the visual notification is displayed at the center of the display area in a manner of overlapping with the image masking 215, it is preferably able to attract the attention of the subject.

[0073] After the determination unit 14 determines the appropriateness of the relative position of the subject's face with respect to the camera 250 (imaging device 509) in the determination step S22, whether or not passing through the determination result notification step (S23), it proceeds to step S24. If the determination result of the determination unit 14 is an appropriate result, it proceeds to the face image data acquisition step (S30, Figure 1)。On the other hand, if the determination result of the determination unit 14 is an inappropriate result, the process proceeds to the correction method generation step (S25). In the correction method generation step (S25), the correction method generation unit 15 generates a method for correcting or modifying the position of the face to be appropriate. The correction method generation unit 15 can compare the position of the face image in the display area of the display 210 with the appropriate position (proper position) where the face should be located, and calculate the direction in which the face should be moved to become the proper position and, if necessary, calculate the amount of movement. In addition, the correction method generation unit 15 can compare the size of the face image in the display area of the display 210 with the appropriate size (proper size) that the face should have, and calculate the direction in which the face (or body) should be moved to become the proper size and, if necessary, calculate the amount of movement.

[0074] After the generation of the correction method, the process proceeds to the correction method notification step (S26). In the correction method notification step (S26), the notification unit 16 notifies the subject of the correction method generated by the correction method generation unit 15 as described above. For example, as Figure 10 shown in (b) and (c) of, during the display step (S21), that is, during the application of the image masking 215, sometimes the face of the subject deviates in a certain direction in the left-right direction (the left-right direction of the subject's body). In this case, the above-described correction method generation unit 15 generates a correction method for correcting the position of the face in the direction opposite to the direction of deviation. The notification unit 16 can notify this correction method as shown in, for example, Figure 10 (c) of. That is, in Figure 10 the example shown, the correction method 217 as an arrow is notified. The notification at this time can be displayed in front of the image masking 215 of the display 210 (display device 502). Thereby, the subject can recognize the direction in which the position of the face should be moved and can correct the position of the face.

[0075] The correction of the position of the face can be a correction for moving the face of the subject along either one or both of the up-down direction and the left-right direction to approach the proper position in the display area of the display 210 (display device 502). In addition, it can be a correction for moving the face of the subject closer to or farther from the camera 250 (imaging device 509) so that the face approaches the proper size in the display area of the display 210 (display device 502).

[0076] After the subject corrects the position of the face, the process returns to the display step (S21). Then, steps S21 to S24 are performed as described above, and depending on the situation, the correction method generation step (S25) and the correction method notification step (S26) are performed again, and the process returns to the display step (S21). Steps S21 to S26 can be repeated.

[0077] In addition, in the above-described determination step (S22), the determination unit 14 may not only determine whether the relative position of the subject's face with respect to the imaging device is appropriate, but also determine whether the orientation of the subject's face with respect to the face of the camera 250 (imaging device 509) is appropriate and / or determine whether the subject's eyelids are open or closed. Here, whether the orientation of the face is appropriate may refer to whether the orientation of the face (the direction in which the line of sight points in the state of natural eyes, or the normal direction of the plane formed by averaging the unevenness on the surface of the face) is substantially the same as the imaging direction of the camera 250. In this case, for example, in the case where the face being directly facing the camera 250 (imaging device 509) is the appropriate orientation of the face, the correction method in the correction method generation step (S25) is an operation of rotating the face so that the orientation of the face faces the camera 250 directly. In addition, in the case where the state of the subject's eyelids being open is appropriate, it is an operation of opening the subject's eyelids.

[0078] <Alternative Example>

[0079] Next, a description will be given of an alternative example of the second display step (S20) with reference to Figures 12 to 14 In the example shown in Figure 12 Similar to the example shown in Figure 10 in the display step (S21), the image mask 215 ( Figure 12 (b)) is displayed. However, the difference is that in the determination result notification step (S23), the notification unit 16 notifies the determination result as a display of both the frame 212 and the current position 218 of the face. The same frame 212 as the one displayed by the display unit 13 in the first display step (S10) shown in Figure 12 (a) is also displayed again in front of the image mask 215 in the second display step (S20). In addition, the current position 218 of the face only needs to roughly show the outline of the face. As shown in Figure 12 (c), it may also be displayed with a circular or elliptical contour line.

[0080] The subject who sees the determination result as shown in Figure 12 (c) can compare the current position 218 of the face with the frame 212 and correct the position of the face by himself / herself. The notification unit 16 may also, when notifying the determination result (such as the current position 218 of the face, etc.) in the determination result notification step (S23), additionally notify the correction method generated by the correction method generation unit 15. For example, it may also notify the correction method 217 (such as an arrow may also be displayed) as shown in Figure 10 (c).

[0081] In Figure 13shows an example of a second display step (S20) configured to include a subject terminal 2 and an examiner terminal 4 ( Figure 5 ). Examples of the subject terminal 2 ( Figure 13 ) are shown in (a1) and (a2) of Figure 5 , and examples of the examiner terminal 4 ( Figure 13 ) are shown in (b1) and (b2) of Figure 5 . Figure 13 (a1) and (b1) of Figure 13 are examples of the first display step (S10), and (a2) and (b2) of

[0082] are examples of the determination result notification step (S23) and the correction method notification step (S26) of the second display step (S20). Figure 13 As shown, a specific example of the subject terminal 2 in this example includes a camera 250 as a photographing device 509 and a display 210 as a display device 502. In addition, a specific example of the examiner terminal 4 may include a camera (not shown) as a photographing device (management photographing device) 609 and a display 410 as a display device (management display device) 609.

[0083] In this example, in the first display step (S10), the display unit 13 displays the face image of the subject captured by the camera 250 (photographing device 509) on the display 210 (display device 502) of the subject terminal 2 and also on the display 410 (management display device 602) of the examiner terminal 4. Thereby, the examiner can confirm the display state of the subject's face in the first display step (S10). In addition, the display unit 13 can also display an image captured by a photographing device other than the camera 250 in a sub-region 220 within the display area of the display 210 of the subject terminal 2. In this example, in the sub-region 220, an image captured by a camera (management photographing device 609) that photographs the examiner's face is displayed. Thereby, the subject can see the examiner's face, receive instructions from the examiner, or talk to the examiner. The examiner can give instructions to the subject or talk to the subject. In addition, the display unit 13 also displays the same image as the image displayed in the sub-region 220 of the subject terminal 2 in a sub-region 420 within the display area of the display 210 of the examiner terminal 4. By observing the sub-region 420 of the examiner terminal 4, the examiner can confirm how his / her posture looks to the subject.

[0084] In the second display step (S20), the display unit 13 only displays the image mask 215 on the subject terminal 2, and displays the face image of the subject on the examiner terminal 4 without the image mask. Therefore, in the display step (S21) of the second display step (S20), the examiner can confirm the position of the face of the subject within the display area of the display 210 of the subject terminal 2 that the subject cannot grasp or has difficulty grasping. For example, the position of the subject's face relative to the frame 212 within the display area of the display 210 of the subject terminal 2, and the ratio of the area occupied by the face to the area of the region surrounded by the frame 212 can be confirmed. Therefore, in the determination step (S22), the determination unit 14 determines whether the relative position of the subject's face with respect to the imaging device is appropriate, and the examiner can also determine whether the relative position of the subject's face with respect to the imaging device is appropriate. As a result, the determination becomes more accurate. In addition, the determination result may be notified to the subject by the examiner together with or instead of the notification of the determination result by the notification unit 16.

[0085] In addition, when the determination result of the determination unit 14 is inappropriate, that is, when it is determined that the relative position of the subject's face with respect to the imaging device is inappropriate, the examiner can also propose a correction method together with or instead of the generation of the correction method by the correction method generation unit 15 in the correction method generation step (S25). As a result, a correction method that is easy to convey to or understand by the subject can be generated. Furthermore, in the correction method notification step (S26), the correction method can be notified to the subject by the examiner together with or instead of the notification of the correction method performed by the notification unit 16. For example, as shown in (a2) of Figure 13 , the sound input by the examiner from the input device 601 etc. can be output from the output device in the subject terminal 2. Specifically, a correction method 217 based on statements such as "Please get a little closer to the camera!", "Please move away from the camera!", "Please move to the left" can be notified to the subject.

[0086] In Figure 14This shows a display example during the process of acquiring facial image data when using two or more facial image data in subsequent evaluations or diagnoses. In such evaluations or diagnoses, two or more facial image data are compared for analysis. For example, two or more facial image data captured with the same facial position but different expressions are compared for analysis. In this case, between two or more facial image data, not only should the relative position of the subject's face with respect to the imaging device be appropriate (not only should the facial image be displayed at an appropriate position and / or appropriate size in the display area), but if the inclination of the face is also different, appropriate analysis may sometimes not be possible. Here, the inclination of the face means that the center line extending along the up-down direction of the face is approximately the same. Even if the faces in two or more facial image data are directly facing the imaging device 509, if the inclination of the face is different, appropriate analysis may sometimes not be possible.

[0087] In Figure 14 the example shown, two facial image data, namely the first facial image data and the second facial image data, are acquired. In Figure 14 (a1) to (a3) of Figure 14 the acquisition of the first facial image data is shown, and in Figure 14 (b1) to (b3) of Figure 14 the acquisition of the second facial image data is shown. Both the first facial image data and the second facial image data are acquired through processing in accordance with the above-described method for acquiring facial image data ( Figure 1 and Figure 11 ).

[0088] Here, the first facial image data is determined by the determination unit 14 to be appropriate facial image data, but as shown in Figure 14 (a1) of Figure 14 , the face in this facial image data may sometimes be inclined to either the left or the right. In other words, the center line extending along the up-down direction of the face may sometimes be inclined with respect to the center line extending along the up-down direction of the display area of the camera 250 (display device 502). However, when acquiring the second facial image data, as shown in Figure 14 (b1) of Figure 14 , the inclination of the face may sometimes not be maintained. In response to this, in the determination step (S22) of the second display step (S20) when acquiring the second facial image data, the determination unit 14 also determines whether the inclination of the face captured by the camera 250 (imaging device 509) and displayed by the display 210 (display device 502) is the same as the inclination of the face in the first facial image data. Furthermore, in the determination result notification step (S23), the notification unit 16 can notify the determination result regarding the inclination of the face. Regarding the form of the notification, as described above, there is no particular limitation. For example, as shown in Figure 14 (b3) of Figure 14 , the current position of the face (the position of the current face and the up-down direction center line) 218 can be displayed in front of the image mask 215.

[0089] In addition, when it is determined in the determination by the determination unit 14 in the determination step (S22) that the inclination of the face is inappropriate, in the correction method generation step (S25), the correction method generation unit 15 generates a correction method for making the inclination of the face appropriate. Then, in the correction method notification step (S26), the generated correction method can be notified to the subject. For example, as shown in (b3) of Figure 14 , the notification unit 16 can display a correction method (the rotation direction of the arrow) 217 that makes the inclination of the face approach the inclination of the face in the first image data in front of the image mask 215 to notify the subject.

[0090] In this way, even when two or more facial image data are required for subsequent processing (for example, when two or more facial image data with different expressions are required), the facial image data acquisition method ( Figure 1 ) can be repeated to make the relative position of the face with respect to the imaging device, the inclination of the face, etc. consistent between two or more facial image data, enabling appropriate analysis.

[0091] Furthermore, an example in which the facial image is a part of the face is shown in Figure 15 . In subsequent processing, when analyzing based on image data of a part of the face rather than the entire face, for example, when analyzing the wrinkles at the corners of the eyes, it is sufficient to acquire at least an image of the corner part of the eye. In this case, whether the relative position of the face with respect to the camera 250 (imaging device 509) is appropriate means whether a given part of the face (in this example, the corner part of the eye) with respect to the camera 250 (imaging device 509) is appropriate.

[0092] The subject, for example, as shown in (a) of Figure 15 , in the first display step (S10), adjusts the relative position of the face with respect to the camera 250 (imaging device 509) so that the corner part of his own eye enters the frame 212. In addition, since it is sometimes difficult to visually determine whether the corner part of the eye is included in the frame 212, in the first display step (S10), the determination unit 14 can also determine whether a given part of the face (in this example, the corner part of the eye) is included in the frame 212. The form of detecting and determining the corner part of the eye can use a known method.

[0093] Next, in the display step (S21) of the second display step (S20), similarly to the above example, the display unit 13 displays the image mask 215 to cover part or all of the display area for the subject. At this time, the position of the subject's face may deviate. In the determination step (S22), the determination unit 14 determines whether the relative position of a part of the face (the corner of the eye part) with respect to the camera 250 (the imaging device 509) is appropriate. For example, this determination is made by determining whether a part of the face (the corner of the eye part) is included in the frame 212 and whether the area occupied by a part of the face (the corner of the eye part) is a given ratio or more with respect to the area surrounded by the frame 212.

[0094] Next, in the determination result notification step (S23), the notification unit 16 notifies the subject of the determination result of the determination unit 14. The form of notification can be the same as above. In addition, in the determination step (S22), when it is determined that the relative position of a part of the face (the corner of the eye part) with respect to the camera 250 (the imaging device 509) is inappropriate (S24), in the correction method generation step (S25), the correction method generation unit 15 generates a correction method for making the relative position of a part of the face (the corner of the eye part) appropriate. In the correction method notification step (S26), the notification unit 16 notifies the subject of the correction method. For example, as shown in (c) of Figure 15 , if the correction method is to move the position of the face upward, the notification unit 16 displays the correction method 217 as an upward arrow to notify the subject. The correction method 217 can be displayed in front of the image mask 215 together with the position of the frame 212. Based on the correction method notified in this way, the subject can correct (amend) the position of his or her own face, and appropriate facial image data can also be obtained in the second display step (S20) where the image mask 215 is applied.

[0095] <Application Example>

[0096] The facial image data obtained by the above method for obtaining facial image data can be suitably used for the evaluation or diagnosis of the physical state and / or mental state of the subject. For example, it can be used for stress evaluation described in International Publication No. 2019 / 073906 (or Japanese Patent No. 7107958).

[0097] Therefore, one embodiment is a method for evaluating the stress of a subject, which includes:

[0098] (1) A step of instructing the subject to face the imaging device and make the biggest smiling face;

[0099] (2) A step of using the imaging device to capture the face of the subject;

[0100] (3) Steps of processing the captured facial image to obtain parameters reflecting pressure; and

[0101] (4) Steps of performing calculations using the parameters obtained from the processing of the image to calculate a pressure index value,

[0102] In step (3), determine the positions of the left and right eye corners and the left and right mouth corners in the facial image, draw a straight line (a1 - b1) passing through one eye corner (a1) and mouth corner (b1) and a straight line (a2 - b2) passing through the other eye corner (a2) and mouth corner (b2), measure the angle (θ1) formed by the straight line (a1 - b1) and the reference line and the angle (θ2) formed by the straight line (a2 - b2) and the reference line, and use the values of these "θ1" and "θ2" as the parameters in step (4).

[0103] In this method, in the photographing of the subject's face in step (2), the above-mentioned facial image data acquisition method can be utilized. That is, the "captured facial image" processed in step (3) can be set as an appropriate facial image obtained by the above-mentioned facial image data acquisition method.

[0104] In addition, the reference line in step (3) of the above method for evaluating pressure can be the horizontal line of the image. In addition, the calculation in step (4) can include calculating the angle difference (θ1 - θ2) between θ1 and θ2, and the angle difference (θ1 - θ2) can be used as the pressure index value. In addition, step (3) can further include: determining the positions of the center point (c1) of the right eye and the center point (c2) of the left eye in the captured image, and drawing the center line of the eyes passing through the center point (c1) of the right eye and the center point (c2) of the left eye. In addition, the reference line in step (3) is the center line of the eyes, the calculation in step (4) includes calculating the angle difference (θ1 - θ2) between θ1 and θ2, and this angle difference (θ1 - θ2) can be used as the pressure index value.

[0105] In addition, another embodiment is a method for evaluating the pressure of a subject, which includes:

[0106] (1A) Steps of instructing the subject to face the imaging device with the face and make the biggest smiling face;

[0107] (2A) Steps of using the imaging device to photograph the biggest smiling face;

[0108] (1B) Steps of instructing the subject to become a serious face;

[0109] (2B) Steps of using the imaging device to photograph the serious face;

[0110] (3A) A step of processing the image processing of the image of the biggest smiling face captured in the above (2A) to obtain a parameter reflecting pressure;

[0111] (3B) A step of processing the image processing of the image of the serious face captured in the above (2B) to obtain a parameter reflecting pressure; and

[0112] (4) A step of using the parameter obtained from the processing of the image to perform an operation to calculate a pressure index value,

[0113] Step (3A) includes the following processing: drawing a straight line (a1 - b1) passing through the outer corner of the eye (a1) and the corner of the mouth (b1) on one side in the image of the biggest smiling face captured in step (2A) and a straight line (a2 - b2) passing through the outer corner of the eye (a2) and the corner of the mouth (b2) on the other side, and measuring the angle (θ1) formed by the straight line (a1 - b1) and the reference line and the angle (θ2) formed by the straight line (a2 - b2) and the reference line as parameters,

[0114] Step (3B) includes the following processing: drawing a straight line (a1" - b1") passing through the outer corner of the eye (a1") and the corner of the mouth (b1") on one side in the image of the serious face captured in step (2B) and a straight line (a2" - b2") passing through the outer corner of the eye (a2") and the corner of the mouth (b2") on the other side, and measuring the angle (θ1") formed by the straight line (a1" - b1") and the reference line and the angle (θ2") formed by the straight line (a2" - b2") and the reference line as parameters,

[0115] The operation in step (4) calculates the angle difference (θ1 - θ2) of the parameters of the biggest smiling face obtained in step (3A), i.e., θ1 and θ2, and the angle difference (θ1" - θ2") of the parameters of the serious face obtained in step (3B), i.e., θ1" and θ2", and further calculates the difference of the two angle differences ((θ1 - θ2) - (θ1" - θ2")), and takes this difference ((θ1 - θ2) - (θ1" - θ2")) as the pressure index value,

[0116] In this method, in the shooting of the biggest smiling face in step (2A) and the shooting of the serious face in step (2B), the above-mentioned method for obtaining facial image data can be utilized. That is, the "image of the biggest smiling face" processed in step (3A) and the "image of the serious face" processed in step (3B) can respectively adopt appropriate facial images obtained by using the above-mentioned method for obtaining facial image data.

[0117] In addition, in the above pressure evaluation method, steps (1A), (2A), (1B), and (2B) can be repeatedly performed in sequence multiple times. The parameter in step (3A) can be the average value of θ1 and θ2 obtained through each measurement, and the parameter in step (3B) can be the average value of θ1" and θ2" obtained through each measurement.

[0118] In Figure 16 , a functional block diagram of an information processing device 1' that can implement the above-described pressure evaluation method for a subject, which is an application example of the present embodiment, is shown. In addition, the hardware structure of the information processing device 1' can be the same as the hardware structure of the information processing device 1 ( Figure 2 ). The hardware structure ( Figure 6 or Figure 7 ). The information processing device 1' has Figure 9 The structure (facial image data acquisition unit 10) included in the information processing device 1 shown, and may further have an operation reception unit 21, a smile value measurement unit 22, a feature point extraction unit 23, an instruction unit 24, a content generation unit 25, a pressure index calculation unit 26, a content display unit 27, and a pressure index storage unit 28. This structure can also be said to be a structure in which the image input unit in the functional blocks described in International Publication No. 2019 / 073906 (or Japanese Patent No. 7107958) is replaced by the facial image data acquisition unit 10.

[0119] The operation reception unit 21 receives operations from at least one of the subject and the examiner, such as an operation to start pressure diagnosis from the subject. For example, when the operation reception unit 21 receives an operation to start pressure diagnosis, it notifies the smile value measurement unit 22 and the instruction unit 24 of the start of pressure diagnosis, causing them to start pressure diagnosis processing.

[0120] The facial image data acquisition unit 10 acquires facial image data as described above (refer to Figures 1 to 14 ), and further provides it to the smile value measurement unit 22, the feature point extraction unit 23, and the content generation unit 25. The content generation unit 25 generates a content image based on the facial image provided from the facial image data acquisition unit 10. The content display unit 27 causes the display device 502 to display the content image generated by the content generation unit 25.

[0121] The smile value measurement unit 22 measures the smile value of the acquired facial image by a known method. The smile value (hereinafter referred to as the measured smile value) measured by the smile value measurement unit 22 is provided to the content generation unit 25 and is displayed in the content image. In addition, the smile value measurement unit 22 provides the measured smile value to the feature point extraction unit 23 and the instruction unit 24.

[0122] After the start of the stress diagnosis process, the instruction unit 24 provides the target value of the smile value (hereinafter referred to as the target smile value) to the content generation unit 25. The target smile value provided to the content generation unit 25 is displayed on the content image. Therefore, the subject can adjust the smile so that the measured smile value reaches the target smile value while observing the content image displayed on the display device 502.

[0123] The feature point extraction unit 23 is provided with the measured smile value from the smile value measurement unit 22 and the target smile value from the indication unit 24. The feature point extraction unit 23 extracts the positions of the left and right eye corners, mouth corners, the center point of the eye, etc. as the positions of the feature points from the facial image of the subject at the timing when the measured smile value becomes the target smile value. For example, for the stress assessment method of one embodiment described above, the positions of the eye corners (a1) and mouth corners (b1) on one side and the eye corners (a2) and mouth corners (b2) on the other side can be determined. Alternatively, for the stress assessment method of another embodiment described above, the positions of the eye corners (a1) and mouth corners (b1) and the center point (c1) of the eye on one side and the positions of the eye corners (a2) and mouth corners (b2) and the center point (c2) of the eye on the other side can be extracted.

[0124] The pressure index calculation unit 26 can calculate the pressure index value based on the positions of the feature points extracted by the feature point extraction unit 23. The pressure index storage unit 28 stores the calculated pressure index value as history data in association with date data and the like.

[0125] The present invention has been described above based on the embodiments, but the present invention is not limited to these embodiments. In addition, the above embodiments can be variously changed, modified, replaced, added, deleted and combined within the scope of the claims, which also belong to the technical scope of the present invention.

[0126] This application claims priority based on Japanese Patent Application No. 2022-198809 filed on December 13, 2022, and all contents thereof are incorporated herein by reference.

[0127] Description of symbols

[0128] 1, 1': Information processing device

[0129] 2: Test subject terminal

[0130] 3: Server device

[0131] 4: Inspector terminal

[0132] 5: Network

[0133] 11: Photography Department

[0134] 12: Acquisition

[0135] 13: Display unit

[0136] 14: Determination unit

[0137] 15: Calibration method generation unit

[0138] 16: Notification unit

[0139] 17: Registration unit

[0140] 21: Operation reception unit

[0141] 22: Smiling face value measurement unit

[0142] 23: Feature point extraction unit

[0143] 24: Instruction unit

[0144] 25: Content generation unit

[0145] 26: Pressure index calculation unit

[0146] 27: Content display unit

[0147] 28: Pressure index storage unit

[0148] 210: Display

[0149] 212: Frame

[0150] 215: Image masking

[0151] 217: Calibration method

[0152] 218: Current position

[0153] 250: Camera

[0154] 501, 601: Input device

[0155] 502, 602: Display device

[0156] 503, 603: External I / F

[0157] 503a, 603a: Recording medium

[0158] 504, 604: RAM

[0159] 505, 605: ROM

[0160] 506, 606: CPU

[0161] 507, 607: Communication I / F

[0162] 508, 608: HDD

[0163] 509, 609: Imaging device

[0164] B: Bus

Claims

1. A method, which is a method for obtaining facial image data, comprising: A first display step of displaying, in an image-unobscured manner, an image including at least a part of the face of the subject on a display device visible to the subject; And A second display step of displaying, in a manner of obscuring at least a part of the face of the subject, an image including at least a part of the face of the subject on the display device, In the second display step, Determine whether the relative position of at least a part of the face of the subject with respect to the imaging device is appropriate, When the relative position is appropriate in the determination, acquire the facial image data of the subject displayed.

2. The method according to claim 1, wherein, The determination of whether the relative position is appropriate includes determining whether at least a part of the face of the subject displayed is located at a given position in the display area of the display device and / or whether it is of a given size.

3. The method according to claim 1, wherein, The determination further includes determining whether the orientation of the face with respect to the imaging device is appropriate and / or the opening and closing state of the eyelids.

4. The method according to claim 2, wherein, In the second display step, it further includes at least notifying the determination.

5. The method according to claim 4, wherein, At the time of the notification, display the given position and / or the given size in the display area and / or the current position and / or the current size of at least a part of the face of the subject displayed in the display area.

6. The method according to claim 5, wherein, In the second display step, it further includes: When the relative position is not appropriate in the determination, generate, based on the determination, a correction method for making the relative position appropriate.

7. The method according to claim 6, wherein, The generation of the correction method includes generating a correction method for making the current position and / or size become the given position and / or the given size.

8. The method according to claim 6, wherein, The notification further includes notifying the correction method.

9. The method according to claim 8, wherein, The notification is performed visually and / or aurally.

10. The method according to claim 1, wherein, The facial image data includes first facial image data and second facial image data which is facial image data of an expression different from the first facial image data, When acquiring the first facial image data and when acquiring the second facial image data, make the relative position of at least a part of the face of the subject with respect to the imaging device the same.

11. The method according to claim 10, wherein, Furthermore, when acquiring the first facial image data and when acquiring the second facial image data, make at least one of the inclination of the face of the subject and the line of sight the same.

12. The method according to claim 6, wherein, Display, in an image-unobscured manner, an image including at least a part of the face of the subject on an additional display device visible to an examiner who is not the subject, which is different from the display device, Notify the examiner.

13. The method according to claim 12, wherein, Do not notify the subject.

14. The method according to any one of claims 1 to 13, wherein, The acquired facial image data is used to evaluate the physical state and / or mental state of the subject.

15. An information processing apparatus, comprising: An acquisition unit; A determination unit; and A registration unit, The acquisition unit acquires: Data of a first image obtained by photographing, by a photographing device, an image that includes at least a part of the face of a subject and is displayed on a display device visible to the subject without image masking; and Data of a second image obtained by photographing, by the photographing device, an image that includes at least a part of the face of the subject and is displayed on the display device with image masking, The determination unit determines, for the data of the second image, whether the relative position of at least a part of the face of the subject with respect to the photographing device is appropriate, When the relative position is appropriate in the determination, the registration unit registers the facial image data of the subject being displayed.

16. A program for causing a computer to function as an acquisition unit, a determination unit, and a registration unit, wherein, The acquisition unit acquires: Data of a first image captured by an imaging device by displaying, in an image-unobscured manner, an image including at least a part of the face of the subject on a display device visible to the subject; And Data of a second image obtained by photographing an image that shows at least a part of the face of the subject with image masking on the display device The determination unit determines whether or not the relative position of at least a part of the face of the subject with respect to the photographing device is appropriate for the data of the second image When the relative position is appropriate in the determination, the registration unit registers the face image data of the subject that is displayed 17. An information processing system, comprising a photographing device, a display device, and an information processing apparatus, The information processing apparatus comprises an acquisition unit, a determination unit, and a registration unit, The acquisition unit acquires: Data of a first image obtained by photographing, by the photographing device, an image that includes at least a part of the face of a subject and is displayed on the display device visible to the subject without image masking; and Data of a second image obtained by photographing, by the photographing device, an image that includes at least a part of the face of the subject and is displayed on the display device with image masking, The determination unit determines the appropriateness of the relative position of at least a part of the face of the subject with respect to the photographing device for the data of the second image, When the relative position is appropriate in the determination, the registration unit registers the facial image data of the subject being displayed.

Images