Terminal device and measurement method

Abstract

To provide a terminal device capable of more appropriately measuring biological information for a subject by photographing the subject.SOLUTION: A terminal device (100) includes an image acquisition unit (11) for acquiring a captured image of a subject photographed with a photographing device (2), an image processing unit (12) for calculating an index value showing the brightness of a face image of the subject in the captured image, an output control unit (14) for outputting the biological information for the subject measured on the basis of the face image, and a determination processing unit (16) for determining whether the index value satisfies prescribed measurability conditions. If the index value does not satisfy the measurability conditions, the output control unit outputs guidance information for changing the orientation of the face image such that the index value satisfies the measurability conditions.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a terminal device for contactlessly measuring biological information of a subject, and a measurement method using the terminal device. [Background technology]

[0002] Conventionally, there are known techniques for guiding a person to be photographed so that the face of the person to be photographed can be photographed appropriately. For example, Patent Document 1 discloses a face photographing device that detects the direction of bright light using a light direction detection means and guides the person to stand in the direction of the detected light. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4811863 specification Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned technology aligns the direction of the light source with the direction of the person being photographed so that shadows that are unsuitable for face matching do not appear on the face image of the person being photographed, and when used in non-contact healthcare sensing that detects the biometric information of the person being measured based on a captured image of the person being measured, the above-mentioned technology is not sufficient as a guide for performing appropriate measurements.

[0005] In other words, if the technology of Patent Document 1 is used to guide the subject to stand in the direction of bright light, the subject's face will be relatively dark due to backlighting, and the brightness of sensing areas such as the cheeks will be insufficient, which may prevent proper measurement.

[0006] An aspect of the present invention is to provide a terminal device and a measurement method that can more appropriately capture an image of a subject and measure the subject's biological information. [Means for solving the problem]

[0007] In order to solve the above problem, a terminal device according to one aspect of the present invention comprises an image acquisition unit that acquires an image of a person being measured captured by an imaging device, an image processing unit that calculates an index value that indicates the brightness of the facial image of the person being measured in the captured image, an output control unit that outputs biometric information of the person being measured measured based on the facial image, and a judgment unit that judges whether the index value satisfies a predetermined measurable condition, and if the index value does not satisfy the measurable condition, the output control unit outputs guidance information for changing the orientation of the facial image so that the index value satisfies the measurable condition.

[0008] In order to solve the above problem, a measurement method according to one aspect of the present invention is a measurement method using a terminal device equipped with an imaging device, and includes: an acquisition step of acquiring an image of a person being measured captured by the imaging device; an image processing step of calculating an index value indicating the brightness of the facial image of the person being measured in the captured image; a biometric information output step of outputting biometric information of the person being measured measured based on the facial image; a determination step of determining whether the index value satisfies a measurable condition; and a guidance information output step of, if the index value does not satisfy the measurable condition, outputting guidance information for changing the orientation of the facial image so that the index value satisfies the measurable condition. [Effects of the Invention]

[0009] According to one aspect of the present invention, it is possible to more appropriately photograph a subject and measure the subject's biological information. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a schematic configuration of a terminal device according to an embodiment of the present invention; [Figure 2] 10A to 10C are diagrams showing the operation of a measurement operation performed using the terminal device together with screen images. [Figure 3] 10 is a flowchart of a guidance example 1 in the terminal device. [Figure 4] 10 is a flowchart of a guidance example 2 in the terminal device. [Figure 5] 10 is a flowchart of a guidance example 3 in the terminal device. [Figure 6] 6 is a diagram showing the state of the person being measured and an example of a guidance screen D4 displayed in S22 in the flowchart of FIG. 5. FIG. [Figure 7] FIG. 11 is a diagram showing an example of a measurement start screen D5 after the measurer has moved in guidance example 3. [Figure 8] 10 is a flowchart of a guidance example 4 in the terminal device. [Figure 9] FIG. 9 is a diagram showing an example of the guidance performed in S37 in the flowchart of FIG. [Figure 10] FIG. 10 is a diagram showing an example of guidance to stop the movement of the person being measured and start measurement in guidance example 4. [Figure 11] 10 is a flowchart of a guidance example 5 in the terminal device. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, one embodiment of the present invention will be described in detail.

[0012] (1. General configuration of terminal device) 1 is a block diagram showing a schematic configuration of a terminal device 100 according to an embodiment of the present invention. As shown in Fig. 1, the terminal device 100 includes a control unit 1, an image capturing device 2, an input device 3, a display device 4, a communication device 5, a storage device 6, a temperature measuring device 7, a microphone 8, and a speaker 9.

[0013] The terminal device 100 is a device used to measure the biometric information of the subject based on a captured image of the subject (i.e., without contact). Examples of the biometric information include, but are not limited to, the subject's body temperature, blood pressure, pulse rate, respiratory rate, etc. The terminal device 100 may be any device that has the function of acquiring and displaying images, and may be, for example, a smartphone or tablet device.

[0014] The person taking the measurement measures the biological information of the person being measured by using the terminal device 100 to photograph the person being measured while checking an image including the person being measured displayed on the screen of the display device. Such a terminal device 100 is used, for example, in a care setting when a caregiver (measurer) measures the biological information of a person being cared for (person being measured).

[0015] The functions of the terminal device 100 described below can also be realized by installing a program on a general-purpose computer. The program can also be called application software. In the following, an example will be described in which the terminal device 100 is a smartphone on which the program is installed.

[0016] The control unit 1 controls each unit of the terminal device 100 in an integrated manner. The photographing device 2 photographs images. The input device 3 accepts input operations to the terminal device 100. The display device 4 displays images photographed by the photographing device 2. The input device 3 may be a touch panel, in which case the input device 3 is configured integrally with the display device 4.

[0017] The communication device 5 transmits and receives various data between an external communication device and the control unit 1. The communication device 5 connects the terminal device 100 and the server 200 via the network N, enabling data exchange between them. The storage device 6 stores various data handled by the terminal device 100. The temperature measurement device 7 detects the body temperature of the person being measured without contact.

[0018] (2. Control Unit) The control unit 1 includes an image acquisition unit 11 , an image processing unit 12 , a measurement processing unit 13 , an output control unit 14 , an authentication processing unit 15 , a determination processing unit 16 , and a specification processing unit 17 .

[0019] The image acquisition unit 11 acquires an image of the subject captured by the imaging device 2 (acquisition step).

[0020] The image processing unit 12 calculates an index value indicating the brightness of the facial image of the subject in the photographed image acquired by the image acquisition unit 11 (image processing step). The image processing unit 12 identifies facial parts such as cheeks, eyes, mouth, and forehead from the photographed image, and detects the brightness of each part and each region of the facial image.

[0021] The measurement processing unit 13 measures the biological information of the subject based on the photographed image of the subject. Specifically, the measurement processing unit 13 measures the biological information of the subject based on the facial image of the subject photographed by the photographing device 2. The measurement processing unit 13 also measures the body temperature of the subject based on the facial image of the subject photographed by the temperature measuring device 7, which is a type of photographing device.

[0022] The output control unit 14 outputs the biometric information of the subject measured by the measurement processing unit 13 based on the face image (biometric information output step). The output target device to which the output control unit 14 outputs the biometric information may be the display device 4, the speaker 9, or both. The output control unit 14 is a control device that controls the display device 4 and also a control device that controls the speaker 9. The output control unit 14 may also transmit the biometric information of the subject to the server 200 via the communication device 5.

[0023] The determination processing unit (determination unit) 16 determines whether the index value satisfies a predetermined measurable condition (determination step). If the index value satisfies the predetermined measurable condition, the measurement accuracy by the measurement processing unit 13 increases, and correct measurement can be performed. If the index value does not satisfy the predetermined measurable condition, the measurement accuracy by the measurement processing unit 13 decreases, and there is a possibility that correct measurement cannot be performed.

[0024] When the index value does not satisfy the measurable condition, the output control unit 14 outputs guidance information for changing the orientation of the face image so that the index value satisfies the measurable condition (guidance information output step). That is, when the determination processing unit 16 determines that the index value does not satisfy the measurable condition, the output control unit 14 outputs guidance information for changing the orientation of the face image so that the index value satisfies the measurable condition.

[0025] In other words, the output control unit 14 outputs guidance information when guidance is necessary based on the determination result by the determination processing unit 16, and provides guidance to enable more accurate measurement. The output target device to which the output control unit 14 outputs the guidance information may be the display device 4, the speaker 9, or both.

[0026] Hereinafter, the output of guidance information by the output control unit 14 may be expressed as "providing guidance." Also, when the output target device is the display device 4, visual guidance may be expressed as "display guidance," and when the output target device is the speaker 9, audio guidance may be expressed as "audio guidance."

[0027] The authentication processing unit 15 transmits the facial image of the person being measured taken by the photographing device 2 to the server 200 via the communication device 5, and acquires personal information as an authentication result from the server 200. The authentication processing unit 15 has a function as a personal information acquisition unit, and acquires the personal information of the person being measured by face recognition using the photographed image. The personal information includes at least the name of the person being measured, and may also include information on the person's past measurement results.

[0028] The identification processing unit (identification unit) 17 refers to the correlation between the index value at the time of measurement and the measurement result, and identifies the item of biological information for which the measurement accuracy will be low if the index value does not satisfy a predetermined measurable condition.

[0029] (3. Measurement operation) 2 is a diagram showing, together with screen images, the operation of a measurement operation (measurement method) performed using the terminal device 100. When starting measurement, the output control unit 14 displays a measurement standby screen D1 on the display device 4.

[0030] As shown in FIG. 2, the measurement standby screen D1 displays a frame image 101, an instruction message 102, a staff member name 103, a staff member change button 104, a logout button 105, and the like.

[0031] The frame image 101 indicates the measurement area, and is an area in which the face image of the person being measured is placed. The instruction message 102 is a message indicating instructions for the person taking the measurement. The staff name 103 is the name of the logged-in operator who will be performing the measurement using the terminal device 100. The staff change button 104 accepts a change of operator, and the logout button 105 accepts logout.

[0032] The person measuring moves the terminal device 100 and places the face of the person being measured within the frame image 101, causing the image acquisition unit 11 to acquire a facial image of the person being measured, and the authentication processing unit 15 to start face detection processing. The authentication processing unit 15 transmits a captured facial image of the face image within the frame image 101 to the server 200, and acquires personal information from the server 200.

[0033] When the person information is acquired, the measurement processing unit 13 starts measurement based on the face image in the frame image 101, and at that time, if guidance is needed according to the determination result of the determination processing unit 16, the output control unit 14 provides guidance to enable more accurate measurement. The guidance may include displaying a guidance screen D4 (see FIG. 6) described later or outputting audio guidance.

[0034] When guidance is not required or the measurement situation improves (when the judgment processing unit 16 judges that the conditions for measurement are met), the measurement processing unit 13 starts measurement, and the output control unit 14 switches the display to the measurement-in-progress screen D2. The measurement-in-progress screen D2 displays a message 107 indicating that measurement is in progress, a facial image 108 of the subject being photographed, information 109 about the time remaining until the end of measurement, and the like.

[0035] If the face image of the person being measured deviates beyond the allowable range during measurement, the measurement processing unit 13 determines that the face has not been detected and stops the measurement, and the output control unit 14 switches the display to the measurement standby screen D1. When the face image re-enters the frame image 101 on the measurement standby screen D1, the measurement processing unit 13 resumes measurement. In this case, if guidance is needed, the output control unit 14 provides guidance to enable more accurate measurement.

[0036] When the measurement by the measurement processing unit 13 is completed, the output control unit 14 switches the display from the measurement in progress screen D2 to a measurement result screen D3. The measurement result screen D3 displays personal information 111 received from the server 200, a facial image 112 of the person being measured, comparison result information 113 received from the server 200, measurement value information 114, a button 115 to go to measurement screen, etc.

[0037] The person information 111 is information such as the name of the person being measured, and the facial image 112 of the person being measured may be the facial image of the person being measured that the authentication processing unit 15 has sent to the server 200 as authentication information. The comparison result information 113 is the result of the server 200 comparing the current measurement result with past measurement results. In the example of Fig. 2, the comparison result information 113 displays "similar level to previous level," which indicates that the level of the current measurement value is the same as the level of the previous measurement value.

[0038] (4. Guidance Examples) Hereinafter, the guidance performed by the terminal device 100 will be specifically described with reference to FIGS.

[0039] <Guidance example 1> The image processing unit 12 calculates index values ​​for the chin, forehead, right cheek, and left cheek of the subject's face. The determination processing unit 16 determines whether the calculated index values ​​satisfy the measurable conditions. If the chin index value does not satisfy the measurable conditions and the forehead index value does, or if any index value other than the forehead does not satisfy the measurable conditions, the output control unit 14 outputs information for changing the orientation of the facial image to face upward as guidance information.

[0040] 3 is a flowchart of guidance example 1 in the terminal device 100. Guidance example 1 provides guidance to have the subject turn their face up. This guidance is provided to change the orientation of the captured facial image. The subject of each step in the flowchart of FIG. 3 will be described as the control unit 1.

[0041] As shown in Fig. 3, the control unit 1 determines whether the index value indicating the brightness of the chin satisfies the measurement enable condition (S1). If the control unit 1 determines YES in S1, it proceeds to S6. In S6, the control unit 1 starts the measurement without providing any guidance. In other words, if the index value of the chin satisfies the measurement enable condition, it is considered that the person being measured is not looking down, so the control unit 1 does not provide any guidance to turn the face up.

[0042] On the other hand, if the control unit 1 determines NO in S1, it next determines whether the index value indicating the brightness of the forehead satisfies the measurement condition (S2). If the control unit 1 determines YES in S2, it provides guidance to turn the face up (S5).

[0043] Even if the chin index value does not meet the measurable conditions, if the forehead index value does, more accurate measurements can be obtained by asking the subject to lift their head. Therefore, guiding the subject to lift their head will enable more accurate measurements.

[0044] In S5, the control unit 1 (output control unit 14) provides voice guidance including, for example, "Please look up until I say yes" as guidance information. Alternatively, the control unit 1 provides display guidance including, for example, "Please look up until I say yes" and has the subject read it out loud. The control unit 1 may provide both voice guidance and display guidance. Furthermore, in the guidance in S5, the control unit 1 may include information such as the subject's name acquired by the authentication processing unit 15 in the guidance information.

[0045] On the other hand, if the control unit 1 determines NO in S2, it then determines whether the index value indicating the brightness of the right forehead meets the measurable condition (S3). If the control unit 1 determines YES in S3, it proceeds to the above-mentioned S6 and does not provide guidance. In other words, even if the index values ​​for both the chin and forehead do not meet the measurable condition, if the index value for the cheek, which is the measurement location, meets the measurable condition, measurement is possible as is, so the control unit 1 does not provide guidance to turn the face up.

[0046] On the other hand, if the control unit 1 determines NO in S3, it then determines whether the index value indicating the brightness of the left forehead meets the measurement condition (S4). If the control unit 1 determines YES in S4, it proceeds to the above-mentioned S6 and does not provide guidance. In other words, even if the index values ​​for the chin, forehead, and right cheek do not meet the measurement condition, if the index value for the left cheek meets the measurement condition, measurement is possible as is, so the control unit 1 does not provide guidance to turn the face up.

[0047] On the other hand, if the control unit 1 determines NO in S4 as well, it proceeds to the above-mentioned S5 and provides guidance to turn the face up. Note that although in Fig. 3, it is determined whether the index values ​​satisfy the measurement condition for the right cheek and then the left cheek, the order may be reversed.

[0048] In this way, if a more accurate measurement can be made by having the person being measured raise their face based on the index values ​​of the chin, forehead, right cheek, and left cheek of the person being measured, a more accurate measurement can be made by guiding the person being measured to raise their face.

[0049] In this embodiment, as a measurement operation procedure, authentication processing is performed by authentication processing unit 15 to acquire personal information such as the subject's name, but guidance example 1 shown in Fig. 3 can be given without acquiring personal information about the subject. The same applies to guidance examples 2 and 3, which will be described later.

[0050] <Guidance example 2> The image processing unit 12 identifies a proximate part that is closer to the image capturing device 2 and a proper part having brightness close to the proper index value from among predetermined symmetrical parts of the subject in the facial image. If the proximate part and the proper part do not match, the output control unit 14 outputs, as guidance information, information for changing the orientation of the facial image so that the proximate part and the proper part match.

[0051] FIG. 4 is a flowchart of guidance example 2 in the terminal device 100. In guidance example 2, guidance is given to measure whichever is more suitable for measurement of symmetrical predetermined parts. This guidance is given to change the orientation of the captured facial image. Note that, although FIG. 4 shows cheeks as an example of symmetrical predetermined parts, parts other than cheeks may also be used. The subject of each step in the flowchart of FIG. 4 will be described as the control unit 1.

[0052] 4, the control unit 1 determines whether the right cheek is closer based on the distance between the photographing device 2 and the right or left cheek (S11). If the control unit 1 determines YES in S11, it identifies the right cheek as the closer part from the photographing device 2 and proceeds to S12. On the other hand, if the control unit 1 determines NO in S11, it identifies the left cheek as the closer part from the photographing device 2 and proceeds to S14.

[0053] In S12, the control unit 1 determines whether the right cheek is brighter based on the index values ​​of the right and left cheeks. If the control unit 1 determines YES in S12, it identifies the right cheek as the appropriate area having brightness close to the appropriate index value, and proceeds to S16. In S16, the control unit 1 starts measurement without providing guidance. In other words, since both the adjacent area and the appropriate area are the right cheek, measurement is possible as is, and no guidance is provided to measure the more appropriate area.

[0054] In S14, the control unit 1 determines whether the left cheek is brighter based on the index values ​​of the right and left cheeks. If the control unit 1 determines YES in S14, it identifies the left cheek as the appropriate area having a brightness close to the appropriate index value, and proceeds to S16. In this case, too, since the adjacent area and the appropriate area are both the left cheek, measurement is possible as is, and no guidance is provided to measure the more appropriate area.

[0055] On the other hand, if the controller 1 determines NO in S12, it proceeds to S13 and provides guidance to measure from the left cheek, which is the appropriate site. In other words, since the adjacent site and the appropriate site do not match, guidance to measure from the left cheek, which is more suitable for measurement, enables more accurate measurement.

[0056] In S13, the control unit (output control unit 14) 1 displays guidance information including, for example, "Please measure from the left cheek," to inform the subject which cheek to measure.

[0057] Similarly, if the controller 1 determines NO in S14, it proceeds to S15 and provides guidance to measure from the right cheek, which is the appropriate site. In this case, too, providing guidance to measure from the right cheek, which is more suitable for measurement, allows for more accurate measurement.

[0058] In S15, the control unit 1 displays guidance information including, for example, "Please start measuring from the right cheek," to inform the subject which cheek to measure.

[0059] In this way, by providing guidance in S13 and S15 to measure from the cheek identified as the appropriate part, the proximal part and the appropriate part are guided to coincide with each other, allowing for more accurate measurement. Note that in Fig. 4, whether the proximal part is determined based on the right cheek in S11 may also be determined based on the left cheek.

[0060] Incidentally, guidance for changing the orientation of a captured facial image can be thought of as guidance in which the measurer moves, or guidance in which the subject moves. Below, guidance in which the measurer moves will be described as Guidance Example 3, and guidance in which the subject moves will be described as Guidance Example 4.

[0061] <Guidance example 3> The output control unit 14 outputs, as guidance information, information for moving the measurer, which is necessary to change the orientation of the face image so that the index value satisfies the measurable condition.

[0062] Here, the output control unit 14 displays and outputs information for moving the subject, and the information for moving the subject may include a graphic indicating the direction of movement.

[0063] Specifically, the image processing unit 12 may identify a proximate area closer to the image capturing device and a proper area having brightness close to the optimal index value from among the predetermined symmetrical areas of the subject in the facial image. If the proximate area and the proper area do not match, the output control unit 14 may output information on the movement required to move the subject so that the predetermined area identified as the proper area becomes the proximate area.

[0064] The predetermined symmetrical parts of the subject are the right and left cheeks of the subject, and when the adjacent part is the left cheek and the appropriate part is the right cheek, the output control unit 14 may output information to move the subject, guiding the subject to take the photograph from the right side, and when the adjacent part is the right cheek and the appropriate part is the left cheek, the output control unit 14 may output information to move the subject, guiding the subject to take the photograph from the left side.

[0065] FIG. 5 is a flowchart of guidance example 3 in the terminal device 100. In guidance example 3, guidance is given to move the subject to measure whichever is more suitable for measurement at the symmetrical predetermined parts. Note that, although FIG. 5 also shows cheeks as an example of the symmetrical predetermined parts, parts other than cheeks may also be used. The subject of each step in the flowchart of FIG. 5 will be explained as the control unit 1.

[0066] The processes of S11, S12, S14, and S16 in FIG. 5 are the same as those of S11, S12, S14, and S16 in FIG. 4, so their explanations are omitted. Only the following will be explained.

[0067] 5, if the adjacent part and the appropriate part do not match, the control unit 1 determines NO in S12 and S14 and proceeds to S21. In S21, the control unit 1 (image processing unit 12) determines the center of area of ​​the dark cheek and the center of area of ​​the bright cheek based on the facial image, and determines a straight line connecting the centers.

[0068] Next, the control unit 1 displays an arrow extending in the direction of the straight line obtained in S21 and pointing from the dark cheek to the light cheek as guidance, enlarging it to fit within a predetermined area of ​​the display screen (S22). The arrow displayed in S22 is an example of a graphic indicating the movement direction.

[0069] FIG. 6 shows the state of the person being measured and an example of the guidance screen D4 displayed in S22 in the flowchart of FIG. 5. As shown in FIG. 6, person A is facing his or her right cheek toward the brighter side. In this state, if the person being measured measures from the left side of the person being measured (the right side as seen from the person being measured), the brightness of the left cheek, which is the closest part to the image capturing device 2, will be insufficient. In this case, the right cheek, which is farthest from the image capturing device 2, is the appropriate part. In such a case, a more accurate measurement can be performed by the person being measured moving and measuring from the right side of the person being measured (the left side as seen from the person being measured).

[0070] Therefore, on the guidance screen D4, an arrow figure f1 extending in the direction of the straight line obtained in S21 and pointing from the dark cheek to the light cheek is displayed below the facial image 108 of the person being measured, which is being photographed, as guidance information, together with a message M1 saying "Please move in the direction of the arrow."

[0071] 7 is a diagram showing an example of a measurement start screen D5 after the subject moves in guidance example 3. As shown in FIG. 7, when the subject moves according to the guidance, that is, when sufficient brightness is ensured, the display switches from guidance screen D4 to measurement start screen D5. On measurement start screen D5, a message M2 such as "Measurement will begin" is displayed. Ensuring sufficient brightness corresponds to the proximity area and the appropriate area matching.

[0072] The measurement start screen D5 may be displayed in S6 in the flowchart of Figure 3 where no guidance is provided, and in S16 in the flowcharts of Figures 4 and 5. In other words, the measurement start screen D5 may be displayed between the measurement standby screen D1 and the measurement in progress screen D2.

[0073] <Guidance example 4> The output control unit 14 outputs, as guidance information, information for moving the subject, which is necessary to change the orientation of the face image so that the index value satisfies the measurable condition.

[0074] Specifically, the image processing unit 12 may identify a proximate part closer to the image capturing device and a proper part having brightness close to the proper index value from among the predetermined symmetrical parts of the subject in the facial image. If the proximate part and the proper part do not match, the output control unit 14 may output information on the movement required to move the subject so that the predetermined part identified as the proximate part becomes the proper part.

[0075] The predetermined symmetrical parts of the subject are the right and left cheeks of the subject, and when the nearby part is the left cheek and the appropriate part is the right cheek, the output control unit 14 may output information to turn the subject's face to the left as information to move the subject, and when the nearby part is the right cheek and the appropriate part is the left cheek, the output control unit 14 may output information to turn the subject's face to the right as information to move the subject.

[0076] Furthermore, the authentication processing unit 15 may acquire personal information including at least the name of the person being measured by facial authentication using a captured image, and the output control unit 14 may output information including the name of the person being measured together with the guidance information.

[0077] In this case, the output control unit 14 may output, as guidance information, information for moving the subject, which is necessary to change the orientation of the facial image so that the index value satisfies the measurable conditions, as audio data, and may include the name of the subject in the audio data.

[0078] FIG. 8 is a flowchart of guidance example 4 in the terminal device 100. In guidance example 4, guidance is given to move the subject to measure whichever is more suitable for measurement at the symmetrical predetermined parts. Note that, although FIG. 8 also shows cheeks as an example of the symmetrical predetermined parts, parts other than cheeks may also be used. The subject of each step in the flowchart of FIG. 8 will be explained as the control unit 1.

[0079] As shown in Fig. 8, while the measurement standby screen D1 (see Fig. 2) is displayed, the control unit 1 (authentication processing unit 15) repeatedly determines whether a facial image has been detected, that is, whether the facial image is within the frame image 101 (see Fig. 2) (S31). If the control unit 1 determines YES in S31, it performs facial authentication and acquires personal information about the person being measured (S32). The personal information includes at least the name of the person being measured, and in this embodiment, past measurement results of the person being measured are also acquired.

[0080] Thereafter, the control unit 1 determines whether a facial image is continuously detected (S33). If the control unit 1 (image processing unit 12) determines YES in S33, it adjusts the brightness of the facial image (S34), and then proceeds to S35. Note that if the control unit 1 determines NO in S33, it ends the process.

[0081] In S35, the control unit 1 determines whether measurement is possible after brightness adjustment, that is, whether the index value of the measurement point in the face image satisfies the measurable condition. If the control unit 1 determines YES in S35, it proceeds to S36. In S36, the control unit 1 (measurement processing unit 13) measures the biological information of the person being measured based on the face image, and then ends the process.

[0082] On the other hand, if the control unit 1 determines NO in S35, it uses the name of the person being measured acquired through face authentication to provide guidance for moving the person being measured (S37), and then proceeds to S36.

[0083] In S37, the control unit 1 (output control unit 14) provides voice guidance including, for example, "Mr. / Ms. XX, please turn your face to the left until you hear the "yes" message" as guidance information. Alternatively, the control unit 1 provides display guidance including, for example, "Mr. / Ms. XX, please turn your face to the left until you hear the "yes" message," and has the subject read it out loud. The control unit 1 may provide both voice guidance and display guidance. In this case, the output control unit 14 may display a message such as "Please remain still."

[0084] Fig. 9 is a diagram showing an example of the guidance performed in S37 in the flowchart of Fig. 8. Guidance screen D4 shown in Fig. 9 is displayed when subject A faces their right cheek toward the brighter side and the measurer measures them from the left side (the right side as seen from the measurer), as in Fig. 6. In this case, the brightness of the left cheek, which is closer to the image capture device 2, is insufficient, so by having the subject turn to the left (the right as seen from the measurer), the measurer can perform a correct measurement without moving.

[0085] 9, on the guidance screen D4, a message M3 saying "Please remain still" is displayed as display guidance below the photographed facial image 108 of the person being measured, directed to the person being measured. While the guidance screen D4 is displayed, a message saying "Mr. / Ms. XX, please turn your face to the left until you hear "Yes"" is output as audio guidance to the person being measured.

[0086] Fig. 10 is a diagram showing an example of guidance for stopping the movement of the person being measured and starting measurement in guidance example 4. As shown in Fig. 10, when the person being measured moves according to the guidance and sufficient brightness is ensured (the proximal part and the appropriate part match), a message such as "OK, measurement will be performed. Please do not move" is output as audio guidance, and the person being measured stops moving. Then, the display is switched from guidance screen D4 to the measurement start screen D5 described above.

[0087] <Guidance example 5> When the specification processing unit 17 specifies an item of biological information that will result in low measurement accuracy, the output control unit 14 includes in the guidance information a message that the measurement result will be affected.

[0088] As described above, the identification processing unit 17 refers to the correlation between the index value at the time of measurement and the measurement result, and identifies the item of biological information for which the measurement accuracy will be low if the index value does not satisfy the specified measurable condition.

[0089] Here, the identification processing unit 17 may refer to the correlation between the index value and the measurement result during past measurements of the subject, and identify items of biometric information for which measurement accuracy will be low if the index value does not satisfy specified measurable conditions.

[0090] Alternatively, the identification processing unit 17 may refer to the correlation between the index values ​​and measurement results from past measurements of multiple subjects, and identify items of biometric information for which measurement accuracy will be low if the index values ​​do not satisfy specified measurable conditions.

[0091] FIG. 11 is a flowchart of guidance example 5 in the terminal device 100. In guidance example 5, guidance is given to move the subject to measure whichever is more suitable for measurement at the symmetrical predetermined sites, including a message that this will affect the measurement results. Note that, although FIG. 11 also shows cheeks as an example of the symmetrical predetermined sites, other sites may also be used. The subject of each step in the flowchart of FIG. 11 will be explained as the control unit 1.

[0092] 11 are the same as the processes of S31 to S37 in FIG. 8, so a description thereof will be omitted, and only S38 to S40 carried out between S37 and S36 will be described.

[0093] In S38, the control unit 1 (specification processing unit 17) analyzes past measurement results. Next, the control unit 1 determines whether measurement using the current brightness, i.e., the index value, will affect the measurement results (S39). Here, the control unit 1 refers to the relationship between the index value at the time of measurement and the measurement results, and identifies any items of biological information for which measurement accuracy will be reduced if the index value does not satisfy a predetermined measurable condition.

[0094] If the control unit 1 determines YES in S39, it advances the process to S40, where it provides guidance to assist with the measurement, and then advances to S36.

[0095] For example, if the item of biological information for which measurement accuracy is reduced as identified in S39 is "vascular information," the control unit 1 may provide voice guidance in S40, including guidance information such as, "If you continue to take an image, the vascular information may not be measured correctly." Alternatively, the control unit 1 may provide display guidance including guidance information with the same content as the voice guidance, and have the subject read it out loud. The control unit 1 may provide both voice guidance and display guidance.

[0096] [Software implementation example] The functions of the terminal device 100 (hereinafter referred to as the "device") can be realized by a program that causes a computer to function as the device, and a program that causes a computer to function as each control block of the device (particularly each part included in the control unit 1).

[0097] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program, thereby realizing the functions described in each of the above embodiments.

[0098] The program may be non-transitory and may be recorded on one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0099] Furthermore, some or all of the functions of the control blocks can be realized by logic circuits. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of the control blocks can also be realized by, for example, a quantum computer.

[0100] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may run on the control device or on another device (for example, an edge computer or a cloud server).

[0101] 〔summary〕 A terminal device according to aspect 1 of the present disclosure comprises an image acquisition unit that acquires an image of a person being measured captured by an imaging device, an image processing unit that calculates an index value indicating the brightness of the facial image of the person being measured in the captured image, an output control unit that outputs biometric information of the person being measured measured based on the facial image, and a judgment unit that judges whether the index value satisfies a predetermined measurable condition, and if the index value does not satisfy the measurable condition, the output control unit outputs guidance information for changing the orientation of the facial image so that the index value satisfies the measurable condition.

[0102] A terminal device according to aspect 2 of the present disclosure may be configured such that, in aspect 1 above, the image processing unit calculates the index values ​​for the chin, forehead, right cheek, and left cheek of the subject's face, and the output control unit outputs, as the guidance information, information for changing the orientation of the facial image upward when the index value for the chin does not satisfy the measurable condition and the index value for the forehead does satisfy the measurable condition, or when the index value other than the forehead does not satisfy the measurable condition.

[0103] A terminal device according to aspect 3 of the present disclosure may be configured such that, in aspect 1 or 2 above, the image processing unit identifies a nearby area that is closer to the photographing device and a suitable area that has brightness close to the suitable index value, relative to the symmetrical predetermined areas of the subject in the facial image, and the output control unit outputs, as the guidance information, information for changing the orientation of the facial image so that the nearby area and the suitable area match, if the nearby area and the suitable area do not match.

[0104] A terminal device according to aspect 4 of the present disclosure may be configured such that, in aspect 1 or 3 above, the output control unit outputs, as the guidance information, information for moving the person being measured, which is necessary to change the orientation of the facial image so that the index value satisfies the measurable condition.

[0105] A terminal device according to aspect 5 of the present disclosure may be configured in the above-mentioned aspect 4 such that, for the predetermined symmetrical parts of the subject in the facial image, the image processing unit identifies a proximate part closer to the photographing device and a proper part having brightness close to the optimal index value, and the output control unit outputs, when the proximate part and the proper part do not match, information on the movement required to move the subject to make the predetermined part identified as the proper part the proximate part.

[0106] A terminal device according to aspect 6 of the present disclosure may be configured such that, in aspect 5 above, the predetermined symmetrical parts of the person being measured are the right and left cheeks of the person being measured, and the output control unit outputs information to guide the person being measured to take a photograph from the right side when the adjacent part is the left cheek and the appropriate part is the right cheek, as information to move the person being measured, and outputs information to guide the person being measured to take a photograph from the left side when the adjacent part is the right cheek and the appropriate part is the left cheek, as information to move the person being measured.

[0107] A terminal device according to aspect 7 of the present disclosure may be configured such that, in aspect 4, 5, or 6 above, the output control unit outputs information for moving the subject to the display device, and the information for moving the subject includes a graphic indicating the direction of movement.

[0108] A terminal device according to aspect 8 of the present disclosure may be configured such that, in aspect 1, 2 or 3 above, the output control unit outputs, as the guidance information, information for moving the subject, which is necessary to change the orientation of the facial image so that the index value satisfies the measurable condition.

[0109] A terminal device according to aspect 9 of the present disclosure may be configured such that, in aspect 8 above, the image processing unit identifies a nearby area closer to the photographing device and a suitable area having a brightness close to the suitable index value from among the predetermined symmetrical areas of the subject in the facial image, and the output control unit outputs, when the nearby area and the suitable area do not match, information on the movement required to make the predetermined area identified as the nearby area the suitable area, as information for moving the subject.

[0110] A terminal device according to aspect 10 of the present disclosure may be configured such that, in aspect 9 above, the predetermined symmetrical parts of the person being measured are the right and left cheeks of the person being measured, and the output control unit outputs information to turn the face of the person being measured to the left as information to move the person being measured when the nearby part is the left cheek and the appropriate part is the right cheek, and outputs information to turn the face of the person being measured to the right as information to move the person being measured when the nearby part is the right cheek and the appropriate part is the left cheek.

[0111] The terminal device according to aspect 11 of the present disclosure, in any of aspects 1 to 10 above, may further include a personal information acquisition unit that acquires personal information including at least the name of the person being measured by facial recognition using the captured image, and the output control unit may be configured to output information including the name of the person being measured together with the guidance information.

[0112] A terminal device according to aspect 12 of the present disclosure may be configured such that, in aspect 11 above, the output control unit outputs, as the guidance information, information for moving the subject, which is necessary to change the orientation of the facial image so that the index value satisfies the measurable condition, as audio data, and the audio data includes the name of the subject.

[0113] A terminal device according to aspect 13 of the present disclosure, in any of aspects 1 to 12 above, may include an identification unit that identifies an item of biometric information for which measurement accuracy will be low if the index value does not satisfy a predetermined measurable condition by referring to the correlation between the index value at the time of measurement and past measurement results, and the output control unit may be configured to include content in the guidance information informing the user that the measurement results will be affected when an item of biometric information for which measurement accuracy will be low is identified.

[0114] The terminal device according to aspect 14 of the present disclosure may further include, in the above-mentioned aspect 13, a personal information acquisition unit that acquires personal information of the subject by facial recognition using the captured image, and the identification unit may be configured to identify items of biometric information for which measurement accuracy will be low if the index value does not satisfy a specified measurable condition, by referring to the correlation between the index value and the measurement result at the time of past measurement of the subject.

[0115] A terminal device according to aspect 15 of the present disclosure may be configured such that, in aspect 13 above, the identification unit refers to the correlation between the index value and the measurement results when measuring multiple subjects in the past, and identifies items of biometric information for which measurement accuracy will be low if the index value does not satisfy specified measurable conditions.

[0116] A measurement method according to aspect 16 of the present disclosure is a measurement method using a terminal device equipped with an imaging device, and includes an acquisition step of acquiring an image of a person being measured captured by the imaging device, an image processing step of calculating an index value indicating the brightness of the facial image of the person being measured in the captured image, a biometric information output step of outputting biometric information of the person being measured measured based on the facial image, a determination step of determining whether the index value satisfies predetermined measurable conditions, and a guidance information output step of outputting guidance information for changing the orientation of the facial image so that the index value satisfies the measurable conditions if the index value does not satisfy the measurable conditions.

[0117] The terminal device according to each aspect of the present invention may be realized by a computer. In this case, the control program of the terminal device that realizes the terminal device on a computer by causing the computer to operate as each part (software element) of the terminal device, and the computer-readable recording medium on which the program is recorded, also fall within the scope of the present invention.

[0118] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. Furthermore, new technical features can be formed by combining the technical means disclosed in each embodiment. [Explanation of symbols]

[0119] 2. Imaging equipment 5. Communications equipment 6 Storage device 11 Image acquisition unit 12 Image processing section 14 Output control section 15 Authentication processing section 13 Measurement processing section 16 Judgment processing unit (judgment unit) 17 Specific Processing Unit (Specific Unit) 100 Terminal Device D1 Measurement standby screen D2 Measuring screen D3 Measurement result screen D4 Guidance Screen D5 Measurement start screen f1 shape M1, M2, M3 text messages

Claims

1. an image acquisition unit that acquires an image of the subject captured by an imaging device; an image processing unit that calculates an index value indicating the brightness of the facial image of the subject in the captured image; an output control unit that outputs biological information of the subject measured based on the face image; a determination unit that determines whether the index value satisfies a predetermined measurable condition, The output control unit outputs guidance information for changing the orientation of the facial image so that the index value satisfies the measurable condition when the index value does not satisfy the measurable condition.

2. the image processing unit calculates the index values ​​for the chin, forehead, right cheek, and left cheek of the face of the subject; 2. The terminal device according to claim 1, wherein the output control unit outputs, as the guidance information, information for changing the orientation of the facial image upward when the index value of the chin does not satisfy the measurable condition and the index value of the forehead satisfies the measurable condition, or when the index value other than the forehead does not satisfy the measurable condition.

3. the image processing unit identifies a proximate portion closer to the photographing device and an appropriate portion having brightness close to the appropriate index value from among predetermined symmetrical portions of the subject in the face image, The terminal device according to claim 1, wherein, when the adjacent part and the appropriate part do not match, the output control unit outputs, as the guidance information, information for changing the orientation of the face image so that the adjacent part and the appropriate part match.

4. The terminal device according to claim 1 , wherein the output control unit outputs, as the guidance information, information for moving the person being measured, which is necessary to change the orientation of the face image so that the index value satisfies the measurable condition.

5. the image processing unit identifies a proximate portion closer to the photographing device and an appropriate portion having brightness close to an optimal index value from among predetermined symmetrical portions of the subject in the facial image, The terminal device according to claim 4, wherein the output control unit outputs, when the proximal part and the appropriate part do not match, information on the movement required to move the subject so that a specific part identified as the appropriate part becomes the proximal part, as information for moving the subject.

6. the predetermined symmetrical parts of the subject are the right cheek and the left cheek of the subject, 6. The terminal device according to claim 5, wherein the output control unit outputs, as information for moving the person being measured, information guiding the person to take the photograph from the right side of the person being measured when the adjacent part is the left cheek and the appropriate part is the right cheek, and outputs, as information for moving the person being measured, information guiding the person to take the photograph from the left side of the person being measured when the adjacent part is the right cheek and the appropriate part is the left cheek.

7. The terminal device according to claim 4 , wherein the output control unit causes a display device to output information for moving the subject, and the information for moving the subject includes a graphic indicating a direction of movement.

8. The terminal device according to claim 1, wherein the output control unit outputs, as the guidance information, information for moving the subject that is necessary to change the orientation of the face image so that the index value satisfies the measurable condition.

9. the image processing unit identifies a proximate portion closer to the photographing device and an appropriate portion having brightness close to the appropriate index value from among predetermined symmetrical portions of the subject in the face image, The terminal device according to claim 8, wherein the output control unit outputs, when the proximal part and the appropriate part do not match, information on the movement required to move the subject so that a specific part identified as the proximal part becomes the appropriate part.

10. the predetermined symmetrical parts of the subject are the right cheek and the left cheek of the subject, The terminal device according to claim 9, wherein the output control unit outputs information for turning the face of the person being measured to the left as information for moving the person being measured when the adjacent part is the left cheek and the appropriate part is the right cheek, and outputs information for turning the face of the person being measured to the right as information for moving the person being measured when the adjacent part is the right cheek and the appropriate part is the left cheek.

11. a personal information acquisition unit that acquires personal information including at least the name of the subject by face authentication using the captured image; The terminal device according to claim 1 , wherein the output control unit outputs information including the name of the person being measured together with the guidance information.

12. The terminal device according to claim 11, wherein the output control unit outputs, as the guidance information, information for moving the subject, which is necessary to change the orientation of the facial image so that the index value satisfies the measurable condition, as audio data, and includes the name of the subject in the audio data.

13. a specifying unit that specifies an item of biological information for which measurement accuracy will be low when the index value does not satisfy a predetermined measurable condition by referring to the correlation between the index value at the time of measurement and past measurement results, The terminal device according to claim 1 , wherein, when an item of biological information for which the measurement accuracy is reduced is identified, the output control unit includes, in the guidance information, content informing the user that the measurement result will be affected.

14. a personal information acquisition unit that acquires personal information of the subject by face authentication using the captured image, The terminal device according to claim 13, wherein the identification unit refers to the correlation between the index value and the measurement result at the time of past measurements of the subject, and identifies an item of biological information for which measurement accuracy will be low if the index value does not satisfy a predetermined measurable condition.

15. The terminal device according to claim 13, wherein the identification unit refers to the correlation between the index values ​​and measurement results from past measurements of multiple subjects, and identifies an item of biological information for which measurement accuracy will be low if the index value does not satisfy a predetermined measurable condition.

16. A measurement method using a terminal device equipped with an imaging device, an acquiring step of acquiring an image of the subject photographed by the photographing device; an image processing step of calculating an index value indicating the brightness of the face image of the subject in the captured image; a biometric information output step of outputting biometric information of the subject measured based on the face image; a determining step of determining whether the index value satisfies a predetermined measurable condition; and a guidance information output step of outputting, when the index value does not satisfy the measurable condition, guidance information for changing the orientation of the face image so that the index value satisfies the measurable condition.

Images