Biological information measuring device, biological information measuring method, and biological information measuring program

Abstract

To provide a biological information measurement device which can enhance accuracy of measurement of biological information.SOLUTION: A biological information measurement device comprises: an initial setting unit which displays a first image obtained by imaging a living body by an imaging unit and a first diagram indicating a position in the first image in a superimposed manner on a display unit, and obtains a corrected position obtained by correcting the position on the basis of a first instruction that is input to an input unit and instructs that the position is moved to a first position of a feature point included in the first image; a measurement preparation unit which displays a second image obtained by imaging the living body by the imaging unit and a second diagram indicating the corrected position in a superimposed manner on the display unit, and causes the living body to align the second position of the feature point included in the second image with the corrected position; and a measurement unit which measures biological information of the living body from a third image obtained by imaging the living body by the imaging unit in response to the input of a signal indicating that the second position is aligned with the corrected position.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a biological information measurement device, a biological information measurement method, and a biological information measurement program. [Background technology]

[0002] When a biological information measuring device measures biological information from a moving image obtained by capturing an image of a face, the accuracy of the measurement may be improved by measuring the measurement information from a moving image within a region of interest (ROI) that strongly reflects the biological information to be measured, rather than measuring the biological information from the entire moving image. For example, when a biological information measuring device measures blood vessel characteristics such as pulse, hemoglobin concentration, and blood pressure, the accuracy of the measurement may be improved by measuring the blood vessel characteristics from a moving image within an ROI that strongly reflects the blood vessel characteristics to be measured.

[0003] FIG. 15A is a diagram illustrating an example of a state in which an appropriate ROI is set within a face region.

[0004] When detecting the position of an ROI in a moving image obtained by capturing an image of a face, the biological information measurement device executes, for example, the following steps.

[0005] 15A, the biological information measuring device displays a moving image 801 of a face and a guide 802 superimposed on a display or the like. The user of the biological information measuring device aligns the position of the outline of a face area 803 with the position of the guide 802 while referring to the display or the like. This enables the biological information measuring device to capture an image of the face.

[0006] 15A, the biological information measuring device uses a face detection algorithm to detect the positions of specific facial features within the guide 802 from the obtained moving image 801 as feature points 804. The specific features include the eyes, nose, and mouth.

[0007] Next, as shown in FIG. 15A, the biological information measurement device calculates the position of the ROI 805 so that the relative position of the ROI 805 with respect to the detected feature point 804 is an appropriate relative position.

[0008] Patent Document 1 discloses a photography assist device that allows a subject to recognize a suitable photography position. In this photography assist device, feature points on the face in a photographed image are extracted. Reference points are set at positions where the feature points exist in an average face model. Image information is output in which a current position display line connecting the extracted feature points and a reference position display line connecting the set reference point are superimposed on a plane. This allows the subject to easily recognize a suitable facial position and orientation for photography, and to adjust the facial orientation by moving their face to the recognized facial position. This allows for highly accurate frontal photography of the subject (paragraphs 0010, 0056, 0059, 0061, and 0073). [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-219645 Summary of the Invention [Problem to be solved by the invention]

[0010] It is desirable for the biological information measuring device to calculate the position of ROI 805 so that the position of ROI 805 within face region 803 does not shift each time biological information is measured. This is because the position of ROI 805 must be in an area that strongly reflects biological information such as blood vessel characteristics to be measured, and therefore if the reproducibility of the position of ROI 805 is low, the accuracy of measurement will be low.

[0011] Ideally, it is desirable for the biometric information measurement device to detect feature points 804 at the same positions within the face region 803 and set ROI 805 at the same position within the face region 803, regardless of the facial individuality and the timing at which the biometric information is measured. It is also desirable that the positions of feature points 804 and ROI 805 do not shift while the biometric information is being measured. However, this is difficult to achieve.

[0012] The reason for this will be further explained.

[0013] 15B, 15C, and 15D are diagrams illustrating an example of a state in which an inappropriate ROI is set in a face region.

[0014] It is difficult to ensure that the positions at which feature points 804 are detected do not deviate at all each time the feature points 804 are detected. For this reason, even when feature points 804 are detected from a video 801 of the face of the same person, the positions of the detected feature points 804 within the face region 803 may differ each time the feature points 804 are detected. In addition, if the facial features have changed due to a change in hairstyle, dieting, aging, or the like, the detection of feature points 804 may not be successful. In addition, detection algorithms for detecting feature points 804 have strengths and weaknesses. For this reason, there are people with facial features for which feature points 804 are easily detected and people with facial features for which feature points 804 are difficult to detect.

[0015] When the detected feature points 804 are used for normal face recognition or detection, a slight deviation of the positions of the detected feature points 804 from the actual positions of the feature points does not pose a significant problem. Therefore, deviation of the positions of the detected feature points 804 from the actual positions of the feature points has not conventionally been a significant problem. However, when the detected feature points 804 are used to measure biometric information, the positions of the detected feature points 804 must be highly accurate. For example, when the detected feature points 804 are used to measure blood vessel characteristics, it is desirable that the deviation of the position of the set ROI 805 from the position of the actual blood vessel image be as small as possible, because facial blood vessels have a fine and complex distribution. However, as shown in FIG. 15B , even if the position of one of the detected feature points 804 is shifted to the right from the actual position of the feature points, the position of the set ROI 805 in the face image 801 is shifted, and the position of the set ROI 805 is shifted from the position of the actual blood vessel image.

[0016] Furthermore, in order to prevent the positions of the feature points 804 and ROI 805 from shifting from their appropriate positions while the biometric information is being measured, face detection is also performed by making the bounding box follow the face region 803. However, if the face region 803 moves quickly, it is difficult to perform face detection in real time by making the bounding box follow the face region 803 perfectly. For this reason, even if the face video 801 is a video of the face of the same person, as shown in Fig. 15C, the feature points 804 and ROI 805 may not be able to follow the face region 803, resulting in the feature points 804 and ROI 805 being shifted from their appropriate positions.

[0017] Furthermore, the shape and size of the face, as well as the positions and shapes of the eyes, nose, and mouth, differ from person to person. Therefore, the relative position of the ROI 805 with respect to the feature points 804 also differs from person to person. For this reason, as illustrated in Fig. 15D, the ROI 805 may overlap undesirable parts such as the outline of the face region 803 and the eye region 806. Alternatively, when it is desired that the ROI 805 be set to avoid obstacle regions such as glasses, a mask, hair, and uneven areas of the face, the position of the ROI 805 may deviate from the appropriate position, causing the ROI 805 to overlap the obstacle regions.

[0018] Therefore, these factors may cause a decrease in the accuracy of measurement of biological information.

[0019] In view of this problem, an aspect of the present disclosure provides a biological information measurement device that can improve the accuracy of biological information measurement, for example. [Means for solving the problem]

[0020] A biometric information measuring device according to one embodiment of the present disclosure comprises an imaging unit, a display unit, an input unit, an initial setting unit that causes a first image obtained by imaging a living body using the imaging unit and a first graphic indicating a position within the first image to be superimposed on the display unit, and that corrects the position based on a first instruction input to the input unit that instructs the position to be moved to a first position of a feature point included in the first image to obtain a corrected position, a measurement preparation unit that causes a second image obtained by imaging the living body using the imaging unit and a second graphic indicating the corrected position to be superimposed on the display unit, and that causes the living body to align the second position of the feature point included in the second image with the corrected position, and a measurement unit that measures biometric information of the living body from a third image obtained by imaging the living body using the imaging unit in response to input of a signal indicating that the second position has been aligned with the corrected position.

[0021] Another aspect of the biometric information measurement method of the present disclosure includes the steps of: (a) superimposing and displaying a first image obtained by imaging a living organism on a first graphic indicating a position within the first image, and correcting the position based on a first instruction instructing the living organism to move the position to a first position of a feature point included in the first image to obtain a corrected position; (b) superimposing and displaying a second image obtained by imaging the living organism on a second graphic indicating the corrected position, and having the living organism align the second position of the feature point included in the second image with the corrected position; and (c) measuring the biometric information of the living organism from a third image obtained by imaging the living organism in response to input of a signal indicating that the second position has been aligned with the corrected position.

[0022] Another aspect of the biometric information measurement program of the present disclosure causes a computer to execute the following steps: (a) superimposing and displaying a first image obtained by imaging a living body on a first graphic indicating a position within the first image, and correcting the position based on a first instruction instructing the user to move the position to a first position of a feature point included in the first image to obtain a corrected position; (b) superimposing and displaying a second image obtained by imaging the living body on a second graphic indicating the corrected position, and having the living body align the second position of the feature point included in the second image with the corrected position; and (c) measuring the biometric information of the living body from a third image obtained by imaging the living body in response to input of a signal indicating that the second position has been aligned with the corrected position. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a block diagram of a biological information measuring device according to a first embodiment. [Figure 2] FIG. 2 is a diagram schematically illustrating an image obtained by the biological information measuring device of the first embodiment, registered feature points registered in the biological information measuring device, and a bounding box and region of interest (ROI) set by the biological information measuring device. [Figure 3]2 is a block diagram of a control unit provided in the biological information measuring device of the first embodiment. FIG. [Figure 4] 4 is a flowchart showing a processing flow when the biological information measurement device of the first embodiment performs initial setting. [Figure 5] 3A and 3B are diagrams illustrating examples of screens displayed when the biological information measurement device of the first embodiment performs initial settings. [Figure 6] 3A and 3B are diagrams illustrating examples of screens displayed when the biological information measurement device of the first embodiment performs initial settings. [Figure 7] 4 is a flowchart showing a processing flow when the biological information measurement device of the first embodiment prepares for and performs measurement. [Figure 8] 3A and 3B are diagrams illustrating examples of screens displayed when the biological information measurement device of the first embodiment prepares for measurement. [Figure 9] 3A and 3B are diagrams illustrating examples of screens displayed when the biological information measurement device of the first embodiment prepares for measurement. [Figure 10] 10A and 10B are diagrams schematically illustrating an image obtained by a biological information measurement device according to a first modified example of the first embodiment, and registered feature points registered in the biological information measurement device. [Figure 11] 10A and 10B are diagrams illustrating examples of screens displayed when the biological information measurement device of the second embodiment performs initial settings. [Figure 12] 10A and 10B are diagrams illustrating examples of screens displayed when the biological information measurement device of the second embodiment performs initial settings. [Figure 13] FIG. 11 is a diagram schematically illustrating an example of a screen displayed when the biological information measurement device of the third embodiment performs initial settings. [Figure 14] 13A and 13B are diagrams illustrating examples of screens displayed when a biological information measurement device according to a fourth embodiment prepares for measurement. [Figure 15A] FIG. 10 is a diagram illustrating an example of a state in which an appropriate ROI is set within a face region. [Figure 15B] 10A and 10B are diagrams illustrating an example of a state in which an inappropriate ROI is set in a face area. [Figure 15C]10A and 10B are diagrams illustrating an example of a state in which an inappropriate ROI is set in a face area. [Figure 15D] 10A and 10B are diagrams illustrating an example of a state in which an inappropriate ROI is set in a face area. DETAILED DESCRIPTION OF THE INVENTION

[0024] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the drawings, the same or equivalent elements are designated by the same reference numerals, and redundant description will be omitted.

[0025] 1. First embodiment 1.1 Overview of the biological information measurement device FIG. 1 is a block diagram of a biological information measuring device according to a first embodiment.

[0026] 1 estimates the positions of feature points included in a first image 21 obtained by capturing an image of a living body 901, and performs initial setting to have a user 902 correct the positions of the estimated feature points. The biological information measuring device 1 also performs measurement preparation to have the user 902 align the positions of feature points included in a second image 22 obtained by capturing an image of the living body 901 with the corrected positions of the feature points. The biological information measuring device 1 also measures biological information 31 of the living body 901 from a third image 23 obtained by capturing an image of the living body 901.

[0027] The living body 901 to be imaged is a human body, but the living body 901 to be imaged may also be the body of an animal.

[0028] The biological information measuring device 1 captures an image of the face 903. However, the biological information measuring device 1 may capture an image of a part other than the face 903. For example, the biological information measuring device 1 may capture an image of the neck, wrist, palm, etc.

[0029] The user 902 is a person having the biological body 901 to be imaged. However, the user 902 may be a person other than a person or an animal having the biological body 901 to be imaged. For example, if the biological body 901 to be imaged is the body of an infant, the user 902 may be the infant's parent. Also, if the biological body 901 to be imaged is the body of an animal, the user 902 may be the animal's owner.

[0030] 1.2 Configuration of the biological information measurement device As shown in FIG. 1, the biological information measuring device 1 includes a control unit 11, an imaging unit 12, a display unit 13, an input unit 14, an output unit 15, an audio input unit 16, a memory unit 17, a detection unit 18, and a measurement unit 19.

[0031] The control unit 11, the imaging unit 12, the display unit 13, the input unit 14, the output unit 15, the audio input unit 16, the storage unit 17, the detection unit 18, and the measurement unit 19 are electrically connected to one another, thereby allowing the control unit 11, the imaging unit 12, the display unit 13, the input unit 14, the output unit 15, the audio input unit 16, the storage unit 17, the detection unit 18, and the measurement unit 19 to exchange information with one another.

[0032] The control unit 11 controls the entire biometric information measuring device 1, and controls the imaging unit 12, display unit 13, input unit 14, output unit 15, audio input unit 16, memory unit 17, detection unit 18 and measurement unit 19 provided in the biometric information measuring device 1.

[0033] The control unit 11 is a processor, etc. The processor is a central processing unit (CPU), a graphic processing unit (GPU), etc.

[0034] The imaging unit 12 captures an image of the living body 901, obtains an image, and transmits the obtained image to the control unit 11. The obtained image is a moving image. The image obtained when initial setting is performed becomes the first image 21. The image obtained when measurement preparation is performed becomes the second image 22. The image obtained when measurement is performed becomes the third image 23.

[0035] The image capturing unit 12 is installed in a location where it can capture an image of the face 903. The location in question may be a personal computer, a monitor, a mirror, a washstand, or the like.

[0036] The imaging unit 12 is a camera or the like. The camera includes a lens and an image sensor. The lens focuses light coming from the living body 901 to be imaged on the image sensor. The image sensor photoelectrically converts the focused light into an image signal. The image sensor is a complementary metal-oxide semiconductor (CMOS) image sensor, a charge-coupled device (CCD) image sensor, or the like.

[0037] The display unit 13 displays the obtained image. The displayed image is a moving image during image capture. The display unit 13 also displays various information. The displayed information includes a graphic showing the position of a feature point included in the image, a graphic showing a region of interest (ROI) set within the image, measured biological information 31, a notice to the user 902, date and time, etc. The biological information 31 includes vital signs, a graph showing changes in vital signs over time, etc.

[0038] The display unit 13 is a display or the like.

[0039] Information is input to the input unit 14 by the user 902. The user 902 can give instructions to the biological information measuring device 1 by inputting information indicating the content of the instruction to the input unit 14.

[0040] The input unit 14 is a keyboard, a mouse, a touch panel, a button, a dial, a slider, or the like.

[0041] The output unit 15 outputs information to the user 902. The output information is a notice or the like to the user 902. The output unit 15 outputs the information by characters, voice, or the like.

[0042] The output unit 15 is a display, a speaker, or the like.

[0043] A user 902 inputs voice into the voice input unit 16. The user 902 can give a cue to the biological information measuring device 1 by inputting voice into the voice input unit 16.

[0044] The voice input unit 16 is a microphone or the like.

[0045] The storage unit 17 stores the obtained images. The storage unit 17 also stores various types of information. The stored information includes information that the manufacturer stores in advance in the biological information measurement device 1, information that the user 902 stores later in the biological information measurement device 1, and the like. The former information includes a face detection algorithm, a measurement program, and the like. The latter information includes the transition of the measured daily biological information 31, data related to correction, which will be described later, and the like.

[0046] The storage unit 17 is a random access memory (RAM), a read only memory (ROM), a hard disk drive, a solid state drive, or the like.

[0047] The measurement unit 19 measures biometric information 31 of the living body 901 from the obtained third image 23. The measurement unit 19 measures the biometric information 31 from the image within the ROI included in the third image 23. The measurement unit 19 measures the biometric information 31 from, for example, the brightness of the image within the ROI, changes in the brightness over time, etc. The measured biometric information 31 includes the state of blood vessels, blood pressure, blood flow, heart rate, etc. The measurement unit 19 measures the biometric information 31 from the moving image, thereby making it possible to measure changes in vital signs over time.

[0048] FIG. 2 is a diagram schematically illustrating an image obtained by the biological information measurement device of the first embodiment, registered feature points registered in the biological information measurement device, and a bounding box and ROI set by the biological information measurement device.

[0049] While the biometric information 31 is being measured, the detection unit 18 makes the ROI 62 shown in Fig. 2 track the face region 61 included in the obtained image shown in Fig. 2, and makes the movement of the ROI 62 match the movement of the face region 61. The method of making the ROI 62 track the face region 61 differs depending on the algorithm. The algorithm is selected from among algorithms suitable for the biometric information 31 to be measured.

[0050] In the first embodiment, the detection unit 18 sets a bounding box 64 shown in FIG. 2 based on the registered feature points 63 shown in FIG. 2 , and sets an ROI 62 based on the registered feature points 63 and the bounding box 64. The detection unit 18 determines the initial position of the bounding box 64 based on the positions of the registered feature points 63, and determines the initial position of the ROI 62 based on the positions of the registered feature points 63 and the initial position of the bounding box 64. For example, if the registered feature points 63 are at the left and right ends of the eye, nose, and mouth regions, the detection unit 18 sets the bounding box 64 to a range that is likely to contain a face region 61, which is estimated from the registered feature points 63, and sets the ROI 62 to a range that is likely to contain a cheek region, which is estimated from the registered feature points 63. The detection unit 18 also holds the relative positions of the registered feature points 63, the bounding box 64, and the ROI 62 while the biometric information 31 is being measured. The detection unit 18 also causes the bounding box 64 to track the face region 61. The detection unit 18 detects image features within the bounding box 64, such as edges (contours), detects movement of the detected image features, and moves the bounding box 64 in accordance with the movement of the detected image features, thereby making the bounding box 64 follow the face region 61. In this way, the detection unit 18 makes the ROI 62 follow the face region 61. This solves the problem shown in Fig. 15C where the feature points 804 and ROI 805 cannot be made to follow the face region 803, resulting in the feature points 804 and ROI 805 being shifted from their appropriate positions.

[0051] The control unit 11, the detection unit 18, and the measurement unit 19 are configured by causing a computer incorporated in the biological information measurement device 1 to execute a biological information measurement program 41 including a face detection algorithm, a measurement program, etc. All or part of the control unit 11, the detection unit 18, and the measurement unit 19 may be configured by dedicated circuits.

[0052] The biological information measurement program 41 is stored in the storage unit 17. The biological information measurement program 41 may be installed in the computer by writing the biological information measurement program 41 received via the network 911 to the storage unit 17, or by reading the biological information measurement program 41 from an external storage medium 912 such as an optical disk, a magnetic disk, or a Universal Serial Bus (USB) memory and writing it to the storage unit 17. The present disclosure is also directed to the biological information measurement program 41 stored in a computer-readable non-transitory storage medium such as the storage unit 17 or an external storage medium.

[0053] 1.3 Control Unit FIG. 3 is a block diagram of a control unit provided in the biological information measuring device of the first embodiment.

[0054] As shown in FIG. 3, the control unit 11 includes an initial setting unit 51 and a measurement preparation unit 52.

[0055] The initial setting unit 51 performs initial settings. Each time a measurement is performed after the initial settings have been performed, the measurement preparation unit 52 performs measurement preparation in accordance with the initial settings before the measurement is performed.

[0056] 1.4 Initial Setup Fig. 4 is a flowchart showing a processing flow when the biological information measurement device of the first embodiment performs initial setting. Fig. 5 and Fig. 6 are diagrams schematically showing examples of screens displayed when the biological information measurement device of the first embodiment performs initial setting.

[0057] The biological information measuring device 1 starts the initial setting in response to the user 902 inputting an instruction to start the initial setting into the input unit 14. After starting the initial setting, the biological information measuring device 1 executes steps S61 to S67 shown in FIG.

[0058] In step S61, the user 902 inputs identification information 32 for identifying the biometric information 901 to be captured into the input unit 14. This allows the initial setting unit 51 to perform initial setting for each biometric information 901. This allows a log of daily biometric information 31 for each biometric information 901 to be accumulated in the database. The input identification information 32 is an identifier (ID), a name, etc. The identification information 32 for identifying the biometric information 901 may be a recognition result obtained by performing face recognition on the first image 21.

[0059] In the following step S62, the image capturing unit 12 captures an image of the face 903 to obtain the first image 21.

[0060] As shown in FIG. 5, the initial setting unit 51 displays the first image 21 and the guide 72 superimposed on the display unit 13, and the user 902 moves the face 903 so that the outline of the face area 81 roughly matches the guide 72.

[0061] The initial setting unit 51 may cause the display unit 13 to display information useful for positioning, such as a grid and the coordinates of the face area 81, superimposed on the first image 21.

[0062] In the next step S63, the detection unit 18 detects feature points included in the first image 21 and estimates the estimated positions of the detected feature points. The detected feature points are six points consisting of the left and right ends of the eye, nose, and mouth regions. However, the detected feature points are determined by an algorithm for detecting feature points and the measured biometric information 31, and are suitably set by the manufacturer. Therefore, the detected feature points may be points other than the six points consisting of the left and right ends of the eye, nose, and mouth regions. The detected feature points may also be two to five or seven or more points. Note that the greater the number of detected feature points, the higher the accuracy of measuring the biometric information 31. When the biometric information measuring device 1 captures an image of the neck, wrist, or palm, the detected feature points are blood vessel regions. The estimated positions of the detected feature points may deviate from the exact positions of the feature points.

[0063] In the following step S64, as shown in FIG. 5, the initial setting unit 51 causes the first image 21, the white star 71, and the guide 72 to be displayed on the display unit 13 in an overlapping manner.

[0064] The open star 71 indicates a position within the first image 21. The initial setting unit 51 sets the estimated position of the estimated feature point to the position indicated by the open star 71 and places the open star 71 at the estimated position of the estimated feature point. The open star 71 is merely an example of a first graphic indicating a position within the first image 21. Therefore, the open star 71 may be replaced with another graphic. For example, the open star 71 may be replaced with a dot placed at the estimated position of the feature point, a wireframe consisting of lines connecting the estimated positions of the feature points, or the like. As described above, the estimated position of the feature point may be deviated from the exact position of the feature point. Therefore, as illustrated in FIG. 5 , the position indicated by the open star 71, i.e., the position where the open star 71 is placed, may be deviated from the exact position of the feature point. The position indicated by the open star 71 may be a position determined without using an algorithm for detecting feature points.

[0065] In the next step S65, the user 902 refers to the first image 21 and the outlined star 71 displayed superimposed on each other, and inputs a first instruction 33 to the input unit 14 to instruct the user to move the position indicated by the outlined star 71, i.e., the position where the outlined star 71 is to be placed, to a first position that is the exact position of the feature point included in the first image 21. The first instruction 33 is input by operating a cursor key to move the outlined star 71 in a direction corresponding to the operated cursor key, or by operating a pointing device such as a mouse or a touch panel to drag the outlined star 71, or the like.

[0066] The first image 21 displayed overlaid on the white star 71 is a still image. This makes it easy to instruct the user to move the position indicated by the white star 71 to the exact position of the feature point included in the first image 21.

[0067] The initial setting unit 51 corrects the position indicated by the open star 71 to obtain a corrected position based on the input first instruction 33. As a result, as shown in Fig. 6, the position indicated by the open star 71 is corrected to the accurate position of the feature point, and is corrected to a position according to the shape of the face and the positions and sizes of the eyes, nose, and mouth.

[0068] The detection unit 18 sets the bounding box 64 and the ROI 62 based on the corrected position. The detection unit 18 determines the initial range of the bounding box 64 to be set within the third image 23 based on the corrected position, and determines the initial range of the ROI 62 to be set within the third image 23. The ROI 62 is set in the cheek region. However, the region in which the ROI 62 is set is determined by the measured biometric information 31 and is suitably set by the manufacturer. For this reason, the ROI 62 may be set in a region other than the cheek region.

[0069] In the following step S66, the initial setting unit 51 associates and registers the identification information 32, the corrected position, the relative position of the bounding box 64 with respect to the corrected position, the relative position of the ROI 62 with respect to the corrected position, and the first image 21. As a result, when measuring the biometric information 31 of the living body 901 identified by the identification information 32, the corrected position, the relative position of the bounding box 64, the relative position of the ROI 62, and the first image 21 associated with the identification information 32 can be reused. The initial setting unit 51 associates and stores the information to be registered in the memory unit 17, thereby registering the information in association with the information. The corrected position is expressed by the positional relationship between the corrected positions, the coordinates of the corrected position, etc. The point having the corrected position is stored in the memory unit 17 as a registered feature point 63.

[0070] An algorithm for detecting feature points may be corrected. For example, the algorithm for detecting feature points may be corrected so that a specific value is added when estimating the estimated positions of feature points from the first image 21 obtained by capturing an image of a specific living body 901.

[0071] In the following step S67, the user 902 utters the input voice 34 and inputs the input voice 34 to the voice input unit 16. The input voice 34 is a phrase that the user 902 likes, such as "OK!" or "Start!".

[0072] The initial setting unit 51 associates and registers the identification information 32, the input voice 34, and the phrase.

[0073] Before causing the biological information measuring device 1 to measure the biological information 31 for the first time, the user 902 causes the biological information measuring device 1 to perform initial settings. The user 902 also causes the biological information measuring device 1 to perform initial settings when the user's facial features have changed due to a change in hairstyle, aging, or the like. This can solve the problem shown in Fig. 15B in which the feature points 804 cannot be detected appropriately, causing the positions of the feature points 804 to deviate from the actual positions of the feature points, resulting in the position of the set ROI 805 within the face region 803 being shifted.

[0074] 1.5 Measurement preparation and measurement Fig. 7 is a flowchart showing the flow of processing when the biological information measurement device of the first embodiment prepares for measurement and performs measurement. Fig. 8 and Fig. 9 are diagrams schematically showing examples of screens displayed when the biological information measurement device of the first embodiment prepares for measurement.

[0075] The biological information measuring device 1 starts preparation for measurement in response to the user 902 inputting an instruction to start measurement into the input unit 14. Having started preparation for measurement, the biological information measuring device 1 executes steps S101 to S107 shown in FIG.

[0076] In step S101, the user 902 inputs identification information 35 for identifying the living body 901 to be imaged into the input unit 14. This allows the measurement preparation unit 52 to perform measurement preparation appropriate for the living body 901 identified by the input identification information 35. The input identification information 35 may be an ID, a name, or the like. The identification information 35 for identifying the living body 901 may be a recognition result obtained by performing face recognition on the second image 22.

[0077] In the following step S102, the imaging unit 12 captures an image of the face 903 to obtain a second image 22. The obtained second image 22 is a moving image.

[0078] In the following step S103, as shown in FIG. 8, the measurement preparation unit 52 causes the display unit 13 to display the second image 22, the black star 111, and the guide 112 in an overlapping manner.

[0079] The black star 111 indicates the corrected position, i.e., the position of the registered feature point 63. The black star 111 is merely an example of a second graphic that indicates the corrected position. Therefore, the black star 111 may be replaced with another graphic. For example, the black star 111 may be replaced with a wireframe consisting of points placed at the corrected positions, lines connecting the corrected positions, or the like.

[0080] The corrected position indicated by the black star 111 is the corrected position registered in association with the identification information 32 corresponding to the identification information 35 input in step S101. Thus, when identification information 35 corresponding to the identification information 32 input when the initial setting is performed is input to the input unit 14, the measurement preparation unit 52 displays the second image 22 and the black star 111 indicating the corrected position registered in association with the identification information 32 on the display unit 13 in a superimposed manner.

[0081] The user 902 moves the face 903 so that the contour of the face region 121 roughly matches the guide 112 .

[0082] The measurement preparation unit 52 may cause the display unit 13 to display information useful for alignment, such as a grid and facial coordinates, superimposed on the second image 22.

[0083] In the next step S104, the user 902 refers to the second image 22 and the black star 111 displayed overlaid, and aligns the second positions, which are the positions of the feature points included in the second image 22, with the corrected positions indicated by the black star 111. This generally reproduces the state when the ROI 62 and the bounding box 64 were set. At this time, the user 902 moves the face 903 to perform the alignment. Preferably, the user 902 simultaneously aligns the positions of all feature points so that they coincide with the corresponding corrected positions.

[0084] In the subsequent step S105, after the alignment, the user 902, without further moving the face 903, utters a voice 36 corresponding to the input voice 34 registered during the initial setup, and inputs the voice 36 to the voice input unit 16. The input voice 36 is merely an example of a cue indicating that the positions of the feature points included in the second image 22 have been aligned with the corrected positions indicated by the black star marks 111. However, when the voice 36 is the cue, body movements that occur when the cue is input to the biological information measuring device 1 can be suppressed compared to when the cue is an operation performed by moving a hand, such as pressing a button. Therefore, the cue by the voice 36 may be replaced with another cue that can suppress body movements that occur when the cue is input to the biological information measuring device 1. For example, the cue by the voice 36 may be replaced with a cue by eye contact, etc.

[0085] In the next step S106, the measurement preparation unit 52 waits until a set time has elapsed since the input of the voice 36. This time is the time required for the movement of the mouth muscles caused by pronouncing the voice 36 to settle down.

[0086] In the next step S107, the measurement preparation unit 52 causes the measurement unit 19 to start measuring the biological information 31. The measurement unit 19 measures the biological information 31 in response to the input of the voice 36, and measures the biological information 31 after a set time has elapsed since the input of the voice 36.

[0087] The imaging unit 12 captures an image of the face 903 to obtain a third image 23. The obtained third image 23 is a moving image.

[0088] The measurement unit 19 measures the biological information 31 of the living body 901 from the obtained third image 23.

[0089] While the measurement unit 19 is measuring the biometric information 31, the detection unit 18 causes the bounding box 64 to track the face region 61, which is an image of the face 903 included in the third image 23. When the distance from the initial range to the bounding box 64 is equal to or greater than a first criterion, the detection unit 18 causes the output unit 15 to output a first warning 37. The output first warning 37 is a warning that the face is moving too much, such as a message such as "Your face is moving." Furthermore, when the distance from the initial range to the bounding box 64 is equal to or greater than a second criterion, the detection unit 18 causes the output unit 15 to output a second warning 38. The output second warning 38 is a warning that prompts the user to redo the measurement, such as a message such as "Please measure again." The first and second criterion are given as thresholds or the like and are predetermined by the manufacturer according to the biometric information 31 to be measured. The second criterion is greater than the first criterion.

[0090] After completing measurements for the set time and the set number of times, the measurement unit 19 performs operations such as sending the measured biometric information 31 to a biometric information analysis unit (not shown), and then ends the processing.

[0091] 1.6 Variations FIG. 10 is a diagram schematically illustrating an image obtained by a biological information measurement device according to a first modified example of the first embodiment, and registered feature points registered in the biological information measurement device.

[0092] In the first embodiment, the detected feature points are six points consisting of the left and right ends of the eye region, nose region, and mouth region. However, the detected feature points may be multiple points 131 arranged along the contours of the eye region, nose region, and mouth region, as shown in FIG.

[0093] 2. Second embodiment The following describes the differences between the second embodiment and the first embodiment. For points that are not described, the second embodiment also employs the same configuration as that employed in the first embodiment.

[0094] 11 and 12 are diagrams schematically illustrating examples of screens displayed when the biological information measurement device of the second embodiment performs initial settings.

[0095] In the second embodiment, in step S64, the initial setting unit 51 causes the display unit 13 to display the first image 21, the white star 71, the guide 72, and the rectangular frame 73 in an overlapping state, as shown in FIG.

[0096] The rectangular frame 73 indicates the ROI 62 in the first image 21. Fig. 11 illustrates a case where the ROI 62 overlaps the outline of the face region 81 and the eye region 82. The rectangular frame 73 is merely an example of a third shape indicating the ROI 62 in the first image 21. Therefore, the rectangular frame 73 may be replaced with another shape.

[0097] Also, in the second embodiment, in step S65, the user 902 refers to the first image 21 and the rectangular frame 73 displayed superimposed on each other and inputs a second instruction 39 to the input unit 14 instructing to move or deform the ROI 62 indicated by the rectangular frame 73.

[0098] Based on the input second instruction 39, the initial setting unit 51 corrects the ROI 62 indicated by the rectangular frame 73 to obtain the corrected ROI 62. The correction of the ROI 62 is a correction of the position and / or shape of the ROI 62. As a result, as shown in FIG. 12 , the ROI 62 indicated by the rectangular frame 73 is corrected to an ROI 62 that does not overlap the outline of the face region 81 and the eye region 82.

[0099] In the second embodiment, in step S107, the measurement unit 19 measures the biological information 31 from the image in the corrected ROI 62 included in the third image 23.

[0100] These measures can solve the problem shown in FIG. 15D, where the ROI 62 overlaps undesirable areas such as the outline of the face area 803 and the eye area 806, reducing the accuracy of measuring biological information.

[0101] In step S65, the initial setting unit 51 may cause the output unit 15 to output advice to help with the correction. The advice to be output is, for example, a message such as "Please select a place that is as flat as possible and does not overlap with obstacles such as glasses, hair, or irregularities in the face."

[0102] 3 Third embodiment The following describes the differences between the third embodiment and the second embodiment. For points that are not described, the third embodiment also employs the same configuration as that employed in the second embodiment.

[0103] FIG. 13 is a diagram schematically illustrating an example of a screen displayed when the biological information measurement device of the third embodiment performs initial settings.

[0104] 13 , in step S64, the initial setting unit 51 causes the first image 21, the outlined star 71, the guide 72, and the rectangular frame 73 to be superimposed and displayed in the first area 151 of the display unit 13. At the same time, the initial setting unit 51 causes the model image 181, in which the model image 161, the outlined star 171, the guide 172, and the rectangular frame 173 are superimposed, to be displayed in the second area 152 of the display unit 13.

[0105] The model image 161 is prepared by the manufacturer. The model image 161 may be either a real-life image or an illustration. The real-life image may be an image of a living body other than the living body 901 to be imaged.

[0106] The open star 171 indicates the position of the feature point included in the model image 161. The open star 171 is merely an example of a graphic that indicates the position of the feature point included in the model image 161. For this reason, the open star 171 may be replaced with another graphic. For example, the open star 171 may be replaced with a dot placed at the position of the feature point, a wireframe made up of lines connecting the positions of the feature points, or the like. The position indicated by the open star 171 does not deviate from the position of the feature point included in the model image 161.

[0107] The rectangular frame 173 indicates an appropriate ROI 62 in the example image 161. The rectangular frame 173 is displayed when the ROI 62 is corrected, and may not be displayed when the ROI 62 is not corrected.

[0108] The first region 151 and the second region 152 are arranged side by side.

[0109] Also, in the third embodiment, in step S65, the user 902 refers to the displayed model image 181 and inputs a first instruction 33 to the input unit 14 instructing the user to move the positions indicated by the white star marks 71 to the exact positions of the feature points included in the first image 21, and inputs a second instruction 39 to the input unit 14 instructing the user to move or deform the ROI 62 indicated by the rectangular frame 73.

[0110] In the third embodiment, by having the user 902 refer to the model image 181, the user 902 can input the appropriate first instruction 33 and second instruction 39.

[0111] The positions and numbers of the feature points and ROIs 62 are not limited to those shown in FIG. 13, but are determined according to the biological information 31 to be measured.

[0112] 4 Fourth embodiment The following describes the differences between the fourth embodiment and the first embodiment. For points that are not described, the fourth embodiment also employs the same configuration as that employed in the first embodiment.

[0113] FIG. 14 is a diagram schematically illustrating an example of a screen displayed when the biological information measurement device of the fourth embodiment prepares for measurement.

[0114] 14, in step S103, the measurement preparation unit 52 displays the second image 22, the black star 111, and the guide 112 in a superimposed state in the first area 151 of the display unit 13. At the same time, the measurement preparation unit 52 displays the example image 211 in which the first image 21, the black star 201, and the guide 202 are superimposed in the second area 152 of the display unit 13.

[0115] The first image 21 is the image obtained when the initial settings are made.

[0116] The black star 201 indicates the corrected position. The black star 201 is merely an example of a graphic that indicates the corrected position. Therefore, the black star 201 may be replaced with another graphic. For example, the black star 201 may be replaced with a wireframe consisting of points placed at the corrected positions and lines connecting the corrected positions. The position indicated by the black star 201 does not deviate from the position of the feature point included in the first image 21.

[0117] The first region 151 and the second region 152 are arranged side by side.

[0118] In the fourth embodiment, in step S104, the user 902 refers to the model image 211 and aligns the positions of the feature points included in the second image 22 with the corrected positions indicated by the black star marks 111.

[0119] In the fourth embodiment, by having the user 902 refer to the model image 211, the user 902 can perform appropriate alignment.

[0120] The positions and number of feature points are not limited to those shown in FIG. 14, but are determined depending on the biometric information 31 to be measured.

[0121] The present disclosure is not limited to the above-described embodiments, and may be replaced with a configuration that is substantially the same as the configuration shown in the above-described embodiments, a configuration that has the same effect, or a configuration that can achieve the same purpose. [Explanation of symbols]

[0122] 1 Biometric information measuring device, 11 Control unit, 12 Imaging unit, 13 Display unit, 14 Input unit, 15 Output unit, 16 Audio input unit, 17 Memory unit, 18 Detection unit, 19 Measurement unit, 21 First image, 22 Second image, 23 Third image, 31 Biometric information, 32 Identification information, 33 First instruction, 34 Input audio, 35 Identification information, 36 Audio, 37 First warning, 38 Second warning, 41 Biometric information measuring program, 51 Initial setting unit, 52 Measurement preparation unit, 61 Face area, 62 ROI, 63 Registered feature point, 71 White star, 72 Guide, 73 Rectangular frame, 81 Face area, 82 Eye area, 111 Black star, 112 Guide, 121 Face area, 131 Point, 151 First area, 152 Second area, 161 Example image, 171, white star, 172, guide, 173, rectangular frame, 181, example image, 801, video image, 802, guide, 803, face region, 804, feature points, 805, ROI, 806, eye region, 901, biological object, 902, user, 903, face, 911, network, 912, external storage medium

Claims

1. An imaging unit; A display unit; an input unit; a detection unit that detects feature points of a plurality of different parts of the living body that are included in a first image obtained by the imaging unit capturing an image of the living body and that correspond to biometric information of the living body; an initial setting unit that displays the first image and a first graphic indicating the positions of the detected feature points of the plurality of different parts in an overlapping manner on the display unit, and corrects the first graphic based on a first instruction input to the input unit by a user to move the first graphic to a first position input to the input unit to obtain a plurality of corrected positions; a measurement preparation unit that displays a second image obtained by imaging the living body by the imaging unit and a second graphic indicating the corrected positions on the display unit in an overlapping manner, and causes the living body to align the second positions of the feature points of the different parts included in the second image with the corrected positions; a measurement unit that measures the biological information from a third image obtained by the imaging unit capturing an image of the living body in response to input of a signal indicating that the second position has been aligned with the corrected plurality of positions; Equipped with the initial setting unit causes the display unit to superimpose the first image and a third graphic indicating a region of interest in the first image, and corrects the region of interest based on a second instruction input to the input unit to move or deform the region of interest to obtain a corrected region of interest; The measurement unit measures the biological information from an image within the corrected region of interest included in the third image. Biometric information measuring device.

2. an output unit; the detection unit determines an initial range of a range to be set in the third image based on the corrected positions, causes the range to track an image of the living body included in the third image, and causes the output unit to output a warning when a distance from the initial range to the range becomes equal to or greater than a reference value. The biological information measuring device according to claim 1 .

3. Equipped with a voice input unit, the initial setting unit registers an input voice input to the voice input unit; The input of the signal means that a voice corresponding to the input voice is input to the voice input unit. The biological information measuring device according to claim 1 or 2.

4. The measurement unit measures the biological information after a set time has elapsed since the signal was input. The biological information measuring device according to any one of claims 1 to 3.

5. The measurement preparation unit causes the display unit to display the second image and the second figure superimposed on each other, and simultaneously causes the display unit to display a first example image in which the first image and the figure indicating the corrected multiple positions are superimposed on each other. The biological information measuring device according to any one of claims 1 to 4.

6. The initial setting unit causes the display unit to display the first image and the first graphic in an overlapping manner, and simultaneously causes the display unit to display a second model image in which a first model image and a graphic indicating the position of a feature point included in the first model image are overlapped. The biological information measuring device according to any one of claims 1 to 5.

7. The initial setting unit causes the display unit to display the first image and the first figure superimposed on each other, and simultaneously causes the display unit to display a third model image in which a second model image and a figure indicating an appropriate region of interest in the second model image are superimposed on each other. The biological information measuring device according to any one of claims 1 to 6.

8. the first image is a still image, The second image is a video. The biological information measuring device according to any one of claims 1 to 7.

9. the initial setting unit registers identification information for identifying the living body, the corrected positions, and the first image in association with each other; The measurement preparation unit causes the second image and the second figure to be superimposed on the display unit when information corresponding to the identification information is input to the input unit. The biological information measuring device according to any one of claims 1 to 8.

10. a) detecting feature points of a plurality of different parts of a living body included in a first image obtained by capturing an image of the living body, the feature points corresponding to biometric information of the living body; b) superimposing and displaying the first image and a first graphic indicating the respective positions of the detected feature points of the plurality of different parts, and correcting the first graphic based on a first instruction to move the first graphic to a first position input by a user to obtain a plurality of corrected positions; c) superimposing and displaying a second image obtained by capturing an image of the living body and a second graphic indicating the corrected positions, and having the living body align the second positions of the feature points of the different parts included in the second image with the corrected positions; d) measuring the biometric information from a third image obtained by capturing an image of the living body in response to input of a signal indicating that the second position has been aligned with the corrected plurality of positions; Equipped with In the step b), the first image and a third figure indicating a region of interest in the first image are displayed on the display unit in an overlapping manner, and the region of interest is corrected based on a second instruction input by the user, the second instruction instructing to move or deform the region of interest, to obtain a corrected region of interest; In the step d), the biological information is measured from an image within the corrected region of interest included in the third image. Methods for measuring biological information.

11. a) detecting feature points of a plurality of different parts of a living body included in a first image obtained by capturing an image of the living body, the feature points corresponding to biometric information of the living body; b) superimposing and displaying the first image and a first graphic indicating the respective positions of the detected feature points of the plurality of different parts, and correcting the first graphic based on a first instruction to move the first graphic to a first position input by a user to obtain a plurality of corrected positions; c) superimposing and displaying a second image obtained by capturing an image of the living body and a second graphic indicating the corrected positions, and having the living body align the second positions of the feature points of the different parts included in the second image with the corrected positions; d) measuring the biometric information from a third image obtained by capturing an image of the living body in response to input of a signal indicating that the second position has been aligned with the corrected plurality of positions; on the computer, In the step b), the first image and a third figure indicating a region of interest in the first image are displayed on the display unit in an overlapping manner, and the region of interest is corrected based on a second instruction input by the user, the second instruction instructing to move or deform the region of interest, to obtain a corrected region of interest; In the step d), the biological information is measured from an image within the corrected region of interest included in the third image. Biometric information measurement program.

Images