Biometric information acquisition device and biological information acquisition program

Abstract

A facial image region, which is an example of a facial region, includes areas where pulse wave signals are easily acquired and areas where they are difficult to acquire. The detection accuracy of a pulse wave measurement device may decrease depending on the area being measured. [Solution] The biological information acquisition device comprises: an imaging unit that generates imaging data including multiple frame image data; a face recognition unit that identifies a face image included in the frame image data and acquires 3D coordinate data indicating the position of each of multiple feature points; and a detection unit that generates multiple feature point data by associating the 3D coordinate data with the light intensity values ​​of pixels included in the frame image data, and detects the pulse wave signal of the biological organism based on the multiple feature point data. The 3D coordinate data includes a width component, a height component, and a depth component. The detection unit extracts measurement feature point data from the multiple feature point data using the 3D coordinate data and detects the pulse wave signal using the measurement feature point data.

Description

[Technical Field]

[0001] This disclosure relates to a biological information acquisition device and a biological information acquisition program. [Background technology]

[0002] A pulse wave measuring device is known for measuring the pulse waves of a living organism. A pulse wave measuring device is an example of a biological information acquisition device. The pulse wave measuring device described in Patent Document 1 comprises a reception unit, a display control unit, and a measurement unit. The reception unit receives an instruction to perform measurement of a pulse wave signal. The display control unit displays the captured image and a guide frame on the display unit. The measurement unit detects the skin area included in the face area contained within the guide frame. The measurement unit measures the pulse wave signal based on the skin area. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-183079 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] A facial image region, which is an example of a facial region, includes areas where pulse wave signals are easily acquired and areas where they are difficult to acquire. The detection accuracy of a pulse wave measurement device may decrease depending on the area being measured. [Means for solving the problem]

[0005] The biological information acquisition device of this disclosure comprises: an imaging unit that images a living body and generates imaging data including a plurality of frame image data; a face recognition unit that identifies a face image included in the frame image data and acquires three-dimensional coordinate data indicating the position of each of a plurality of feature points included in the face image; and a detection unit that generates a plurality of feature point data by associating the three-dimensional coordinate data with the light intensity values ​​of pixels included in the frame image data, and detects the pulse wave signal of the living body based on the plurality of feature point data, wherein the three-dimensional coordinate data includes a width component, a height component, and a depth component, and the detection unit extracts measurement feature point data from the plurality of feature point data using the three-dimensional coordinate data included in the feature point data, and detects the pulse wave signal using the measurement feature point data.

[0006] The biological information acquisition program of this disclosure causes a computer connected to an imaging unit that images a living organism and generates multiple frame image data to identify a face image included in the frame image data, acquire 3D coordinate data indicating the position of each of the multiple feature points included in the face image, generate multiple feature point data by associating the 3D coordinate data with the light intensity values ​​of pixels included in the frame image data, extract measurement feature point data from the multiple feature point data using the 3D coordinate data included in the feature point data, and detect the pulse wave signal using the measurement feature point data. [Brief explanation of the drawing]

[0007] [Figure 1] A diagram showing the schematic configuration of the measuring device. [Figure 2] A diagram showing the block configuration of the measuring device. [Figure 3] A diagram showing an image captured with facial mesh information. [Figure 4] A diagram showing a portion of the facial mesh information. [Figure 5] A diagram showing an example of grayscale detection values. [Figure 6] A diagram showing an example of mesh point-related information. [Figure 7] A diagram showing an example of the control flow performed by the measuring device. [Figure 8] A diagram showing the time series data of the n-th facial feature points. [Figure 9] A diagram showing an example of an analysis procedure for detecting a pulse wave signal. [Figure 10] A diagram showing the analysis results of a noise-removed signal. [Figure 11] A diagram showing an example of a control flow executed by a measuring device. [Figure 12] A diagram showing an example of a control flow executed by a measuring device. [Figure 13] A diagram showing an example of a control flow executed by a measuring device. [Figure 14] A diagram showing an example of a control flow executed by a measuring device. [Figure 15] A diagram showing an example of a control flow executed by a measuring device. [Figure 16] A diagram showing an example of a control flow executed by a measuring device.

Mode for Carrying Out the Invention

[0008] FIG. 1 shows a schematic configuration of a measuring device 10. The measuring device 10 corresponds to an example of a biological information acquisition device. The measuring device 10 detects a pulse wave signal of a measurer M using video data. The pulse wave signal is a signal indicating the pulse wave of the measurer M. The measurer M corresponds to an example of a living body. The measuring device 10 calculates biological information of the measurer M based on the pulse wave signal. The biological information includes pulse rate, pulse rate variation, oxygen saturation concentration, blood pressure, etc. The measuring device 10 may perform an evaluation of sleep apnea syndrome, etc. based on the biological information. The measuring device 10 displays the biological information calculated based on the pulse wave signal.

[0009] The measuring device 10 is composed of an information processing device such as a personal computer. The measuring device 10 shown in Figure 1 is a laptop computer, but is not limited to this. The measuring device 10 can be any device that has the function of capturing video, or a device that can be connected to a device that captures video. The measuring device 10 is composed of a desktop computer, a tablet terminal, a smartphone, etc. The measuring device 10 includes an imaging unit 11, a display unit 13, and an input unit 15. The measuring device 10 may also include a communication unit, etc., which is not shown.

[0010] The imaging unit 11 captures an image of the person measuring M by receiving reflected light, ambient light, etc., reflected by the person measuring M as detection light. The imaging unit 11 generates video data including the face of the person measuring M. The video data is data that displays a video and is composed of multiple image data. The video data corresponds to an example of imaging data. The video data is composed of multiple image data. The imaging unit 11 generates video data that includes multiple image data. The image data corresponds to an example of frame image data. The image data is composed of multiple output values ​​output on a pixel-by-pixel basis. The image data is data that displays the captured image 100 on the display unit 13. The captured image 100 is a still image. The imaging unit 11 captures at a predetermined frame rate and generates video data. The imaging unit 11 corresponds to an example of an imaging unit.

[0011] The imaging unit 11 is, as one example, a camera including an optical element, an imaging device, and the like. The optical element condenses light onto the imaging device. The imaging device converts the detected light into an output value of an electrical signal. The imaging device generates an output value for each of a plurality of pixels. The output value of each pixel indicates the intensity of light of each pixel. The imaging device generates the output value of each pixel. The imaging device is a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like. The output value includes gradation values of a plurality of color lights. The imaging unit 11 generates gradation values of a plurality of color lights for each pixel. The plurality of color lights are, as one example, red light, green light, and blue light. The red light, the green light, and the blue light are lights having different wavelength bands, respectively. The red wavelength band, which is the wavelength band of the red light, is 600 nm to 800 nm. The green wavelength band, which is the wavelength band of the green light, is 520 nm to 550 nm. The blue wavelength band, which is the wavelength band of the blue light, is 430 nm to 490 nm. The imaging unit 11 may include IR (Infrared) color light or the like. The imaging unit 11 generates, for each pixel, a red gradation value that is the gradation value of the red light, a green gradation value that is the gradation value of the green light, and a blue gradation value that is the gradation value of the blue light. The image data is composed of output values for each pixel including the red gradation value, the green gradation value, and the blue gradation value. The image data includes the luminance and the like of each pixel.

[0012] The imaging unit 11 shown in FIG. 1 is a camera built in the measuring device 10, but is not limited thereto. The imaging unit 11 may be an external camera connected to the measuring device 10. The external camera is a near-infrared camera, a web camera, a smartphone camera, or the like.

[0013] The display unit 13 displays various information such as the captured image 100. The display unit 13 displays various biological information based on the pulse wave signal. The display unit 13 may also display comments based on the biological information. The display unit 13 is composed of a liquid crystal panel, an organic EL (electro-luminescence) panel, etc. The display unit 13 may have a touch input function. When it has a touch input function, the display unit 13 functions as an input unit 15. The display unit 13 shown in Figure 1 is included in the measuring device 10, but is not limited thereto. The display unit 13 may also be an external display attached to the measuring device 10.

[0014] The input unit 15 accepts various input operations from the measurer M. The input unit 15 generates various input signals in response to the input operations. The input unit 15 shown in Figure 1 is a keyboard included in the measuring device 10, but is not limited to this. The input unit 15 may be a mouse, keyboard, touch panel, pen tablet, etc., connected to the measuring device 10.

[0015] Operator M operates the measuring device 10 at a position opposite the imaging unit 11 of the measuring device 10. Operator M operates the measuring device 10 when causing the measuring device 10 to detect biological information. Operator M may also operate the measuring device 10 when performing tasks such as document creation. The measuring device 10 detects operator M's biological information in the background while operator M is performing tasks such as document creation. By detecting biological information in the background, the measuring device 10 can detect the biological information of operator M in a normal activity state.

[0016] Figure 2 shows the block configuration of the measuring device 10. The measuring device 10 comprises an imaging unit 11, a display unit 13, an input unit 15, a control unit 31, and a storage unit 41.

[0017] The imaging unit 11 transmits video data to the control unit 31. The imaging unit 11 transmits video data to the control unit 31 at predetermined timings. The imaging unit 11 may also transmit image data included in the video data to the control unit 31 at predetermined time intervals. The imaging unit 11 transmits image data including red, green, and blue gradation values ​​for each pixel to the control unit 31. The imaging unit 11 may also transmit video data to the storage unit 41 and have the storage unit 41 store the video data.

[0018] The display unit 13 displays various images based on the control of the control unit 31. The display unit 13 receives display data from the control unit 31 and displays various images based on the display data. The display unit 13 may also display video captured by the imaging unit 11 based on video data. The display unit 13 may also display captured images 100 based on image data included in the video data.

[0019] The input unit 15 transmits an input signal to the control unit 31. By transmitting the input signal to the control unit 31, the input unit 15 causes the control unit 31 to perform various controls. As an example, the input unit 15 transmits a display instruction signal to the control unit 31 to display biological information. The display instruction signal is an example of an input signal. Based on the display instruction signal, the control unit 31 generates biological information display data to display biological information based on the pulse wave signal on the display unit 13. The control unit 31 transmits the biological information display data to the display unit 13. The display unit 13 displays a screen containing biological information based on the biological information display data.

[0020] The control unit 31 is a control controller that controls the operation of various units. The control unit 31 is, for example, a processor having a CPU (Central Processing Unit). The control unit 31 is composed of one or more processors. The control unit 31 is connected to the imaging unit 11, the display unit 13, etc., in a communicative manner. By executing the biological analysis program PG, the control unit 31 functions as an image recognition processing unit 33, a data processing unit 35, and a display control unit 37. The control unit 31 may also function as a functional unit other than the image recognition processing unit 33, the data processing unit 35, and the display control unit 37 by executing the biological analysis program PG. The control unit 31 corresponds to an example of a computer.

[0021] The image recognition processing unit 33 acquires video data transmitted from the imaging unit 11. The image recognition processing unit 33 acquires multiple image data contained in the video data. The image recognition processing unit 33 identifies face images contained in the image data. The image recognition processing unit 33 identifies face images by performing face recognition processing on each image data. Face recognition processing is a process that detects face image feature points contained in the image data and extracts face image regions 121 by matching the face image feature points with a pre-registered face image database. The face image database is a database that stores information related to face image feature points used for face recognition. Examples of face image feature points contained in the image data are the positions and contours of the eyes, nose, and mouth. The face image database is pre-stored in the storage unit 41. If the measuring device 10 is connected to a server via a network, the face image database may be pre-stored on the server. The face image region 121 is the region where the face of the measurer M is displayed. The image recognition processing unit 33 identifies the face image region 121 by performing face recognition processing. The image recognition processing unit 33 corresponds to an example of a face recognition unit.

[0022] For face recognition processing, one example is FaceMesh (hereinafter referred to as FaceMesh), which is included in the Media Pipe provided by Google. FaceMesh is a machine learning model that detects keypoints of a face from an image. The image recognition processing unit 33 performs face recognition processing using FaceMesh on multiple image data, and obtains face image regions 121 and face mesh information 131. The face mesh information 131 is represented by multiple keypoints contained in the face image region 121.

[0023] Figure 3 shows an image 100 containing facial mesh information 131. Figure 3 shows an example of facial mesh information 131. Facial mesh information 131 varies depending on the measurer M. Figure 3 shows mesh points 131a and mesh lines 131b representing the facial mesh information 131.

[0024] Figure 3 shows the XYZ coordinate system. The X-axis is the axis along the width direction of the captured image 100. The Y-axis is the axis along the height direction of the captured image 100. The Z-axis is the axis along the vertical axis passing through the center of the image sensor in the imaging unit 11.

[0025] A mesh point 131a is a point corresponding to a keypoint. There are multiple keypoints, for example, 468 points. The image recognition processing unit 33 identifies multiple mesh points 131a contained within the face image region 121. The image recognition processing unit 33 identifies multiple mesh points 131a for each of the multiple image data. The number of mesh points 131a identified for each of the multiple image data is the same. Each of the multiple mesh points 131a is assigned a predetermined code. Multiple mesh points 131a assigned the same code correspond to the same position within the face image region 121. A mesh point 131a corresponds to an example of a feature point. A mesh line 131b is a line connecting two adjacent mesh points 131a.

[0026] A mesh point 131a is represented by a 3D coordinate system that includes width, height, and depth components, with one of the mesh points 131a as the origin. Figure 3 shows the 3D coordinate system of a mesh point 131a projected onto a 2D plane. Each of the multiple mesh points 131a is represented by a mesh point coordinate system that includes width, height, and depth components. The mesh point coordinate system corresponds to an example of 3D coordinate data.

[0027] Figure 4 shows a portion of the facial mesh information 131. Figure 4 shows a magnified view of the facial mesh information 131 near the left eye of the measurer M. Figure 4 shows multiple mesh points 131a and multiple mesh lines 131b. Figure 4 shows the code names of some of the mesh points 131a among the multiple mesh points 131a.

[0028] Figure 4 shows the code names assigned to some of the mesh points 131a. Examples of code names assigned to mesh points 131a include the first code name TBL-01, the second code name TBL-02, the third code name TBL-03, and the fourth code name TBL-04. The code names shown in Figure 4 represent the tear trough of the left eye of the measurer M. The code name configuration can be set as appropriate. The image recognition processing unit 33 assigns a unique code name to each of the multiple mesh points 131a.

[0029] The image recognition processing unit 33 shown in Figure 2 calculates the mesh point coordinates indicating the positions of each of the multiple mesh points 131a. The image recognition processing unit 33 calculates relative mesh point coordinates with an arbitrary point among the multiple mesh points 131a as the origin. The image recognition processing unit 33 converts the relative mesh point coordinates into mesh point coordinates on the image data. The method for calculating the mesh point coordinates will be described later.

[0030] The data processing unit 35 shown in Figure 2 acquires video data transmitted from the imaging unit 11. The data processing unit 35 acquires multiple image data contained in the video data. The data processing unit 35 acquires the red tone value, green tone value, blue tone value, pixel brightness, etc., for each pixel contained in the image data. The data processing unit 35 appropriately performs data processing such as correction processing on the red tone value, green tone value, blue tone value, etc.

[0031] The data processing unit 35 detects the pulse wave signal of the measurer M. The data processing unit 35 analyzes biological information based on the pulse wave signal. The data processing unit 35 corresponds to an example of a detection unit. The data processing unit 35 acquires image data from the imaging unit 11, consisting of output values ​​for each pixel, including red gradation values, green gradation values, and blue gradation values. The data processing unit 35 acquires face mesh information 131 from the image recognition processing unit 33. The data processing unit 35 associates the mesh points 131a included in the face mesh information 131 with the output values ​​of the pixels. The output values ​​correspond to an example of light intensity values. For each of the multiple image data, the data processing unit 35 associates the mesh points 131a with the output values ​​of the pixels.

[0032] The data processing unit 35 tracks predetermined mesh points 131a included in the face mesh information 131 of each of the multiple image data. The multiple image data is generated in a time series. By identifying each predetermined mesh point 131a in the multiple image data, the data processing unit 35 tracks the position of the predetermined mesh points 131a included in each of the multiple image data generated in a time series.

[0033] The data processing unit 35 may track one of the multiple mesh points 131a, or it may track all of the multiple mesh points 131a. The data processing unit 35 may also track one or more mesh points 131a selected from the multiple mesh points 131a based on predetermined conditions. The selection of mesh points 131a is set as appropriate.

[0034] The data processing unit 35 associates a predetermined mesh point 131a with the output value of a pixel for each image data. The data processing unit 35 obtains the output value associated with the predetermined mesh point 131a. The data processing unit 35 obtains the output value associated with the predetermined mesh point 131a for each image data. By obtaining the output value associated with the predetermined mesh point 131a for each image data generated in time series, the data processing unit 35 obtains the time-series output values ​​associated with the predetermined mesh point 131a as time-series data of facial feature points.

[0035] The data processing unit 35 detects the pulse wave signal of the measurer M using time-series data of facial feature points associated with predetermined mesh points 131a. The data processing unit 35 generates a detection value using multiple output values ​​included in the time-series data of facial feature points. The detection value includes the calculated grayscale detection value. The detection value may be time-series data of facial feature points associated with one mesh point 131a, or a value calculated based on multiple time-series data of facial feature points associated with each of multiple mesh points 131a. The calculated value is the average value of the output values ​​associated with each of the multiple mesh points 131a for each image data. The detection value is calculated for each image data in the video data. The data processing unit 35 acquires the pulse wave signal based on the detection value of each predetermined mesh point 131a in the multiple image data.

[0036] The data processing unit 35 detects the pulse wave signal using each grayscale detection value included in the detected value. As an example, the data processing unit 35 detects the pulse wave signal using at least one of the red grayscale detection value Dr, the green grayscale detection value Dg, and the blue grayscale detection value Db. The red grayscale detection value Dr is calculated using the red grayscale value. The green grayscale detection value Dg is calculated using the green grayscale value. The blue grayscale detection value Db is calculated using the blue grayscale value. As an example, the pulse wave signal is detected based on the green grayscale detection value Dg. The pulse wave signal may also be detected based on the difference between the green grayscale detection value Dg and at least one of the red grayscale detection value Dr and the blue grayscale detection value Db.

[0037] Figure 5 shows an example of grayscale detection values. Figure 5 shows the red grayscale detection value Dr, the green grayscale detection value Dg, and the blue grayscale detection value Db. Figure 5 shows the time-dependent changes of the red grayscale detection value Dr, the green grayscale detection value Dg, and the blue grayscale detection value Db as waveform signals. Figure 5 shows the red grayscale detection value Dr, the green grayscale detection value Dg, and the blue grayscale detection value Db in the body movement interval S1 and the red grayscale detection value Dr, the green grayscale detection value Dg, and the blue grayscale detection value Db in the resting interval S2. The body movement interval S1 is the interval in which facial movement and changes in expression occur. The resting interval S2 is the interval in which facial movement and changes in expression are smaller than a predetermined amount of change.

[0038] Figure 5 shows the green tone detection value Dg. The green tone detection value Dg corresponds to the amount of green light contained in the output value at one or more predetermined mesh points 131a. The green tone detection value Dg is included in the detected value.

[0039] As shown in Figure 5, in the movement interval S1, the green grayscale detection value Dg fluctuates due to the effects of body movement. The pulse wave signal included in the green grayscale detection value Dg is difficult to detect due to the fluctuating noise. In the resting interval S2, the influence of the fluctuating noise caused by body movement on the green grayscale detection value Dg decreases, and the pulse wave signal becomes detectable.

[0040] Figure 5 shows the red tone detection value Dr. The red tone detection value Dr corresponds to the light intensity value of red light included in the output value at one or more predetermined mesh points 131a. The red tone detection value Dr is included in the detected value.

[0041] As shown in Figure 5, in the motion interval S1, the red gradation detection value Dr fluctuates due to the effects of body movement. The pulse wave signal included in the red gradation detection value Dr is difficult to detect due to fluctuating noise. In the resting interval S2, the effect of fluctuating noise due to body movement on the red gradation detection value Dr decreases, but the signal-to-noise ratio is small, making it difficult to detect the pulse wave signal.

[0042] Figure 5 shows the blue tone detection value Db. The blue tone detection value Db corresponds to the amount of blue light contained in the output value at one or more predetermined mesh points 131a. The blue tone detection value Db is included in the detected value.

[0043] As shown in Figure 5, in the motion interval S1, the blue gradation detection value Db fluctuates due to the effects of body movement. The pulse wave signal included in the blue gradation detection value Db is difficult to detect due to fluctuating noise. In the resting interval S2, the effect of fluctuating noise due to body movement on the blue gradation detection value Db decreases, but the signal-to-noise ratio is small, making it difficult to detect the pulse wave signal.

[0044] The data processing unit 35 detects the pulse wave signal using the red gradation detection value Dr, the green gradation detection value Dg, and the blue gradation detection value Db shown in Figure 5. The data processing unit 35 detects the pulse wave signal using the analysis procedure described later.

[0045] The data processing unit 35 calculates biological information such as pulse rate by calculating the period, amplitude, etc., of the pulse wave signal. The data processing unit 35 transmits the biological information, including the pulse wave signal, to the display control unit 37. The data processing unit 35 may also store the biological information, etc., in the storage unit 41.

[0046] The display control unit 37 shown in Figure 2 controls the display by the display unit 13. The display control unit 37 acquires biological information, including pulse wave signals, from the data processing unit 35. The display control unit 37 generates biological information display data, including the biological information. The display control unit 37 transmits the biological information display data to the display unit 13. The display control unit 37 displays the biological information display data on the display unit 13. By displaying the biological information display data on the display unit 13, the display control unit 37 can notify the person performing the measurement M of the detection results of the biological information.

[0047] The display control unit 37 may generate message data indicating the operating status of the biological analysis program PG. The message data may include a start message, an execution message, and an end message. The start message indicates that the detection of biological information has begun. The execution message indicates that biological information is being detected. The end message indicates that the detection of biological information has ended. The display control unit 37 transmits the message data to the display unit 13. The display control unit 37 causes the display unit 13 to display the message data.

[0048] The memory unit 41 stores various programs, various data, etc. The memory unit 41 corresponds to an example of a memory unit. The memory unit 41 stores a bioanalysis program PG and mesh point-related information MT. The memory unit 41 stores document creation programs, spreadsheet programs, etc. The memory unit 41 may also store various detection data such as video data and biological information. The memory unit 41 may also store a facial image database. The memory unit 41 is composed of semiconductor memory such as RAM (Random Access Memory) and ROM (Read Only Memory). The memory unit 41 may have an HDD (Hard Disk Drive). The memory unit 41 may also function as a work area for the control unit 31.

[0049] The bioanalysis program PG is a program that causes the measuring device 10 to detect pulse wave signals. The bioanalysis program PG is executed by the control unit 31. By executing the bioanalysis program PG in the control unit 31, the control unit 31 functions as various functional units. The bioanalysis program PG detects various biological information based on the pulse wave signals. The bioanalysis program PG may be executed in the background when the control unit 31 is running a document creation program or the like. The bioanalysis program PG corresponds to an example of a biological information acquisition program.

[0050] The mesh point-related information MT is information relating to the adjacent mesh points 131a for each of the multiple mesh points 131a. The mesh point-related information MT is generated when the image recognition processing unit 33 performs face recognition processing and is stored in the storage unit 41. The mesh point-related information MT is used when the data processing unit 35 selects one or more mesh points 131a from the multiple mesh points 131a.

[0051] Figure 6 shows an example of mesh point-related information MT. Figure 6 shows mesh point-related information MT in table format. Figure 6 shows information related to mesh point 131a for the first code name TBL-01, second code name TBL-02, third code name TBL-03, and fourth code name TBL-04 shown in Figure 4.

[0052] The mesh point association information MT shows each mesh point 131a and the adjacent mesh points 131a. Two adjacent mesh points 131a are connected by a mesh line 131b. The mesh point association information MT shows two or more mesh points 131a connected by a mesh line 131b. For example, the mesh point association information MT shows that a mesh point 131a of the first code name TBL-01 is adjacent to a mesh point 131a of the second code name TBL-02. The mesh point association information MT shows that a mesh point 131a of the second code name TBL-02 is adjacent to a mesh point 131a of the first code name TBL-01 and a mesh point 131a of the third code name TBL-03.

[0053] Figure 7 shows an example of the control flow performed by the measuring device 10. Figure 7 shows the control flow for acquiring a pulse wave signal using the output value of the pixel corresponding to the mesh point 131a. The control flow is performed by running the bioanalysis program PG. Figure 7 shows the control flow as a flowchart.

[0054] In step S101, the measuring device 10 acquires video data. The measuring device 10 causes the imaging unit 11 to generate video data. The imaging unit 11 images the person being measured M and generates video data. The video data includes multiple image data. The multiple image data is generated in a time series. The imaging unit 11 generates multiple image data. For example, when the frame rate when the imaging unit 11 generates the video data is 30 fps (frames per second) and the measurement time is 8 seconds, the number of image data will be 240. The imaging unit 11 transmits the video data to the control unit 31.

[0055] After acquiring video data, the measuring device 10 starts acquiring time-series data of facial feature points in step S103. The data processing unit 35 of the control unit 31 acquires the video data transmitted from the imaging unit 11. The control unit 31 acquires multiple image data contained in the video data. The data processing unit 35 acquires time-series data of facial feature points for each mesh point 131a by performing the processing from step S103 to step S111. Details of the time-series data of facial feature points will be described later.

[0056] In step S105, the measuring device 10 acquires image data. The image recognition processing unit 33 of the control unit 31 sequentially acquires the image data generated in time series. When the video data contains k image data, the image recognition processing unit 33 sequentially acquires the first to the kth image data. k is an arbitrary integer. k is set by the frame rate of the video data and the measurement time.

[0057] After acquiring image data, the measuring device 10 performs face recognition processing in step S107. The image recognition processing unit 33 performs face recognition processing for each of the multiple image data. The image recognition processing unit 33 generates face mesh information 131 included in each image data. The face mesh information 131 includes multiple mesh points 131a and multiple mesh lines 131b. If a face image region 121 is included in each image data, the number of multiple mesh points 131a included in each image data is the same.

[0058] After performing face recognition processing, the measuring device 10 acquires the mesh point coordinates of each mesh point 131a in step S109. The image recognition processing unit 33 acquires face mesh information 131 for each image data. The image recognition processing unit 33 acquires the mesh point coordinates of multiple mesh points 131a included in each image data. The image recognition processing unit 33 generates relative mesh point coordinates with a predetermined mesh point 131a in the face mesh information 131 as the origin. The face mesh information 131 includes N mesh points 131a, including the nth mesh point 131a. The nth relative mesh point coordinate Crn, which is the relative mesh point coordinate of the nth mesh point 131a in one image data, is expressed by the following equation (1). Crn=(An,Bn,Wn) (1) Here, A is the distance in the width direction from mesh point 131a, which is set as the origin, and is an example of the width component. B is the distance in the height direction from mesh point 131a, which is set as the origin, and is an example of the height component. W is the distance in the depth direction from mesh point 131a, which is set as the origin. n is any integer from 1 to N. N is an integer greater than or equal to 2.

[0059] The image recognition processing unit 33 obtains the nth relative mesh point coordinate Crn and then converts it into a mesh point coordinate on the image data with an arbitrary position in the image data as the origin. The nth mesh point coordinate Cn, which is a mesh point coordinate on the image data, is expressed by the following equation (2). Cn=(xn,yn,zn) (2) Here, x is the distance along the X-axis from the origin in the image data. xn is the distance along the X-axis from the origin in the image data to mesh point 131a at the nth mesh point coordinate Cn. x ​​and xn are examples of width components. y is the distance along the Y-axis from the origin in the image data. yn is the distance along the Y-axis from the origin in the image data to mesh point 131a at the nth mesh point coordinate Cn. y and yn are examples of height components. z is a value calculated based on W. For example, z is the difference value with the average value obtained by averaging the W of multiple mesh points 131a as the origin. z is calculated by converting the difference value to the same units as x and y. zn is the difference value from the average value to mesh point 131a at the nth mesh point coordinate Cn. z and zn are examples of depth components.

[0060] The nth mesh point 131a contained in each of the multiple image data points indicates the same position within the face image region 121. The coordinates Cn of the nth mesh point in each of the multiple image data points vary depending on the position and orientation of the measurer M's face in the image data. The control unit 31 can track the nth mesh point 131a over time by acquiring the coordinates Cn of the nth mesh point contained in each image data point. The control unit 31 can track the positions of all mesh points 131a over time by acquiring the coordinates Cn of the nth mesh point in each of the multiple image data points.

[0061] After acquiring the coordinates of each mesh point, the measuring device 10 acquires the output value of each mesh point coordinate in step S111. The data processing unit 35 acquires the mesh point coordinates from the image recognition processing unit 33. The data processing unit 35 acquires the output value of the pixel corresponding to the mesh point coordinate. The data processing unit 35 may also acquire the output value of the pixel corresponding to the mesh point coordinate and the output value of the surrounding pixels, which are pixels within a predetermined area for the pixel corresponding to the mesh point coordinate. The predetermined area is set in advance. When acquiring the output value of the surrounding pixels, the data processing unit 35 acquires, as an example, the average value of the output value of the pixel corresponding to the mesh point coordinate and the output value of the surrounding pixels as the output value. The nth mesh point output value Bn, which is the output value of the nth mesh point coordinate Cn within a single image data, is expressed by the following equation (3). Bn = (rn, gn, bn) (3) Here, rn is the red gradation value included in the output value of the mesh point coordinate corresponding to the nth mesh point coordinate Cn. gn is the green gradation value included in the output value of the mesh point coordinate corresponding to the nth mesh point coordinate Cn. bn is the blue gradation value included in the output value of the mesh point coordinate corresponding to the nth mesh point coordinate Cn.

[0062] The data processing unit 35 generates the nth face feature point data Dn by associating the nth mesh point coordinate Cn with the nth mesh point output value Bn within a single image data. The nth face feature point data Dn is an example of face feature point data. Face feature point data corresponds to an example of feature point data. The data processing unit 35 generates the nth face feature point data Dn for the nth mesh point coordinate Cn shown in equation (4) below. The nth face feature point data Dn is generated for each of the N mesh point coordinates. Dn=(xn,yn,zn,rn,gn,bn) (4)

[0063] The data processing unit 35 acquires the nth facial feature point data Dn for each image data. The data processing unit 35 generates the nth facial feature point time series data Dn(t) by tracking the nth facial feature point data Dn for each image data. The nth facial feature point time series data Dn(t) is an example of facial feature point time series data for the nth mesh point coordinate Cn. The facial feature point time series data corresponds to an example of a feature point data set. The nth facial feature point time series data Dn(t) includes the T-time nth facial feature point data Dn(T), the T-dt-time nth facial feature point data Dn(T-dt), and the T+dt-time nth facial feature point data Dn(T+dt), all acquired at an arbitrary time T with a frame rate of dt. The nth facial feature point time series data Dn(t) is shown in Figure 8.

[0064] The data processing unit 35 obtains the nth face feature point time series data Dn(t) shown in Figure 8 by acquiring the nth face feature point data Dn at the nth mesh point coordinate Cn for each image data. The nth face feature point time series data Dn(t) is the face feature point time series data of the nth mesh point 131a tracked for each image data.

[0065] After acquiring the output values ​​of each mesh point coordinate, the measuring device 10 determines in step S113 whether or not it has acquired time-series data of face feature points for all mesh point coordinates. The data processing unit 35 determines whether or not it has acquired all nth face feature point time-series data Dn(t) from n = 1 to N. If the data processing unit 35 determines that it has acquired all nth face feature point time-series data Dn(t), the measuring device 10 proceeds to step S115 (step S113: YES). If the data processing unit 35 determines that it has not acquired all nth face feature point time-series data Dn(t), the measuring device 10 returns to step S109 (step S113: NO). The measuring device 10 continues to acquire the nth face feature point time-series data Dn(t).

[0066] In step S115, the measuring device 10 finishes acquiring the time-series data of facial feature points. The data processing unit 35 acquires the nth facial feature point time-series data Dn(t) for n from 1 to N. The data processing unit 35 acquires the nth facial feature point time-series data Dn(t) which includes the nth facial feature point data Dn.

[0067] After acquiring time-series data of facial feature points, the measurement device 10 selects a measurement point in step S117. The data processing unit 35 selects one or more mesh points 131a from among a plurality of mesh points 131a as a measurement point. The measurement device 10 acquires time-series data of facial feature points corresponding to the measurement point as measurement point time-series data. The measurement point time-series data corresponds to an example of a group of feature point data. The measurement point time-series data includes multiple measurement point data, which are facial feature point data corresponding to the measurement point. The measurement point data corresponds to an example of measurement feature point data. The data processing unit 35 extracts measurement point data, which are facial feature point data of the measurement point, by setting the measurement point. The method for setting the measurement point will be described later.

[0068] After setting the measurement point, the measuring device 10 detects the pulse wave signal in step S119. The data processing unit 35 detects the pulse wave signal using the measurement point time series data, or the measurement point data included in the measurement point time series data. The measurement point time series data includes a plurality of measurement point data associated with a predetermined mesh point 131a. The data processing unit 35 generates detected values ​​using the measurement point data. As an example, the data processing unit 35 detects the red gradation detection value Dr, the green gradation detection value Dg, and the blue gradation detection value Db shown in Figure 5 using the measurement point time series data.

[0069] Figure 9 shows an example of the analysis procedure for detecting a pulse wave signal. Figure 9 shows an example of the analysis procedure as a flowchart. The analysis procedure shown in Figure 9 is executed by the data processing unit 35. The analysis procedure shown in Figure 9 detects the pulse wave signal using the red gradation detection value Dr, the green gradation detection value Dg, and the blue gradation detection value Db.

[0070] In step S201, the data processing unit 35 performs sampling of each grayscale detection value at predetermined time intervals. The time interval and sampling frequency are set as appropriate. Preferably, the time interval is a time that includes one or more pulse waves. For example, the time interval is 3 to 10 seconds. For example, the sampling frequency is 10 Hz to 50 Hz. The data processing unit 35 acquires sampled data by performing sampling. The sampled data includes the sampled red grayscale detection value Dr, the green grayscale detection value Dg, and the blue grayscale detection value Db.

[0071] After performing sampling, the data processing unit 35 normalizes the sampled data in step S203. The data processing unit 35 normalizes the green tone detection value Dg included in the sampled data.

[0072] The data processing unit 35 calculates the green mean Gmean, which is the average of multiple green tone detection values ​​Dg, and the green standard deviation Gstd, which is the standard deviation of multiple green tone detection values ​​Dg. The data processing unit 35 normalizes each green tone detection value Dg using the following equation (5). Gnorm m =(Gm-Gmean) / Gstd (5) Here, m is any integer greater than or equal to 1. Gm is the m-th green tone detection value Dg. Gnorm m This is the normalized value of the m-th green tone detection value Dg.

[0073] Similar to the green gradation detection value Dg, the data processing unit 35 normalizes a plurality of red gradation detection values Dr and a plurality of blue gradation detection values Db included in the sampling data. The data processing unit 35 calculates a red average value Rmean, which is the average value of the plurality of red gradation detection values Dr, and a red standard deviation value Rstd, which is the standard deviation value of the plurality of red gradation detection values Dr. The data processing unit 35 calculates a blue average value Bmean, which is the average value of the plurality of blue gradation detection values Db, and a blue standard deviation value Bstd, which is the standard deviation value of the plurality of blue gradation detection values Db. The data processing unit 35 normalizes each red gradation detection value Dr and each blue gradation detection value Db using the following equations (6) and (7). Rnorm m =(Rm - Rmean) / Rstd (6) Bnorm m =(Bm - Bmean) / Bstd (7) Here, m is an arbitrary integer of 1 or more. Rm is the m-th red gradation detection value Dr. Rnorm m is the value obtained by normalizing the m-th red gradation detection value Dr. Bm is the m-th blue gradation detection value Db. Bnorm m is the value obtained by normalizing the m-th blue gradation detection value Db.

[0074] After normalizing the sampling data, the data processing unit 35 performs noise removal processing in step S205. The data processing unit 35 performs noise removal processing using the normalized green gradation detection value Dg, the normalized red gradation detection value Dr, and the blue gradation detection value Db. The data processing unit 35 performs noise removal processing using the following equation (8) to generate a noise removal signal S. Sm = Gnorm m + αBnorm m + βRnorm m (8) Here, m is an arbitrary integer of 1 or more. Sm is the m-th noise removal signal S. α is the first coefficient and β is the second coefficient.

[0075] For example, α and β are -0.5 and -0.5, respectively. When α and β are negative values, the data processing unit 35 detects the noise reduction signal S by subtracting the normalized red tone detection value Dr and the normalized blue tone detection value Db from the normalized green tone detection value Dg. At least one of α and β may be 0. When α=0 and β=-0.5, the data processing unit 35 detects the noise reduction signal S by calculating the difference between the green tone detection value Dg and the red tone detection value Dr. When α=-0.5 and β=0, the data processing unit 35 detects the noise reduction signal S by calculating the difference between the green tone detection value Dg and the blue tone detection value Db. α and β are set appropriately depending on the noise reduction status.

[0076] Figure 10 shows the analysis results of the noise-reduced signal S. Figure 10 is analyzed based on the red gradation detection value Dr, green gradation detection value Dg, and blue gradation detection value Db shown in Figure 5. Figure 10 shows the noise-reduced signal S when α = -0.5 and β = -0.5 are substituted into equation (8). Figure 10 shows the noise-reduced signal S for the motion section S1 and the resting section S2.

[0077] As shown in Figure 10, the denoised signal S corresponds to the pulse wave signal. Noise components such as body movement are removed from the denoised signal S. The data processing unit 35 detects the denoised signal S as a pulse wave signal. In the resting section S2, the denoised signal S has a more clearly detectable signal waveform compared to the green gradation detection value Dg. In the moving section S1, the denoised signal S is adjusted to a signal waveform corresponding to the pulse wave signal. Due to the noise reduction processing, the data processing unit 35 can detect the pulse wave signal in both the moving section S1 and the resting section S2.

[0078] The data processing unit 35 may calculate biological information such as pulse waves using the noise reduction signal S. The data processing unit 35 acquires the noise reduction signal S as a pulse wave signal. The data processing unit 35 calculates biological information such as pulse rate by calculating the period, amplitude, etc., of the pulse wave signal. The data processing unit 35 transmits the biological information, including the pulse wave signal, to the display control unit 37. The data processing unit 35 may store the biological information, etc., in the storage unit 41.

[0079] The data processing unit 35 acquires a pulse wave signal using one or any number of time series data Dn(t) of the nth facial feature point, from among the N time series data Dn(t) of the nth facial feature point, where n is from 1 to N.

[0080] Figure 11 shows an example of the control flow performed by the measuring device 10. Figure 11 shows an example of how to set measurement points. Figure 11 shows an example of the control flow performed in step S117 shown in Figure 7. Figure 11 shows a control flow for setting measurement points using the depth component included in the mesh point coordinates.

[0081] In step S301, the measuring device 10 acquires face feature point data. The data processing unit 35 acquires face feature point data included in the face feature point time series data. The data processing unit 35 acquires zn included in the nth face feature point data Dn. zn is the depth component of the nth mesh point coordinate Cn.

[0082] After acquiring facial feature point data, the measuring device 10 reads out the depth setting range in step S303. The data processing unit 35 reads out the depth setting range stored in the storage unit 41. The depth setting range is numerical range information that specifies the depth coordinate range of the depth component included in the mesh point coordinates of the mesh point 131a extracted as a measurement point. The depth setting range corresponds to an example of a depth range threshold. The depth setting range is stored in the storage unit 41 in advance. As an example, the depth setting range is set by the minimum depth coordinate value zmin and the maximum depth coordinate value zmax. The minimum depth coordinate value zmin is the lower limit of the depth coordinate range. The maximum depth coordinate value zmax is the upper limit of the depth coordinate range.

[0083] After reading the depth setting range, the measuring device 10 determines in step S305 whether the depth component included in the mesh point coordinates is within the depth setting range. The data processing unit 35 compares the depth component included in the mesh point coordinates with the depth setting range. The data processing unit 35 determines whether the depth component included in the mesh point coordinates of each mesh point 131a is in the relationship of equation (9) below. zmin <zn<zmax (9)

[0084] The average of multiple depth components is set to 0, for example. Positions closer to the imaging unit 11 than the position where the average value of the depth components is obtained have negative values, and positions further from the imaging unit 11 than the position where the average value of the depth components is obtained have positive values. The edges of the face will have a value less than 0. The depth components of the tip of the nose and the area around the columella will have a value greater than 0. The output values ​​of areas such as the edges of the face, the tip of the nose, and the area around the columella are prone to fluctuations depending on the orientation of the face. The output values ​​of areas such as the edges of the face, the tip of the nose, and the area around the columella are excluded from pulse wave signal detection depending on the depth setting range. By using mesh point 131a of the mesh point coordinates within the depth setting range as measurement points, the accuracy of pulse wave signal detection is improved.

[0085] When zn included in the nth mesh point coordinate Cn is related to equation (9), the data processing unit 35 determines that the depth component is within the depth setting range. The measuring device 10 proceeds to step S307 (step S305: YES). When zn included in the nth mesh point coordinate Cn is not related to equation (9), the data processing unit 35 determines that the depth component is not within the depth setting range. The measuring device 10 proceeds to step S309 (step S305: NO).

[0086] The measuring device 10 may, in step S305, normalize zn and then compare the normalized zn with the depth setting range. The depth setting range is set to a numerical range corresponding to the normalized zn. The data processing unit 35 normalizes zn using the following equation (10). znorm n =(z / (zmax-zmin))n (10) Here, znorm n The normalized zn is shown. zmax is the maximum value of zn included in the facial feature point data. zmin is the minimum value of zn included in the facial feature point data.

[0087] For example, the depth setting range corresponding to the normalized zn is a range of 25% above and below the median. The depth setting range is set as appropriate and stored in the storage unit 41. The data processing unit 35 determines whether the normalized zn is within the depth setting range.

[0088] In step S307, the measuring device 10 sets a mesh point 131a whose depth component is within the depth setting range as a measurement point. The data processing unit 35 acquires the time-series data of the facial feature points of the mesh point 131a set as the measurement point as measurement point time-series data. The measurement point time-series data includes data from multiple measurement points. Using the measurement point time-series data including data from multiple measurement points, the data processing unit 35 detects a pulse wave signal in step S119 shown in Figure 7.

[0089] In step S309, the measuring device 10 excludes mesh points 131a whose depth component is not within the depth setting range from the measurement points. The data processing unit 35 does not use the time-series data of the facial feature points of the mesh points 131a that have been excluded from the measurement points for pulse wave signal detection.

[0090] The data processing unit 35 compares zn, which is included in the nth mesh point coordinate Cn where n is from 1 to N, with the depth setting range, according to the control flow shown in Figure 11. By comparing zn with the depth setting range, the data processing unit 35 sets one or more mesh points 131a as measurement points. The data processing unit 35 acquires the time-series data of the face feature points of the measurement points as measurement point time-series data. The measurement point time-series data includes data from multiple measurement points.

[0091] The data processing unit 35 detects a pulse wave signal in step S119 shown in Figure 7 using the measurement point time series data, which is time series data of facial feature points from one or more measurement points. When multiple measurement points are set, the data processing unit 35 acquires multiple facial feature point data included in the time series data of facial feature points from each measurement point. As an example, the data processing unit 35 calculates the average mesh point output value Bave using the mesh point output value of the measurement point for each image data. The average mesh point output value Bave is expressed by the following equation (11). Bave=(rave,gave,bave) (11) Here, rave is the average red gradation value calculated based on the output values ​​of each measurement point. given is the average green gradation value calculated based on the output values ​​of each measurement point. bave is the average blue gradation value calculated based on the output values ​​of each measurement point.

[0092] The data processing unit 35 calculates the average mesh point output value time series data Bave(t) using the average mesh point output value Bave. The average mesh point output value time series data Bave(t) is expressed by the following equation (12). Bave(t)=(rave(t),gave(t),bave(t)) (12)

[0093] The data processing unit 35 acquires the pulse wave signal in step S119 shown in Figure 7, using the average mesh point output value time series data Bave(t) calculated based on the facial feature point time series data of each measurement point. The data processing unit 35 improves the measurement accuracy of the pulse wave signal by tracking multiple mesh points 131a and acquiring the pulse wave signal using the facial feature point data of multiple mesh points 131a.

[0094] The measuring device 10 includes an imaging unit 11 that captures images of the person being measured M and generates video data containing multiple image data; an image recognition processing unit 33 that identifies a face image contained in the image data and obtains mesh point coordinates indicating the position of each of the multiple mesh points 131a contained in the face image; and a data processing unit 35 that generates multiple face feature point data by associating the mesh point coordinates with the output values ​​of pixels contained in the image data, and detects the pulse wave signal of the person being measured M based on the face feature point data. The mesh point coordinates include a width component, a height component, and a depth component. The data processing unit 35 extracts measurement point data from the multiple face feature point data using the mesh point coordinates contained in the face feature point data, and detects the pulse wave signal using the measurement point data. The measuring device 10 improves the accuracy of pulse wave signal detection by detecting the pulse wave signal using measurement point data.

[0095] The measuring device 10 includes a storage unit 41 that stores a depth setting range that specifies the depth coordinate range of the depth component. The data processing unit 35 preferably extracts measurement point data by comparing the depth component included in the mesh point coordinates with the depth setting range. The data processing unit 35 extracts measurement point data using the depth component and depth setting range included in the mesh point coordinates, thereby excluding mesh points 131a within the face image region 121 that reduce the accuracy of pulse wave signal detection from the measurement points. The accuracy of pulse wave signal detection is improved.

[0096] The bioanalysis program PG instructs the control unit 31, which is connected to the imaging unit 11 that images the subject M and generates multiple image data, to identify the face image contained in the image data, to obtain the mesh point coordinates indicating the position of each of the multiple mesh points 131a contained in the face image, to generate multiple face feature point data by associating the mesh point coordinates with the output values ​​of the pixels contained in the image data, to extract measurement point data from the multiple face feature point data using the mesh point coordinates contained in the face feature point data, and to detect the pulse wave signal using the measurement point data. The bioanalysis program PG improves the accuracy of pulse wave signal detection by using measurement point data to detect pulse wave signals.

[0097] Figure 12 shows an example of the control flow performed by the measuring device 10. Figure 12 shows an example of how to set measurement points. Figure 12 shows an example of the control flow performed in step S117 shown in Figure 7. Figure 12 shows a control flow for setting measurement points using the depth component and output value included in the mesh point coordinates.

[0098] In step S401, the measuring device 10 acquires face feature point data. The data processing unit 35 acquires face feature point data included in the face feature point time series data. The data processing unit 35 acquires zn included in the nth face feature point data Dn. zn is the depth component of the nth mesh point coordinate Cn.

[0099] After acquiring facial feature point data, the measuring device 10 reads out the depth setting range in step S403. The data processing unit 35 reads out the depth setting range stored in the storage unit 41.

[0100] After reading the depth setting range, the measuring device 10 determines in step S405 whether the depth component of the mesh point coordinates is within the depth setting range. The data processing unit 35 compares the depth component of the mesh point coordinates with the depth setting range. The data processing unit 35 determines whether the depth component included in the mesh point coordinates of each mesh point 131a is related by equation (9).

[0101] When zn included in the nth mesh point coordinate Cn is related to equation (9), the data processing unit 35 determines that the depth component is within the depth setting range. The measuring device 10 proceeds to step S407 (step S405: YES). When zn included in the nth mesh point coordinate Cn is not related to equation (9), the data processing unit 35 determines that the depth component is not within the depth setting range. The measuring device 10 proceeds to step S413 (step S405: NO).

[0102] In step S407, the measuring device 10 reads out the output value setting range. The data processing unit 35 reads out the output value setting range from the storage unit 41. The output value setting range corresponds to an example of a light intensity threshold. The output value setting range is stored in the storage unit 41 in advance. The output value setting range is information that specifies the output value range of the output value associated with the mesh point coordinates. The output value range corresponds to an example of a light intensity range. The output value setting range includes information that specifies the lower limit of at least one of the red gradation value, green gradation value, and blue gradation value included in the face feature point data. The output value setting range is compared with the output value associated with the mesh point coordinates.

[0103] After reading the output value setting range, the measuring device 10 determines in step S409 whether the output value is within the output value setting range. In step S405, the data processing unit 35 compares the output value corresponding to the mesh point coordinates whose depth component is within the depth setting range with the output value setting range. The data processing unit 35 compares at least one of the red gradation value, green gradation value, and blue gradation value included in the face feature point data of each mesh point 131a with the output value setting range.

[0104] If the data processing unit 35 determines that the output value is within the output value setting range, the measuring device 10 proceeds to step S411 (step S409: YES). If the data processing unit 35 determines that the output value is not within the output value setting range, the measuring device 10 proceeds to step S413 (step S409: NO).

[0105] In step S411, the measuring device 10 sets a mesh point 131a of facial feature point data where the depth component is within the depth setting range and the output value is within the output value setting range as a measurement point. The data processing unit 35 acquires the facial feature point time series data of the mesh point 131a set as a measurement point.

[0106] In step S413, the measuring device 10 excludes mesh points 131a of face feature point data whose depth component is not within the depth setting range and mesh points 131a of face feature point data whose output value is not within the output value setting range from the measurement points. The data processing unit 35 does not use the time-series data of face feature points of mesh points 131a that have been excluded from the measurement points for pulse wave signal detection.

[0107] The data processing unit 35 executes the control flow shown in Figure 12 for the nth facial feature point data Dn, where n is from 1 to N, and sets one or more mesh points 131a as measurement points. The data processing unit 35 extracts the facial feature point time series data corresponding to one or more measurement points as measurement point time series data. The measurement point time series data, which includes the measurement point data, is used to detect the pulse wave signal in step S119 shown in Figure 7.

[0108] The memory unit 41 stores an output value setting range that specifies the range of output values. The data processing unit 35 preferably extracts measurement point data by comparing the output values ​​included in the facial feature point data with the output value setting range. The measuring device 10 can exclude the output values ​​of mesh points 131a at locations where shadows are present due to the position of the light source, etc. This improves the measurement accuracy of the pulse wave signal.

[0109] Figure 13 shows an example of the control flow performed by the measuring device 10. Figure 13 shows an example of how to set measurement points. Figure 13 shows an example of the control flow performed in step S117 shown in Figure 7. Figure 13 shows a control flow for setting measurement points using mesh point related information MT and depth components included in the mesh point coordinates.

[0110] In step S501, the measuring device 10 acquires facial feature point data. The data processing unit 35 acquires facial feature point data included in the facial feature point time series data. The data processing unit 35 acquires the coordinates Cn of the nth mesh point included in the nth facial feature point data Dn.

[0111] In step S503, the measuring device 10 reads the mesh point-related information MT from the storage unit 41. The data processing unit 35 reads the mesh point-related information MT shown in Figure 6 as an example.

[0112] In step S505, the measuring device 10 reads out the gradient setting range. The gradient setting range is numerical range information that specifies the gradient range of the depth component included in the coordinates of each adjacent mesh point 131a. The gradient setting range is stored in advance in the storage unit 41. As an example, the gradient setting range is set by the minimum set gradient value gmin and the maximum set gradient value gmax. The minimum set gradient value gmin is the lower limit of the gradient setting range. The maximum set gradient value gmax is the upper limit of the gradient setting range.

[0113] After reading the gradient setting range, the measuring device 10 determines in step S507 whether the gradients of two adjacent mesh points 131a are within the gradient setting range. The gradients of two adjacent mesh points 131a are expressed by the following equation (13). gΔn=(zn―zn+1) / dΔn (13) Here, gΔn is the gradient between mesh point 131a at the nth mesh point coordinate Cn and the adjacent mesh point 131a at the (n+1)th mesh point coordinate Cn+1. zn+1 is the depth component included in the (n+1)th mesh point coordinate Cn+1. dΔn is the distance between mesh point 131a at the nth mesh point coordinate Cn and mesh point 131a at the (n+1)th mesh point coordinate Cn+1.

[0114] The data processing unit 35 compares the gradient of two adjacent mesh points 131a with the gradient setting range. The data processing unit 35 determines whether the gradients of two adjacent mesh points 131a are related by the following equation (14). G-Ken <gΔn<gmax (14)

[0115] In the facial periphery, the tip of the nose, and the area around the columella, the gradient of two adjacent mesh points 131a will be outside the gradient setting range. By setting mesh points 131a that are within the gradient setting range as measurement points, the likelihood of detecting output values ​​within a predetermined range is improved. The decrease in detection accuracy of the pulse wave signal is suppressed.

[0116] When the gradient between mesh point 131a and the adjacent mesh point 131a is given by equation (14), the data processing unit 35 determines that mesh point 131a is within the gradient setting range relative to the adjacent mesh point 131a. The data processing unit 35 determines that mesh point 131a can be set as a measurement point. The measuring device 10 proceeds to step S509 (step S507: YES). When the gradient between mesh point 131a and the adjacent mesh point 131a is not given by equation (14), the data processing unit 35 determines that mesh point 131a is not within the gradient setting range relative to the adjacent mesh point 131a. The data processing unit 35 determines that mesh point 131a will not be extracted as a measurement point. The measuring device 10 proceeds to step S511 (step S507: NO).

[0117] In step S509, the measuring device 10 sets one mesh point 131a as a measurement point. The data processing unit 35 acquires the time-series data of the facial feature points of the mesh point 131a set as a measurement point as measurement point time-series data. The measurement point time-series data includes data from multiple measurement points. Using the measurement point time-series data including the measurement point data, the data processing unit 35 detects a pulse wave signal in step S119 shown in Figure 7.

[0118] In step S511, the measuring device 10 excludes one mesh point 131a from the measurement points. The data processing unit 35 does not use the time-series facial feature data of the mesh point 131a that was excluded from the measurement points for pulse wave signal detection.

[0119] Figure 14 shows an example of the control flow performed by the measuring device 10. Figure 14 shows an example of how to set measurement points. Figure 14 shows an example of the control flow performed in step S117 shown in Figure 7. Figure 14 shows a control flow for setting measurement points using the width, height, and depth components included in the mesh point coordinates.

[0120] In step S601, the measuring device 10 acquires facial feature point data. The data processing unit 35 acquires facial feature point data included in the facial feature point time series data. The data processing unit 35 acquires zn included in the nth facial feature point data Dn.

[0121] After acquiring facial feature point data, the measuring device 10 reads out the depth setting range in step S603. The data processing unit 35 reads out the depth setting range stored in the storage unit 41.

[0122] After reading the depth setting range, the measuring device 10 determines in step S605 whether the depth component of the mesh point coordinates is within the depth setting range. The data processing unit 35 compares the depth component of the mesh point coordinates with the depth setting range. The data processing unit 35 determines whether the depth component included in the mesh point coordinates of each mesh point 131a is related by equation (9).

[0123] When zn included in the nth mesh point coordinate Cn is related to equation (9), the data processing unit 35 determines that the depth component is within the depth setting range. The measuring device 10 proceeds to step S607 (step S605: YES). When zn included in the nth mesh point coordinate Cn is not related to equation (9), the data processing unit 35 determines that the depth component is not within the depth setting range. The measuring device 10 proceeds to step S617 (step S605: NO).

[0124] In step S607, the measuring device 10 reads the mesh point-related information MT from the storage unit 41. The data processing unit 35 reads the mesh point-related information MT shown in Figure 6 as an example.

[0125] After reading the mesh point-related information MT, the measuring device 10 calculates the distance between mesh points in step S609. The data processing unit 35 uses the mesh point-related information MT to determine which mesh point 131a is adjacent to which mesh point 131a whose depth component is within the depth setting range. The data processing unit 35 calculates the distance between mesh points using the mesh point coordinates of the mesh point 131a whose depth component is within the depth setting range and the mesh point coordinates of the adjacent mesh point 131a. The distance between mesh points is the distance between the mesh point 131a whose depth component is within the depth setting range and the adjacent mesh point 131a. Here, the mesh point 131a whose depth component is within the depth setting range is, for example, represented as the i-th mesh point. The adjacent mesh point 131a adjacent to the i-th mesh point is represented as the j-th mesh point. The coordinates Ci of the i-th mesh point and the j-th mesh point Cj of the j-th mesh point are expressed by equations (15) and (16) below, respectively. Ci=(xi,yi,zi) (15) Cj=(xj,yj,zj) (16)

[0126] The data processing unit 35 calculates the distance between mesh points ij dij using the coordinates Ci and j mesh point Cj. The distance between mesh points ij dij is an example of a distance between mesh points. The distance between mesh points ij dij is expressed by equation (17).

[0127]

number

[0128] In step S611, the measuring device 10 reads out the set distance range. The data processing unit 35 reads out the set distance range from the storage unit 41. The set distance range corresponds to an example of a plane coordinate threshold. The set distance range is stored in the storage unit 41 in advance. The set distance range is information that specifies the plane range related to the width component and height component of the mesh point coordinates. The plane range corresponds to an example of a plane coordinate range. The set distance range is compared with the distance between mesh points calculated using the width component and height component included in the mesh point coordinates.

[0129] After reading the set distance range, the measuring device 10 determines in step S613 whether the distance between mesh points is within the set distance range. If the distance between mesh points is within the set distance range, the two adjacent mesh points 131a are in a position visible from the imaging unit 11 in a plan view. The data processing unit 35 determines that the two adjacent mesh points 131a have not moved to a position where they are not visible due to the direction of the face, etc. When the distance between mesh points ij dij is within the set distance range, the data processing unit 35 determines that the i-th mesh point can be set as a measurement point. The measuring device 10 proceeds to step S615 (step S613: YES). If the distance between mesh points is not within the set distance range, one of the two adjacent mesh points 131a is not in a position visible from the imaging unit 11 in a plan view. The data processing unit 35 determines that one of the two adjacent mesh points 131a has moved to a position where they are not visible due to the direction of the face, etc. When the distance dij between mesh points ij is not within the set distance range, the data processing unit 35 determines that the i-th mesh point cannot be set as a measurement point. The measuring device 10 proceeds to step S617 (step S613: NO).

[0130] In step S615, the measuring device 10 sets the i-th mesh point as a measurement point. The data processing unit 35 acquires the time-series data of the facial feature points of the i-th mesh point set as the measurement point as measurement point time-series data. Using the measurement point time-series data, which includes data from multiple measurement points, the data processing unit 35 detects the pulse wave signal in step S119 as shown in Figure 7.

[0131] In step S617, the measuring device 10 excludes mesh points 131a and i-mesh points whose depth components are not within the depth setting range from the measurement points. The data processing unit 35 does not use the time-series data of face feature points for mesh points 131a and i-mesh points whose depth components are not within the depth setting range for pulse wave signal detection.

[0132] In the control flow shown in Figure 14, the depth component is first determined to be within the depth setting range, and then the distance between mesh points is determined to be within the set distance range, but this is not limited to this. The measuring device 10 may perform the determination of whether the depth component is within the depth setting range and the determination of whether the distance between mesh points is within the set distance range at the same time. The measuring device 10 may first determine whether the distance between mesh points is within the set distance range, and then determine whether the depth component is within the depth setting range.

[0133] The memory unit 41 has a set distance range that specifies a planar range relating to the width and height components. The data processing unit 35 preferably extracts measurement point data using the set distance range. The measuring device 10 can exclude mesh points 131a that are not visible from a plan view from the imaging unit 11 from the measurement points. This improves the accuracy of pulse wave signal detection.

[0134] Figure 15 shows an example of the control flow performed by the measuring device 10. Figure 15 shows an example of how to set measurement points. Figure 15 shows an example of the control flow performed in step S117 shown in Figure 7. Figure 15 shows a control flow for extracting measurement point data using facial feature point time series data.

[0135] In step S701, the measuring device 10 acquires time-series data of facial feature points. The data processing unit 35 acquires time-series data of facial feature points, including the facial feature point data. The data processing unit 35 acquires the nth time-series data of facial feature points Dn(t).

[0136] After acquiring time-series data of facial feature points, the measuring device 10 calculates the mesh point fluctuation amount in step S703. The mesh point fluctuation amount corresponds to an example of fluctuation amount. The mesh point fluctuation amount indicates the fluctuation of the mesh point coordinates within the measurement time. The mesh point fluctuation amount is a body movement index that indicates the magnitude of the body movement of the measurer M. When body movement increases, the measurement accuracy of the pulse wave signal decreases. The mesh point fluctuation amount is expressed by equation (18) as an example. In equation (18), RMS represents the root mean square.

[0137]

number

[0138] The calculation of mesh point variation may be done using the root mean square, or a different statistical value. The mesh point variation may also be calculated using variance, standard deviation, and coefficient of variation. The method for calculating mesh point variation will be set as appropriate.

[0139] In step S705, the measuring device 10 reads the fluctuation threshold from the storage unit 41. The fluctuation threshold is stored in the storage unit 41 beforehand. The fluctuation threshold is compared with the mesh point fluctuation amount. The fluctuation threshold is set to a predetermined value beforehand. When the mesh point fluctuation amount is smaller than the fluctuation threshold, it indicates that the body movement of the measurer M is within a range where the pulse wave signal can be measured. When the mesh point fluctuation amount is larger than the fluctuation threshold, it indicates that the body movement of the measurer M is within a range where the pulse wave signal cannot be measured.

[0140] After reading the fluctuation threshold, the measuring device 10 determines in step S707 whether the mesh point fluctuation amount is smaller than the fluctuation threshold. The data processing unit 35 compares the mesh point fluctuation amount with the fluctuation threshold. If the data processing unit 35 determines that the mesh point fluctuation amount is smaller than the fluctuation threshold, the measuring device 10 proceeds to step S709 (step S707: YES). If the data processing unit 35 determines that the mesh point fluctuation amount is larger than the fluctuation threshold, the measuring device 10 proceeds to step S711 (step S707: NO).

[0141] In step S709, the measuring device 10 sets a mesh point 131a where the mesh point fluctuation amount is smaller than the fluctuation threshold as a measurement point. The data processing unit 35 acquires the time-series data of the facial feature points of the mesh point 131a set as the measurement point as measurement point time-series data. Using the measurement point time-series data, which includes data from multiple measurement points, the data processing unit 35 detects a pulse wave signal in step S119 as shown in Figure 7.

[0142] In step S711, the measuring device 10 excludes mesh points 131a whose mesh point fluctuation amount is greater than the fluctuation threshold from the measurement points. The data processing unit 35 does not use the time-series data of face feature points of mesh points 131a that have been excluded from the measurement points for pulse wave signal detection. By excluding the time-series data of face feature points of mesh points 131a whose mesh point fluctuation amount is greater than the fluctuation threshold, the accuracy of pulse wave signal detection is improved.

[0143] The data processing unit 35 generates facial feature point time-series data by tracking the mesh points 131a included in the facial image in a time series. The data processing unit 35 improves the accuracy of pulse wave signal detection by using the time-series data of facial feature points from the tracked mesh points 131a.

[0144] The data processing unit 35 preferably calculates the amount of mesh point variation of the mesh point coordinates within the multiple facial feature point data included in the facial feature point time series data, and extracts measurement point data based on the amount of mesh point variation. The measuring device 10 can improve the detection accuracy of the pulse wave signal by excluding mesh points 131a where the mesh point fluctuation amount is greater than the fluctuation threshold.

[0145] Figure 16 shows an example of the control flow performed by the measuring device 10. Figure 16 shows an example of how to set measurement points. Figure 16 shows an example of the control flow performed in step S117 shown in Figure 7. Figure 16 shows a control flow for extracting measurement point data using mesh point coordinates included in the facial feature point data.

[0146] In step S801, the measuring device 10 acquires facial feature point data. The data processing unit 35 acquires facial feature point data included in the facial feature point time series data. The data processing unit 35 acquires the coordinates Cn of the nth mesh point included in the nth facial feature point data Dn.

[0147] After acquiring facial feature point data, the measuring device 10 normalizes the mesh point coordinates in step S803. The data processing unit 35 normalizes the nth mesh point coordinate Cn for n from 1 to N. The calculation formula for normalizing the nth mesh point coordinate Cn is shown in equation (19) as an example. The normalized nth mesh point coordinate Cn is shown in the nth mesh point facial feature point data Dn in equation (19).

[0148]

number

[0149] Dnorm n This shows the normalized nth mesh point facial feature point data Dn. Here, xmax is the maximum value of xn from n 1 to N. xmin is the minimum value of xn from n 1 to N. ymax is the maximum value of yn from n 1 to N. ymin is the minimum value of yn from n 1 to N. zmax is the maximum value of zn from n 1 to N. zmin is the minimum value of zn from n 1 to N.

[0150] The measuring device 10 normalizes the mesh point coordinates and then reads the position setting range in step S805. The data processing unit 35 reads the position setting range from the storage unit 41. The position setting range is stored in the storage unit 41 in advance. The position setting range is information that specifies the setting range for the width component, height component, and depth component of the normalized mesh point coordinates. As an example, the position setting range is a range of 25% above and below the median value of each of the normalized width component, height component, and depth component. The position setting range is a setting range for setting at least one value among the width component, height component, and depth component. By using the position setting range, the data processing unit 35 can exclude mesh points 131a in areas where pulse wave signals are difficult to detect, such as the periphery of the face, around the nose, and the orbit, from the measurement points.

[0151] After reading the position setting range, the measuring device 10 determines in step S807 whether the normalized mesh point coordinates are within the position setting range. The measuring device 10 compares the normalized mesh point coordinates with the position setting range. If the data processing unit 35 determines that the normalized mesh point coordinates are within the position setting range, the measuring device 10 proceeds to step S809 (step S807: YES). If the data processing unit 35 determines that the normalized mesh point coordinates are not within the position setting range, the measuring device 10 proceeds to step S815 (step S807: NO).

[0152] In step S809, the measuring device 10 reads out the output value setting range. The data processing unit 35 reads out the output value setting range from the storage unit 41. The output value setting range is stored in the storage unit 41 in advance. The output value setting range is information that specifies the output value range of the output value associated with the mesh point coordinates. The output value setting range includes information that specifies the lower limit of at least one of the red gradation value, green gradation value, and blue gradation value included in the facial feature point data. The output value setting range is compared with the output value associated with the mesh point coordinates.

[0153] After reading the output value setting range, the measuring device 10 determines in step S811 whether the output value is within the output value setting range. In step S807, the data processing unit 35 compares the output value of the mesh point 131a whose mesh point coordinates are within the position setting range with the output value setting range. The data processing unit 35 compares the red gradation value, green gradation value, and blue gradation value included in the face feature point data of the mesh point 131a whose mesh point coordinates are within the position setting range with the output value setting range.

[0154] If the data processing unit 35 determines that the output value is within the output value setting range, the measuring device 10 proceeds to step S813 (step S811: YES). If the data processing unit 35 determines that the output value is not within the output value setting range, the measuring device 10 proceeds to step S815 (step S811: NO).

[0155] In step S813, the measuring device 10 sets a mesh point 131a as a measurement point, where the mesh point coordinates are within the position setting range and the output value is within the output value setting range. The data processing unit 35 acquires the time-series data of the facial feature points of the mesh point 131a set as the measurement point as measurement point time-series data. Using the measurement point time-series data, which includes data from multiple measurement points, the data processing unit 35 detects a pulse wave signal in step S119 as shown in Figure 7.

[0156] In step S815, the measuring device 10 excludes mesh points 131a whose mesh point coordinates are not within the position setting range and mesh points 131a whose output values ​​are not within the output value setting range from the measurement points. The data processing unit 35 does not use the facial feature point time series data of the mesh points 131a that have been excluded from the measurement points for pulse wave signal detection.

[0157] In the control flow shown in Figure 16, the measurement point is set by determining whether the output value is within the output value setting range, but this is not limited to this. The control flow for setting the measurement point does not have to include steps S809 and S811. The measuring device 10 may also determine whether the mesh point coordinates are within the position setting range and set the measurement point based on the determination result.

[0158] The control flows shown in Figures 12, 13, 14, 15, and 16 can be modified as appropriate. For example, the measuring device 10 may execute the process in step S303, followed by the process in step S301, using the control flow shown in Figure 11. The execution order of each step can be set as appropriate. [Explanation of Symbols]

[0159] 10... Measuring device, 11... Imaging unit, 13... Display unit, 15... Input unit, 31... Control unit, 33... Image recognition processing unit, 35... Data processing unit, 37... Display control unit, 41... Storage unit, 100... Acquired image, 121... Face image area, 131... Face mesh information, 131a... Mesh point, 131b... Mesh line, Db... Blue tone detection value, Dg... Green tone detection value, Dr... Red tone detection value, M... Measurer, MT... Mesh point related information, PG... Biological analysis program, S... Noise reduction signal, S1... Body movement section, S2... Resting section, TBL-01... First code name, TBL-02... Second code name, TBL-03... Third code name, TBL-04... Fourth code name.

Claims

1. An imaging unit that captures images of a living organism and generates imaging data including multiple frame image data, A face recognition unit identifies a face image contained in the frame image data and obtains three-dimensional coordinate data indicating the position of each of the multiple feature points contained in the face image. The system includes a detection unit that generates a plurality of feature point data by associating the three-dimensional coordinate data with the light intensity values ​​of pixels included in the frame image data, and detects the pulse wave signal of the living organism based on the plurality of feature point data, The aforementioned three-dimensional coordinate data includes a width component, a height component, and a depth component. The detection unit extracts measurement feature point data from a plurality of feature point data using the three-dimensional coordinate data included in the feature point data, and detects the pulse wave signal using the measurement feature point data. A device for acquiring biological information.

2. The system includes a storage unit that stores a depth range threshold that specifies the depth coordinate range of the aforementioned depth component, The detection unit extracts the measurement feature point data by comparing the depth component included in the three-dimensional coordinate data with the depth range threshold. A biological information acquisition device according to claim 1.

3. The storage unit has a planar coordinate threshold that specifies the planar coordinate range relating to the width component and the height component, The detection unit extracts the measurement feature point data using the plane coordinate threshold. The biological information acquisition device according to claim 2.

4. The storage unit stores a light intensity threshold that specifies the light intensity range of the light intensity value, The detection unit extracts the measured feature point data by comparing the light intensity value included in the feature point data with the light intensity threshold. A biological information acquisition device according to claim 2 or 3.

5. The detection unit generates a set of feature point data by tracking the feature points included in the face image in a time series. A biological information acquisition device according to claim 1.

6. The amount of variation of the three-dimensional coordinate data within a plurality of feature point data included in the feature point data set is calculated, and the measured feature point data is extracted based on the amount of variation. The biological information acquisition device according to claim 5.

7. A computer connected to an imaging unit that captures images of living organisms and generates multiple frame image data, To identify the face image contained in the frame image data, Obtain 3D coordinate data indicating the position of each of the multiple feature points included in the aforementioned facial image. Multiple feature point data are generated by associating the three-dimensional coordinate data with the light intensity values ​​of pixels included in the frame image data. Using the three-dimensional coordinate data included in the feature point data, measure feature point data is extracted from a plurality of feature point data. The pulse wave signal is detected using the aforementioned measurement feature point data. A program for acquiring biometric information.

Images