Biological information acquisition device and biological information acquisition program product

Abstract

This invention provides a biological information acquisition device and a biological information acquisition program product, which solves the following problem: A facial image region, as an example, includes areas where pulse wave signals are easily obtained and areas where they are difficult to obtain; the pulse wave measurement device may have reduced detection accuracy depending on the measurement location. The biological information acquisition device includes: a face recognition unit that recognizes a facial image contained in image data and acquires three-dimensional coordinate data representing the positions of multiple feature points; and a detection unit that generates multiple feature point data that establishes a correlation between the three-dimensional coordinate data and the light intensity values ​​of pixels contained in the image data, extracts measurement feature point data for detecting the pulse wave signal of the biological organism from the multiple feature point data using the three-dimensional coordinate data, and uses the measurement feature point data to detect the pulse wave signal.

Description

Technical Field

[0001] This disclosure relates to a biological information acquisition device and a biological information acquisition program. Background Technology

[0002] A pulse wave measuring device for measuring the pulse wave of a living organism is known. This pulse wave measuring device is an example of a biological information acquisition device. The pulse wave measuring device described in Patent Document 1 includes a receiving unit, a display control unit, and a measuring unit. The receiving unit receives execution instructions for measuring the pulse wave signal. The display control unit displays an image and a guide frame on a display unit. The measuring unit detects the skin area encompassed by the facial region contained within the guide frame. The measuring unit measures the pulse wave signal based on the skin area.

[0003] As an example of a facial region, a facial image region includes areas where pulse wave signals are easily obtained and areas where pulse wave signals are difficult to obtain. Pulse wave measurement devices have the potential for detection accuracy to decrease depending on the measurement location.

[0004] Patent document 1: Japanese Patent Application Publication No. 2021-183079. Summary of the Invention

[0005] The biological information acquisition device disclosed herein includes: an imaging unit that captures images of a biological organism to generate imaging data comprising multiple frame image data; a face recognition unit that recognizes a face image contained in the frame image data and acquires three-dimensional coordinate data representing the positions of multiple feature points contained in the face image; and a detection unit that generates multiple feature point data that establishes a correlation between the three-dimensional coordinate data and the light intensity values ​​of pixels contained in the frame image data, detects the pulse wave signal of the biological organism based on the multiple feature point data, wherein the three-dimensional coordinate data includes a width component, a height component, and a depth component, and the detection unit uses the three-dimensional coordinate data contained in the feature point data to extract measurement feature point data from the multiple feature point data, and uses the measurement feature point data to detect the pulse wave signal.

[0006] The biological information acquisition program disclosed herein causes a computer connected to a camera unit that generates frame image data by photographing a biological organism to perform the following processing: identifying a facial image contained in the frame image data; acquiring three-dimensional coordinate data representing the positions of multiple feature points contained in the facial image; generating multiple feature point data that establishes a correlation between the three-dimensional coordinate data and the light intensity values ​​of the pixels contained in the frame image data; using the three-dimensional coordinate data contained in the feature point data, extracting measurement feature point data from the multiple feature point data; and using the measurement feature point data to detect the pulse wave signal. Attached Figure Description

[0007] Figure 1 It is a diagram showing the general structure of the measuring device.

[0008] Figure 2 This is a diagram showing the modular structure of the measuring device.

[0009] Figure 3 It is a diagram representing a captured image that includes facial mesh information.

[0010] Figure 4 It is a diagram representing a portion of the facial mesh information.

[0011] Figure 5 This is an example diagram representing grayscale detection values.

[0012] Figure 6 This is a diagram illustrating an example of grid point association information.

[0013] Figure 7 This is a diagram illustrating an example of a control flow executed by a measuring device.

[0014] Figure 8 This is a graph representing the time series data of the nth facial feature point.

[0015] Figure 9 This is a diagram illustrating an example of the analytical steps involved in detecting pulse wave signals.

[0016] Figure 10 This is a graph representing the analysis results of the noise-removed signal.

[0017] Figure 11 This is a diagram illustrating an example of a control flow executed by a measuring device.

[0018] Figure 12 This is a diagram illustrating an example of a control flow executed by a measuring device.

[0019] Figure 13 This is a diagram illustrating an example of a control flow executed by a measuring device.

[0020] Figure 14 This is a diagram illustrating an example of a control flow executed by a measuring device.

[0021] Figure 15 This is a diagram illustrating an example of a control flow executed by a measuring device.

[0022] Figure 16 This is a diagram illustrating an example of a control flow executed by a measuring device. Detailed Implementation

[0023] Figure 1A general structure of the measuring device 10 is shown. The measuring device 10 corresponds to an example of a biological information acquisition device. The measuring device 10 uses video data to detect the pulse wave signal of the measurer M. The pulse wave signal is a signal representing the pulse wave of the measurer M. The measurer M corresponds to an example of a biological organism. The measuring device 10 calculates biological information of the measurer M based on the pulse wave signal. The biological information includes pulse, pulse variability, oxygen saturation concentration, blood pressure, etc. The measuring device 10 can also evaluate sleep apnea syndrome, etc., based on the biological information. The measuring device 10 displays the biological information calculated based on the pulse wave signal.

[0024] The measuring device 10 is composed of information processing devices such as personal computers. Figure 1 The measuring device 10 shown is a laptop computer, but it is not limited to this. The measuring device 10 can be any device with video recording capabilities or capable of connecting to a video recording device. The measuring device 10 can be a desktop computer, tablet computer, smartphone, etc. The measuring device 10 includes a recording unit 11, a display unit 13, and an input unit 15. The measuring device 10 may also include a communication unit (not shown).

[0025] The capturing unit 11 captures images of the measuring person M by receiving reflected light, external light, etc., reflected by the measuring person M, as detection light. The capturing unit 11 generates video data containing the face of the measuring person M. The video data is data used to display the video and consists of multiple image data. This video data corresponds to an example of capturing data. The video data consists of multiple image data. The capturing unit 11 generates video data containing multiple image data. The image data corresponds to an example of frame image data. The image data consists of multiple output values ​​output in pixels. The image data is data used to display the captured image 100 on the display unit 13. The captured image 100 is a still image. The capturing unit 11 captures images at a predetermined frame rate to generate video data. The capturing unit 11 corresponds to an example of a capturing unit.

[0026] As an example, the imaging unit 11 is a camera including optical elements, an imaging element, etc. The optical elements focus light onto the imaging element. The imaging element converts the detected light into an output value of an electrical signal. The imaging element generates output values ​​for each of the multiple pixels. The output value of each pixel represents the intensity of light at that pixel. The imaging element generates the output value for each pixel. The imaging element is a CCD (Charge Coupled Device), CMOS (Complementary Metal Oxide Semiconductor), etc. The output values ​​contain grayscale values ​​of multiple colors of light. The imaging unit 11 generates grayscale values ​​of multiple colors of light per pixel. As an example, the multiple colors of light are red light, green light, and blue light. Red light, green light, and blue light are light in different wavelength bands. The red wavelength band is 600nm–800nm. The green wavelength band is 520nm–550nm. The blue wavelength band is 430nm–490nm. The imaging unit 11 may also include IR (Infrared) light, etc. The imaging unit 11 generates, for each pixel, a red grayscale value as the grayscale value of red light, a green grayscale value as the grayscale value of green light, and a blue grayscale value as the grayscale value of blue light. The image data consists of the output value of each pixel, including the red, green, and blue grayscale values. The image data includes the brightness of each pixel, etc.

[0027] Figure 1 The shooting unit 11 shown is a camera built into the measuring device 10, but it is not limited to this. The shooting unit 11 can also be an external camera connected to the measuring device 10. External cameras include near-infrared cameras, webcams, smartphone cameras, etc.

[0028] Display unit 13 displays various information such as the captured image 100. Display unit 13 displays various biological information based on pulse wave signals. Display unit 13 can also display comments based on biological information. Display unit 13 is composed of a liquid crystal panel, an organic EL (electro-luminescence) panel, etc. Display unit 13 can also have touch input functionality. When it has touch input functionality, display unit 13 functions as input unit 15. Figure 1 The display unit 13 shown is included in the measuring device 10, but is not limited thereto. The display unit 13 may also be an external display attached to the measuring device 10.

[0029] Input unit 15 accepts various input operations from the measurer M. Input unit 15 generates various input signals corresponding to the input operations. Figure 1The input unit 15 shown is the keyboard included in the measuring device 10, but it is not limited to this. The input unit 15 can also be a mouse, keyboard, touch panel, graphics tablet, etc., connected to the measuring device 10.

[0030] The user M operates the measuring device 10 from a position opposite the imaging unit 11 of the measuring device 10. The user M operates the measuring device 10 while it detects biometric information. The user M can also operate the measuring device 10 while performing tasks such as document creation. While the user M is performing tasks such as document creation, the measuring device 10 detects the user M's biometric information in the background. By detecting biometric information in the background, the measuring device 10 is able to detect the biometric information of the user M in his normal active state.

[0031] Figure 2 The modular structure of the measuring device 10 is shown. The measuring device 10 includes an image capturing unit 11, a display unit 13, an input unit 15, a control unit 31, and a storage unit 41.

[0032] The capturing unit 11 sends video data to the control unit 31. The capturing unit 11 sends video data to the control unit 31 at predetermined time intervals. The capturing unit 11 can also send image data contained in the video data to the control unit 31 at predetermined time intervals. The capturing unit 11 sends image data containing the red, green, and blue grayscale values ​​of each pixel to the control unit 31. The capturing unit 11 can also send video data to the storage unit 41, causing the storage unit 41 to store the video data.

[0033] Display unit 13 displays various images based on the control of control unit 31. Display unit 13 receives display data from control unit 31 and displays various images based on the display data. Display unit 13 can also display video captured by shooting unit 11 based on video data. Display unit 13 can also display captured image 100 based on image data contained in video data.

[0034] Input unit 15 sends input signals to control unit 31. Input unit 15 causes control unit 31 to perform various controls by sending input signals to control unit 31. As an example, input unit 15 sends a display indication signal to control unit 31 to display organism information. The display indication signal is an example of an input signal. Control unit 31 generates organism information display data based on the display indication signal, causing display unit 13 to display organism information based on pulse wave signals. Control unit 31 sends the organism information display data to display unit 13. Display unit 13 displays a screen containing organism information based on the organism information display data.

[0035] The control unit 31 is a controller that controls the actions of various units. As an example, the control unit 31 is a processor with a CPU (Central Processing Unit). The control unit 31 is composed of one or more processors. The control unit 31 is connected to the imaging unit 11, display unit 13, etc., in a communicative manner. The control unit 31 functions as an image recognition processing unit 33, a data processing unit 35, and a display control unit 37 by executing the biometric analysis program PG. The control unit 31 can also function as a functional unit other than the image recognition processing unit 33, the data processing unit 35, and the display control unit 37 by executing the biometric analysis program PG. The control unit 31 corresponds to an example of a computer.

[0036] The image recognition processing unit 33 acquires video data sent from the capturing unit 11. The image recognition processing unit 33 acquires multiple image data contained in the video data. The image recognition processing unit 33 identifies facial images contained in the image data. The image recognition processing unit 33 identifies facial images by performing facial recognition processing on each image data. Facial recognition processing is the process of extracting the facial image region 121 by detecting facial image feature points contained in the image data and matching these facial image feature points with a pre-registered facial image database. The facial image database is a database that stores information related to facial image feature points used for facial recognition. As an example, the facial image feature points contained in the image data are the position and outline of the eyes, nose, and mouth. The facial image database is pre-stored in the storage unit 41. If the measuring device 10 is connected to a server via a network, the facial image database can also be pre-stored in the server. The facial image region 121 is the region displaying the face of the person being measured, M. The image recognition processing unit 33 identifies the facial image region 121 by performing facial recognition processing. Image recognition processing unit 33 corresponds to an example of face recognition unit.

[0037] In face recognition processing, as an example, FaceMesh (hereinafter referred to as FaceMesh), which is included in MediaPipe (a multimedia machine learning framework) provided by Google, is used. FaceMesh is a machine learning model that detects key points of the face from an image. The image recognition processing unit 33 performs face recognition processing using FaceMesh on multiple image data respectively, and obtains face image region 121 and face mesh information 131. The face mesh information 131 is represented by multiple key points included in the face image region 121.

[0038] Figure 3 An image 100 including facial grid information 131 is shown. Figure 3An example of facial mesh information 131 is shown. Facial mesh information 131 varies depending on the measurer M. Figure 3 The grid points 131a and grid lines 131b representing the face grid information 131 are shown.

[0039] Figure 3 The XYZ coordinate system is shown. The X-axis is the axis along the width direction of the captured image 100. The Y-axis is the axis along the height direction of the captured image 100. The Z-axis is the axis along the vertical axis passing through the center of the capturing element within the capturing unit 11.

[0040] Grid point 131a is a point corresponding to a key point. As an example, there are 468 key points. The image recognition processing unit 33 determines the multiple grid points 131a contained within the face image region 121. The image recognition processing unit 33 determines multiple grid points 131a for each of the multiple image data. The number of grid points 131a determined from the multiple image data is the same. A predetermined code is assigned to each of the multiple grid points 131a. The multiple grid points 131a assigned the same code correspond to the same position within the face image region 121. Grid point 131a corresponds to an example of a feature point. Grid line 131b is a line connecting two adjacent grid points 131a.

[0041] Grid point 131a is represented by three-dimensional coordinates including the width component, height component, and depth component, with one of the grid points 131a as the origin. Figure 3 The grid points 131a in three-dimensional coordinates are projected onto a two-dimensional plane. Multiple grid points 131a are represented by grid point coordinates that include width, height, and depth components. The grid point coordinates correspond to an example of three-dimensional coordinate data.

[0042] Figure 4 A portion of the face mesh information 131 is shown. Figure 4 A magnified view shows the facial mesh information 131 near the left eye of the measurer M. Figure 4 Multiple grid points 131a and multiple grid lines 131b are shown. Figure 4 The code names of a portion of the multiple grid points 131a are shown.

[0043] Figure 4 The code names assigned to a subset of grid points 131a are shown. As an example, the code names assigned to grid point 131a are first code name TBL-01, second code name TBL-02, third code name TBL-03, and fourth code name TBL-04. Figure 4The code name shown represents the tear duct of the left eye of the person measuring M. The structure of the code name can be appropriately set. The image recognition processing unit 33 assigns a unique code name to each of the multiple grid points 131a.

[0044] Figure 2 The image recognition processing unit 33 shown calculates grid point coordinates representing the positions of each of the plurality of grid points 131a. The image recognition processing unit 33 calculates relative grid point coordinates with any point among the plurality of grid points 131a as the origin. The image recognition processing unit 33 converts the relative grid point coordinates into grid point coordinates on the image data. The method for calculating the grid point coordinates will be described later.

[0045] Figure 2 The data processing unit 35 shown acquires video data transmitted from the imaging unit 11. The data processing unit 35 acquires multiple image data contained in the video data. The data processing unit 35 acquires the red grayscale value, green grayscale value, blue grayscale value, and pixel brightness of each pixel contained in the image data. The data processing unit 35 performs appropriate correction processing and other data processing on the red grayscale value, green grayscale value, and blue grayscale value.

[0046] The data processing unit 35 detects the pulse wave signal of the person being measured, M. The data processing unit 35 analyzes biological information based on the pulse wave signal. The data processing unit 35 corresponds to an example of the detection unit. The data processing unit 35 acquires image data from the imaging unit 11, consisting of the output values ​​of each pixel containing red, green, and blue grayscale values. The data processing unit 35 acquires facial grid information 131 from the image recognition processing unit 33. The data processing unit 35 establishes a correspondence between the grid points 131a contained in the facial grid information 131 and the pixel output values. An example of an output value corresponding to a light intensity value is provided. The data processing unit 35 establishes a correspondence between the grid points 131a and the pixel output values ​​for each of the multiple image data sets.

[0047] The data processing unit 35 tracks predetermined grid points 131a contained in the facial grid information 131 of each of the multiple image data. The multiple image data are generated in a time sequence. The data processing unit 35 tracks the position of the predetermined grid points 131a contained in each of the multiple image data generated in a time sequence by determining the predetermined grid points 131a in each of the multiple image data.

[0048] The data processing unit 35 can track either one grid point 131a from a plurality of grid points 131a, or all of the plurality of grid points 131a. The data processing unit 35 can also track one or more grid points 131a that have been pre-selected based on predetermined conditions from the plurality of grid points 131a. The selection of grid points 131a is appropriately set.

[0049] The data processing unit 35 establishes a correspondence between predetermined grid points 131a and pixel output values ​​for each image data. The data processing unit 35 obtains the output values ​​that have established a correspondence with the predetermined grid points 131a. The data processing unit 35 obtains the output values ​​that have established a correspondence with the predetermined grid points 131a for each image data. By obtaining the output values ​​that have established a correspondence with the predetermined grid points 131a for each image data generated according to a time series, the data processing unit 35 obtains the time series output values ​​corresponding to the predetermined grid points 131a as facial feature point time series data.

[0050] The data processing unit 35 uses time-series facial feature points that correspond to predetermined grid points 131a to detect the pulse wave signal of the person being measured, M. The data processing unit 35 generates detection values ​​using multiple output values ​​contained in the facial feature point time-series data. The detection values ​​include calculated grayscale detection values. The detection values ​​can be either time-series facial feature points that correspond to one grid point 131a, or values ​​calculated based on multiple time-series facial feature points that correspond to multiple grid points 131a respectively. The calculated values ​​are the average of the output values ​​that correspond to multiple grid points 131a for each image data. Detection values ​​are calculated for each image data within the video data. The data processing unit 35 obtains the pulse wave signal based on the detection values ​​of predetermined grid points 131a for each of the multiple image data.

[0051] The data processing unit 35 uses the grayscale detection values ​​included in the detection values ​​to detect the pulse wave signal. As an example, the data processing unit 35 uses at least one of a red grayscale detection value Dr, a green grayscale detection value Dg, and a blue grayscale detection value Db to detect the pulse wave signal. The red grayscale detection value Dr is calculated using red grayscale values. The green grayscale detection value Dg is calculated using green grayscale values. The blue grayscale detection value Db is calculated using blue grayscale values. As an example, the pulse wave signal is detected based on the green grayscale detection value Dg. The pulse wave signal can also be detected based on the difference between at least one of the red grayscale detection value Dr and the blue grayscale detection value Db and the green grayscale detection value Dg.

[0052] Figure 5 An example of a grayscale detection value is shown. Figure 5 The red grayscale detection value Dr, the green grayscale detection value Dg, and the blue grayscale detection value Db are shown. Figure 5 The waveform signals show the changes of the red grayscale detection value Dr, the green grayscale detection value Dg, and the blue grayscale detection value Db over time. Figure 5The diagram shows the red grayscale detection values ​​Dr, Dg, and Db of the body movement interval S1, and the red grayscale detection values ​​Dr, Dg, and Db of the quiet interval S2. The body movement interval S1 is the interval where facial movements and expressions occur. The quiet interval S2 is the interval where facial movements and expressions change by a smaller amount than a predefined range.

[0053] Figure 5 The green grayscale detection value Dg is shown. The green grayscale detection value Dg corresponds to the amount of green light contained in the output value at one or more predetermined grid points 131a. The green grayscale detection value Dg is included in the detection value.

[0054] like Figure 5 As shown, in the body movement interval S1, the green grayscale detection value Dg varies due to the influence of body movement. The pulse wave signal contained in the green grayscale detection value Dg is difficult to detect due to the changing noise. In the quiet interval S2, the influence of the changing noise caused by body movement on the green grayscale detection value Dg is reduced, and the pulse wave signal can be detected.

[0055] Figure 5 The red grayscale detection value Dr is shown. The red grayscale detection value Dr corresponds to the amount of red light contained in the output value at one or more predetermined grid points 131a. The red grayscale detection value Dr is included in the detection value.

[0056] like Figure 5 As shown, in the body movement interval S1, the red grayscale detection value Dr changes due to the influence of body movement. The pulse wave signal contained in the red grayscale detection value Dr is difficult to detect due to the changing noise. In the quiet interval S2, the influence of the changing noise caused by body movement on the red grayscale detection value Dr is reduced, but the SN ratio is small, making it difficult to detect the pulse wave signal.

[0057] Figure 5 The blue grayscale detection value Db is shown. The blue grayscale detection value Db corresponds to the amount of blue light contained in the output value at one or more predetermined grid points 131a. The blue grayscale detection value Db is included in the detection value.

[0058] like Figure 5 As shown, in the body movement interval S1, the blue grayscale detection value Db varies due to the influence of body movement. The pulse wave signal contained in the blue grayscale detection value Db is difficult to detect due to the changing noise. In the quiet interval S2, the influence of the changing noise caused by body movement on the blue grayscale detection value Db is reduced, but the SN ratio is small, making it difficult to detect the pulse wave signal.

[0059] Data Processing Department 35 uses Figure 5 The red grayscale detection value Dr, the green grayscale detection value Dg, and the blue grayscale detection value Db shown are used to detect the pulse wave signal. The data processing unit 35 detects the pulse wave signal according to the analysis steps described later.

[0060] The data processing unit 35 calculates biological information such as pulse by calculating the period and amplitude of the pulse wave signal. The data processing unit 35 sends the biological information including the pulse wave signal to the display control unit 37. The data processing unit 35 may also enable the storage unit 41 to store the biological information.

[0061] Figure 2 The display control unit 37 controls the display of the display unit 13. The display control unit 37 obtains biological information including pulse wave signals from the data processing unit 35. The display control unit 37 generates biological information display data containing the biological information. The display control unit 37 sends the biological information display data to the display unit 13. The display control unit 37 causes the display unit 13 to display the biological information display data. The display control unit 37 can notify the observer M of the detection result of the biological information by causing the display unit 13 to display the biological information display data.

[0062] The display control unit 37 can also generate message data indicating the operation status of the organism analysis program PG. The message data includes start messages, execution messages, and end messages. A start message indicates the start of organism information detection. An execution message indicates that organism information detection is in progress. An end message indicates that organism information detection has ended. The display control unit 37 sends the message data to the display unit 13. The display control unit 37 then causes the display unit 13 to display the message data.

[0063] Storage unit 41 stores various programs and data. Storage unit 41 corresponds to an example of a storage unit. Storage unit 41 stores the biological analysis program PG and grid point association information MT. Storage unit 41 stores document creation programs, spreadsheet calculation programs, etc. Storage unit 41 can also store various detection data such as video data and biological information. Storage unit 41 can also store a facial image database. Storage unit 41 is composed of semiconductor memory such as RAM (Random Access Memory) and ROM (Read Only Memory). Storage unit 41 can also have an HDD (Hard Disk Drive). Storage unit 41 can also function as the working area of ​​control unit 31.

[0064] The bioanalysis program PG is a program that enables the measuring device 10 to detect pulse wave signals. The bioanalysis program PG is executed by the control unit 31. By executing the bioanalysis program PG, the control unit 31 functions as various functional units. The bioanalysis program PG detects various biological information based on pulse wave signals. The bioanalysis program PG can also run in the background while the control unit 31 is executing document creation programs, etc. The bioanalysis program PG corresponds to an example of a biological information acquisition program.

[0065] The grid point association information MT is information about grid points 131a that are adjacent to the plurality of grid points 131a. The grid point association information MT is generated during face recognition processing by the image recognition processing unit 33 and stored in the storage unit 41. The grid point association information MT is used by the data processing unit 35 when selecting one or more grid points 131a from the plurality of grid points 131a.

[0066] Figure 6 An example of grid point association information (MT) is shown. Figure 6 The grid point association information MT is shown in tabular form. Figure 6 It shows the relationship with Figure 4 Information related to grid point 131a of the first code name TBL-01, the second code name TBL-02, the third code name TBL-03, and the fourth code name TBL-04.

[0067] The grid point association information MT shows the grid points 131a adjacent to each grid point 131a. Two adjacent grid points 131a are connected by grid lines 131b. The grid point association information MT shows two or more grid points 131a connected by grid lines 131b. As an example, the grid point association information MT shows that grid point 131a of the first code name TBL-01 is located adjacent to grid point 131a of the second code name TBL-02. The grid point association information MT shows that grid point 131a of the second code name TBL-02 is located adjacent to grid points 131a of the first code name TBL-01 and grid points 131a of the third code name TBL-03.

[0068] Figure 7 An example of a control flow executed by the measuring device 10 is shown. Figure 7 The control flow for obtaining a pulse wave signal using the output value of the pixel corresponding to grid point 131a is shown. The control flow is executed by causing the bioanalysis program PG to perform actions. Figure 7 The control flow is illustrated in a flowchart.

[0069] In step S101, the measuring device 10 acquires video data. The measuring device 10 causes the imaging unit 11 to generate video data. The imaging unit 11 captures images of the observer M, generating video data. The video data contains multiple image data. Multiple image data are generated in a time sequence. The imaging unit 11 generates multiple image data. As an example, when the frame rate of the imaging unit 11 generating video data is 30 fps (frames per second) and the measurement time is 8 seconds, the number of image data becomes 240. The imaging unit 11 sends the video data to the control unit 31.

[0070] After acquiring video data, the measuring device 10 begins acquiring time-series data of facial feature points in step S103. The data processing unit 35 of the control unit 31 acquires the video data sent from the capturing unit 11. The control unit 31 acquires multiple image data contained in the video data. The data processing unit 35 acquires time-series data of facial feature points 131a for each grid point by performing steps S103 to S111. Details of the facial feature point time-series data will be described later.

[0071] In step S105, the measuring device 10 acquires image data. The image recognition processing unit 33 of the control unit 31 sequentially acquires image data generated in a time sequence. When the video data contains k image data, the image recognition processing unit 33 acquires them sequentially from the first image data to the kth image data. k is any integer. k is set according to the frame rate of the video data and the measurement time.

[0072] After acquiring image data, the measuring device 10 performs face recognition processing in step S107. The image recognition processing unit 33 performs face recognition processing on multiple image data respectively. The image recognition processing unit 33 generates face grid information 131 contained in each image data. The face grid information 131 includes multiple grid points 131a and multiple grid lines 131b. If the face image region 121 is contained in each image data, the number of multiple grid points 131a contained in each image data is the same.

[0073] After performing face recognition processing, the measuring device 10 obtains the grid point coordinates of each grid point 131a in step S109. The image recognition processing unit 33 obtains the face grid information 131 of each image data. The image recognition processing unit 33 obtains the grid point coordinates of the multiple grid points 131a contained in each image data. The image recognition processing unit 33 generates relative grid point coordinates with a predetermined grid point 131a in the face grid information 131 as the origin. The face grid information 131 includes N grid points 131a, including the nth grid point 131a. The relative grid point coordinates of the nth grid point 131a in an image data, i.e., the nth relative grid point coordinate Crn, are represented by the following formula (1).

[0074]

[0075] Here, A is the distance in the width direction from grid point 131a, which is set as the origin, and is an example of the width component. B is the distance in the height direction from grid point 131a, which is set as the origin, and is an example of the height component. W is the distance in the depth direction from grid point 131a, which is set as the origin. n is any integer from 1 to N. N is an integer greater than 2.

[0076] After obtaining the coordinates Crn of the nth relative grid point, the image recognition processing unit 33 converts them into grid point coordinates on the image data with any position within the image data as the origin. The coordinates Cn of the nth grid point on the image data are represented by the following equation (2).

[0077]

[0078] Here, x is the distance along the X-axis from the origin within the image data. xn is the distance along the X-axis from the origin within the image data for grid point 131a at coordinate Cn of the nth grid point. x and xn are examples of the width component. y is the distance along the Y-axis from the origin in the image data. yn is the distance along the Y-axis from the origin in the image data for grid point 131a at coordinate Cn of the nth grid point. y and yn are examples of the height component. z is a value calculated based on W. As an example, z is the difference from the origin by averaging the W values ​​of multiple grid points 131a. z is calculated by converting the difference to the same units as x and y. zn is the difference from the average value to grid point 131a at coordinate Cn of the nth grid point. z and zn are examples of the depth component.

[0079] Each of the multiple image data sets contains an nth grid point 131a, representing the same location within the face image region 121. The coordinates Cn of the nth grid point of each of the multiple image data sets vary depending on the position and orientation of the face of the observer M in the image data. The control unit 31 can track the nth grid point 131a in a time sequence by acquiring the coordinates Cn of the nth grid point contained in each image data set. The control unit 31 can track the positions of all grid points 131a in a time sequence by acquiring the coordinates Cn of the nth grid point of each of the multiple image data sets.

[0080] After obtaining the coordinates of each grid point, the measuring device 10 obtains the output value of each grid point coordinate in step S111. The data processing unit 35 obtains the grid point coordinates from the image recognition processing unit 33. The data processing unit 35 obtains the output value of the pixel corresponding to the grid point coordinate. The data processing unit 35 may also obtain the output value of the pixel corresponding to the grid point coordinate and the pixels in a predetermined area relative to the pixel corresponding to the grid point coordinate, i.e., the surrounding pixels. The predetermined area is preset. When obtaining the output value of the surrounding pixels, as an example, the data processing unit 35 obtains the average value of the output value of the pixel corresponding to the grid point coordinate and the output value of the surrounding pixels as the output value. The output value of the coordinates Cn of the nth grid point in an image data, i.e., the output value Bn of the nth grid point, is represented by the following formula (3).

[0081]

[0082] Here, rn is the red grayscale value contained in the output value of the grid point coordinates corresponding to the nth grid point coordinate Cn. gn is the green grayscale value contained in the output value of the grid point coordinates corresponding to the nth grid point coordinate Cn. bn is the blue grayscale value contained in the output value of the grid point coordinates corresponding to the nth grid point coordinate Cn.

[0083] The data processing unit 35 generates nth facial feature point data Dn, which establishes a relationship between the coordinates Cn of the nth grid point in an image data and the output value Bn of the nth grid point. The nth facial feature point data Dn is an example of facial feature point data. The facial feature point data corresponds to an example of feature point data. The data processing unit 35 generates the nth facial feature point data Dn with the coordinates Cn of the nth grid point shown in the following equation (4). The nth facial feature point data Dn is generated for each of the N grid point coordinates.

[0084]

[0085] The data processing unit 35 acquires the nth facial feature point data Dn for each image data. The data processing unit 35 generates the nth facial feature point time series data Dn(t) by tracking the nth facial feature point data Dn for each image data. The nth facial feature point time series data Dn(t) is an example of the facial feature point time series data with the coordinates Cn of the nth grid point. The facial feature point time series data corresponds to an example of a feature point data group. The nth facial feature point time series data Dn(t) includes the nth facial feature point data Dn(T) acquired at a frame rate of dt at any time T, the nth facial feature point data Dn(T-dt) at time T-dt, and the nth facial feature point data Dn(T+dt) at time T+dt. The nth facial feature point time series data Dn(t) is as follows: Figure 8 As shown.

[0086] The data processing unit 35 obtains the nth facial feature point data Dn by acquiring the coordinates Cn of the nth grid point for each image data, thereby obtaining... Figure 8 The nth facial feature point time series data Dn(t) is shown. The nth facial feature point time series data Dn(t) is the facial feature point time series data of the nth grid point 131a tracked for each image data.

[0087] After obtaining the output values ​​of the coordinates of each grid point, the measuring device 10 determines in step S113 whether it has obtained the time series data of facial feature points for all grid point coordinates. The data processing unit 35 determines whether it has obtained the time series data Dn(t) of all nth facial feature points from n1 to N. If the data processing unit 35 determines that it has obtained all the time series data Dn(t) of the nth facial feature points, the measuring device 10 proceeds to step S115 (step S113: Yes). When the data processing unit 35 determines that it has not obtained all the time series data Dn(t) of the nth facial feature points, the measuring device 10 returns to step S109 (step S113: No). The measuring device 10 continues to obtain the time series data Dn(t) of the nth facial feature points.

[0088] In step S115, the measuring device 10 finishes acquiring the time series data of facial feature points. The data processing unit 35 acquires the time series data Dn(t) of the nth facial feature point, where n is 1 to N. The data processing unit 35 acquires the time series data Dn(t) of the nth facial feature point, which includes the data Dn of the nth facial feature point.

[0089] After acquiring facial feature point time-series data, the measuring device 10 selects measurement points in step S117. The data processing unit 35 selects one or more grid points 131a from a plurality of grid points 131a as measurement points. The measuring device 10 acquires facial feature point time-series data corresponding to the measurement points as measurement point time-series data. The measurement point time-series data corresponds to an example of a feature point data group. The measurement point time-series data contains multiple measurement point data as facial feature point data corresponding to the measurement points. The measurement point data corresponds to an example of measurement feature point data. The data processing unit 35 extracts the facial feature point data of the measurement points, i.e., the measurement point data, by setting the measurement points. The method for setting the measurement points will be described later.

[0090] After setting the measurement points, the measuring device 10 detects the pulse wave signal in step S119. The data processing unit 35 uses measurement point time series data or measurement point data contained in the measurement point time series data to detect the pulse wave signal. The measurement point time series data contains multiple measurement point data that correspond to predetermined grid points 131a. The data processing unit 35 uses the measurement point data to generate a detection value. As an example, the data processing unit 35 uses the measurement point time series data to detect... Figure 5 The red grayscale detection value Dr, the green grayscale detection value Dg, and the blue grayscale detection value Db are shown.

[0091] Figure 9 An example of the analytical steps for detecting pulse wave signals is shown. Figure 9 An example of the analysis steps is shown in a flowchart. Figure 9 The analysis steps shown are performed by the data processing unit 35. Figure 9 The analysis steps shown use red grayscale detection value Dr, green grayscale detection value Dg, and blue grayscale detection value Db to detect pulse wave signals.

[0092] In step S201, the data processing unit 35 samples each grayscale detection value at predetermined time intervals. The time interval and sampling frequency are appropriately set. Preferably, the time interval is the time of a pulse wave containing one or more pulses. As an example, the time interval is 3 to 10 seconds. As an example, the sampling frequency is 10 Hz or higher and 50 Hz or lower. The data processing unit 35 acquires sampled data by performing sampling. The sampled data includes the sampled red grayscale detection value Dr, green grayscale detection value Dg, and blue grayscale detection value Db.

[0093] After performing sampling, the data processing unit 35 normalizes the sampled data in step S203. The data processing unit 35 normalizes the green grayscale detection value Dg contained in the sampled data.

[0094] The data processing unit 35 calculates the average value of multiple green grayscale detection values ​​Dg, i.e., the green average value Gmean, and the standard deviation value of multiple green grayscale detection values ​​Dg, i.e., the green standard deviation value Gstd. The data processing unit 35 normalizes each green grayscale detection value Dg using the following formula (5).

[0095]

[0096] Here, m is any integer greater than or equal to 1. Gm is the m-th green grayscale detection value Dg. Gnorm m It is the normalized value of the m-th green grayscale detection value Dg.

[0097] Similar to the green grayscale detection value Dg, the data processing unit 35 normalizes the multiple red grayscale detection values ​​Dr and multiple blue grayscale detection values ​​Db contained in the sampled data. The data processing unit 35 calculates the average value of the multiple red grayscale detection values ​​Dr, i.e., the red average value Rmean, and the standard deviation value of the multiple red grayscale detection values ​​Dr, i.e., the red standard deviation value Rstd. The data processing unit 35 calculates the average value of the multiple blue grayscale detection values ​​Db, i.e., the blue average value Bmean, and the standard deviation value of the multiple blue grayscale detection values ​​Db, i.e., the blue standard deviation value Bstd. The data processing unit 35 normalizes each red grayscale detection value Dr and each blue grayscale detection value Db using the following equations (6) and (7).

[0098]

[0099] Here, m is any integer greater than or equal to 1. Rm is the m-th red grayscale detection value Dr. Rnorm m Bm is the normalized value of the m-th red grayscale detection value Dr. Bm is the m-th blue grayscale detection value Db. m It is the normalized value of the m-th blue grayscale detection value Db.

[0100] After normalizing the sampled data, the data processing unit 35 performs noise removal processing in step S205. The data processing unit 35 uses the normalized green grayscale detection value Dg, the normalized red grayscale detection value Dr, and the blue grayscale detection value Db to perform noise removal processing. The data processing unit 35 uses the following formula (8) to perform noise removal processing and generate a noise removal signal S.

[0101]

[0102] Here, m is any integer greater than or equal to 1. Sm is the m-th noise-removed signal S. α is the first coefficient, and β is the second coefficient.

[0103] As an example, α and β are -0.5 and -0.5, respectively. When α and β are negative, the data processing unit 35 detects the noise removal signal S by subtracting the normalized red grayscale detection value Dr and the normalized blue grayscale detection value Db from the normalized green grayscale detection value Dg. At least one of α and β can also be 0. When α = 0 and β = -0.5, the data processing unit 35 detects the noise removal signal S by calculating the difference between the green grayscale detection value Dg and the red grayscale detection value Dr. When α = -0.5 and β = 0, the data processing unit 35 detects the noise removal signal S by calculating the difference between the green grayscale detection value Dg and the blue grayscale detection value Db. α and β are appropriately set according to the noise removal status.

[0104] Figure 10 The analysis results of the noise-removed signal S are shown. Figure 10 based on Figure 5 The red grayscale detection value Dr, the green grayscale detection value Dg, and the blue grayscale detection value Db shown are analyzed. Figure 10 The noise removal signal S is shown when α=-0.5 and β=-0.5 are substituted into equation (8). Figure 10 The noise-removed signal S is shown for the body movement range S1 and the quiet range S2.

[0105] like Figure 10 As shown, the noise removal signal S corresponds to the pulse wave signal. Noise components such as body movements are removed from the noise removal signal S. The data processing unit 35 detects the noise removal signal S as a pulse wave signal. The noise removal signal S in the quiet zone S2 is clearly detected by comparing it with the green grayscale detection value Dg. The noise removal signal S in the body movement zone S1 is adjusted to a signal waveform corresponding to the pulse wave signal. By performing noise removal processing, the data processing unit 35 is able to detect the pulse wave signal in both the body movement zone S1 and the quiet zone S2.

[0106] The data processing unit 35 can also use the noise-removed signal S to calculate biological information such as pulse waves. The data processing unit 35 acquires the noise-removed signal S as a pulse wave signal. The data processing unit 35 calculates biological information such as pulse by calculating the period and amplitude of the pulse wave signal. The data processing unit 35 sends the biological information including the pulse wave signal to the display control unit 37. The data processing unit 35 can also cause the storage unit 41 to store the biological information, etc.

[0107] The data processing unit 35 uses one or any number of nth facial feature point time series data Dn(t) from N nth facial feature point time series data Dn(t) where n is 1 to N to obtain the pulse wave signal.

[0108] Figure 11An example of a control flow executed by the measuring device 10 is shown. Figure 11 An example of how to set up measurement points is shown. Figure 11 It shows in Figure 7 An example of the control flow executed in step S117 is shown. Figure 11 The control flow for setting measurement points using the depth component contained in the grid point coordinates is shown.

[0109] In step S301, the measuring device 10 acquires facial feature point data. The data processing unit 35 acquires facial feature point data contained in the facial feature point time series data. The data processing unit 35 acquires zn contained in the nth facial feature point data Dn. zn is the depth component of the coordinate Cn of the nth grid point.

[0110] After acquiring facial feature point data, the measuring device 10 reads the depth setting range in step S303. The data processing unit 35 reads the depth setting range stored in the storage unit 41. The depth setting range is numerical range information that specifies the range of depth coordinates of the depth components contained in the grid point coordinates of the grid point 131a extracted as the measurement point. The depth setting range corresponds to an example of a depth range threshold. The depth setting range is pre-stored in the storage unit 41. For example, as an example, the depth setting range is set by a minimum depth coordinate value zmin and a maximum depth coordinate value zmax. The minimum depth coordinate value zmin is the lower limit of the depth coordinate range. The maximum depth coordinate value zmax is the upper limit of the depth coordinate range.

[0111] After reading the depth setting range, the measuring device 10 determines in step S305 whether the depth component contained in the grid point coordinates is within the depth setting range. The data processing unit 35 compares the depth component contained in the grid point coordinates with the depth setting range. The data processing unit 35 determines whether the depth component contained in the grid point coordinates of each grid point 131a is related to the following formula (9).

[0112]

[0113] As an example, the average value of multiple depth components is set to 0. Positions closer to the shooting unit 11 than the position that becomes the average value of the depth components are negative, while positions farther away from the shooting unit 11 are positive. The value is less than 0 for the edges of the face. The depth components of the nose tip and bridge of the face are greater than 0. The output values ​​of the edges of the face, the nose tip, and the bridge of the nose are easily affected by factors such as the orientation of the face. The output values ​​of the edges of the face, the nose tip, and the bridge of the nose are excluded from the detection of pulse wave signals based on the depth setting range. By using grid point 131a with grid point coordinates within the depth setting range as the measurement point, the detection accuracy of pulse wave signals is improved.

[0114] When the zn contained in the coordinates Cn of the nth grid point is in the relationship of equation (9), the data processing unit 35 determines that the depth component is within the depth setting range. The measuring device 10 proceeds to step S307 (step S305: Yes). When the zn contained in the coordinates Cn of the nth grid point is not in the relationship of equation (9), the data processing unit 35 determines that the depth component is not within the depth setting range. The measuring device 10 proceeds to step S309 (step S305: No).

[0115] In step S305, the measuring device 10 may also compare the normalized zn with the depth setting range after normalizing zn. The depth setting range is set to the numerical range corresponding to the normalized zn. The data processing unit 35 normalizes zn using the following formula (10).

[0116]

[0117] Here, znorm n This represents the normalized zn. zmax is the maximum value of zn contained in the facial feature point data. zmin is the minimum value of zn contained in the facial feature point data.

[0118] As an example, the depth setting range corresponding to the normalized zn is a range of 25% above and below the center value. The depth setting range is appropriately set and stored in the storage unit 41. The data processing unit 35 determines whether the normalized zn is within the depth setting range.

[0119] In step S307, the measuring device 10 sets grid points 131a within the depth setting range as measuring points. The data processing unit 35 acquires the time-series data of facial feature points of the grid points 131a set as measuring points, and uses it as time-series data of measuring points. The time-series data of measuring points includes multiple measuring point data. The data processing unit 35 uses the time-series data of measuring points including multiple measuring point data to... Figure 7The pulse wave signal is detected in step S119 shown.

[0120] In step S309, the measuring device 10 excludes grid points 131a whose depth component is not within the depth setting range from the measuring point. The data processing unit 35 uses the time series data of facial feature points of grid points 131a excluded from the measuring point for pulse wave signal detection.

[0121] Data Processing Department 35 Figure 11 In the control flow shown, the coordinates Cn of the nth grid point (n=1 to N) are compared with the depth setting range, including zn. The data processing unit 35 sets one or more grid points 131a as measurement points by comparing zn and the depth setting range. The data processing unit 35 acquires the facial feature point time series data of the measurement points as measurement point time series data. The measurement point time series data contains data from multiple measurement points.

[0122] The data processing unit 35 uses time-series data of facial feature points from one or more measurement points, i.e., measurement point time-series data, in... Figure 7 In step S119 shown, the pulse wave signal is detected. When multiple measurement points are set, the data processing unit 35 acquires multiple facial feature point data contained in the time series data of facial feature points at each measurement point. As an example, the data processing unit 35 calculates the average grid point output value Bave for each image data using the grid point output value of the measurement point. The average grid point output value Bave is represented by the following formula (11).

[0123]

[0124] Here, rave is the average red grayscale value calculated based on the output values ​​of each measurement point. given is the average green grayscale value calculated based on the output values ​​of each measurement point. bave is the average blue grayscale value calculated based on the output values ​​of each measurement point.

[0125] The data processing unit 35 uses the average grid point output value Bave to calculate the average grid point output value time series data Bave(t). The average grid point output value time series data Bave(t) is represented by the following equation (12).

[0126]

[0127] The data processing unit 35 uses the average grid point output value time series data Bave(t) calculated based on the facial feature point time series data of each measurement point, in... Figure 7In step S119 shown, the pulse wave signal is acquired. The data processing unit 35 acquires the pulse wave signal by tracking multiple grid points 131a and using facial feature point data of the multiple grid points 131a, thereby improving the measurement accuracy of the pulse wave signal.

[0128] The measuring device 10 includes: an imaging unit 11 that captures images of the person being measured (M) to generate video data containing multiple image data; an image recognition processing unit 33 that recognizes the facial image contained in the image data and obtains grid point coordinates representing the positions of each of the multiple grid points 131a contained in the facial image; and a data processing unit 35 that generates multiple facial feature point data that establishes a relationship between the grid point coordinates and the output values ​​of the pixels contained in the image data, and detects the pulse wave signal of the person being measured (M) based on the facial feature point data. The grid point coordinates include width, height, and depth components. The data processing unit 35 uses the grid point coordinates contained in the facial feature point data to extract measurement point data from the multiple facial feature point data, and uses the measurement point data to detect the pulse wave signal.

[0129] The measuring device 10 improves the detection accuracy of the pulse wave signal by using measurement point data to detect the pulse wave signal.

[0130] The measuring device 10 includes a storage unit 41 that stores a depth setting range for specifying the depth coordinate range of the depth component. Preferably, the data processing unit 35 extracts the measuring point data by comparing the depth component contained in the grid point coordinates with the depth setting range.

[0131] The data processing unit 35 can extract measurement point data by using the depth component and depth setting range contained in the grid point coordinates, thereby excluding grid points 131a in the face image region 121 that reduce the detection accuracy of the pulse wave signal from the measurement points. The detection accuracy of the pulse wave signal is improved.

[0132] The biological analysis program PG enables the control unit 31, which is connected to the imaging unit 11 that generates multiple image data with the subject M, to identify the facial image contained in the image data, obtain the grid point coordinates representing the positions of the multiple grid points 131a contained in the facial image, generate multiple facial feature point data that establishes a relationship between the grid point coordinates and the output values ​​of the pixels contained in the image data, extract measurement point data from the multiple facial feature point data using the grid point coordinates contained in the facial feature point data, and use the measurement point data to detect the pulse wave signal.

[0133] The bioanalysis program PG improves the accuracy of pulse wave signal detection by using measurement point data.

[0134] Figure 12An example of a control flow executed by the measuring device 10 is shown. Figure 12 An example of how to set up measurement points is shown. Figure 12 It shows in Figure 7 An example of the control flow executed in step S117 is shown. Figure 12 The control flow for setting measurement points is shown using the depth component and output values ​​contained in the grid point coordinates.

[0135] In step S401, the measuring device 10 acquires facial feature point data. The data processing unit 35 acquires facial feature point data contained in the facial feature point time series data. The data processing unit 35 acquires zn contained in the nth facial feature point data Dn. zn is the depth component of the coordinate Cn of the nth grid point.

[0136] After acquiring facial feature point data, the measuring device 10 reads the depth setting range in step S403. The data processing unit 35 reads the depth setting range stored in the storage unit 41.

[0137] After reading the depth setting range, the measuring device 10 determines in step S405 whether the depth component of the grid point coordinates is within the depth setting range. The data processing unit 35 compares the depth component of the grid point coordinates with the depth setting range. The data processing unit 35 determines whether the depth component contained in the grid point coordinates of each grid point 131a is related to equation (9).

[0138] When the zn contained in the coordinates Cn of the nth grid point is in the relationship of equation (9), the data processing unit 35 determines that the depth component is within the depth setting range. The measuring device 10 proceeds to step S407 (step S405: Yes). When the zn contained in the coordinates Cn of the nth grid point is not in the relationship of equation (9), the data processing unit 35 determines that the depth component is not within the depth setting range. The measuring device 10 proceeds to step S413 (step S405: No).

[0139] In step S407, the measuring device 10 reads the output value setting range. The data processing unit 35 reads the output value setting range from the storage unit 41. The output value setting range corresponds to an example of a light intensity threshold. The output value setting range is pre-stored in the storage unit 41. The output value setting range is information that specifies the range of output values ​​corresponding to grid point coordinates. The output value setting range corresponds to an example of a light intensity range. The output value setting range contains information specifying the lower limit of at least one of the red grayscale value, green grayscale value, and blue grayscale value contained in the facial feature point data. The output value setting range is compared with the output values ​​corresponding to the grid point coordinates.

[0140] After reading the output value setting range, the measuring device 10 determines whether the output value is within the output value setting range in step S409. In step S405, the data processing unit 35 compares the output value corresponding to the coordinates of the grid points whose depth component is within the depth setting range with the output value setting range. The data processing unit 35 compares at least one of the red grayscale value, green grayscale value, and blue grayscale value contained in the facial feature point data of each grid point 131a with the output value setting range.

[0141] If the data processing unit 35 determines that the output value is within the output value setting range, the measuring device 10 proceeds to step S411 (step S409: Yes). If the data processing unit 35 determines that the output value is not within the output value setting range, the measuring device 10 proceeds to step S413 (step S409: No).

[0142] In step S411, the measuring device 10 sets grid points 131a of facial feature point data whose depth components are within the depth setting range and whose output values ​​are within the output value setting range as measurement points. The data processing unit 35 acquires the time series data of facial feature points 131a set as measurement points.

[0143] In step S413, the measuring device 10 excludes grid points 131a of facial feature point data whose depth component is not within the depth setting range and grid points 131a of facial feature point data whose output value is not within the output value setting range from the measurement point. The data processing unit 35 uses the facial feature point time series data of grid points 131a excluded from the measurement point for pulse wave signal detection.

[0144] Data processing unit 35 executes the data Dn of the nth facial feature point where n is 1 to N. Figure 12 The control flow shown sets one or more grid points 131a as measurement points. The data processing unit 35 extracts time-series data of facial feature points corresponding to the one or more measurement points as measurement point time-series data. The one or more measurement point time-series data contains data from multiple measurement points. Using the measurement point time-series data containing measurement point data, in Figure 7 The pulse wave signal is detected in step S119 shown.

[0145] Storage unit 41 stores the output value setting range specified for the output value range. Preferably, data processing unit 35 extracts measurement point data by comparing the output values ​​contained in the facial feature point data with the output value setting range.

[0146] The measuring device 10 can exclude the output value of grid points 131a where shadows are cast due to the position of the light source, etc. The measurement accuracy of the pulse wave signal is improved.

[0147] Figure 13 An example of a control flow executed by the measuring device 10 is shown. Figure 13 An example of how to set up measurement points is shown. Figure 13 It shows in Figure 7 An example of the control flow executed in step S117 is shown. Figure 13 The control flow for setting measurement points is shown using grid point association information MT and the depth component contained in the grid point coordinates.

[0148] In step S501, the measuring device 10 acquires facial feature point data. The data processing unit 35 acquires the facial feature point data contained in the facial feature point time series data. The data processing unit 35 acquires the coordinates Cn of the nth grid point contained in the nth facial feature point data Dn.

[0149] In step S503, the measuring device 10 reads the grid point association information MT from the storage unit 41. As an example, the data processing unit 35 reads... Figure 6 The grid point association information MT is shown.

[0150] In step S505, the measuring device 10 reads the gradient setting range. The gradient setting range is a numerical range of information specifying the tilt range of the depth component contained in the coordinates of each adjacent grid point 131a. The gradient setting range is pre-stored in the storage unit 41. As an example, the gradient setting range is set by setting a minimum gradient value gmin and a maximum gradient value gmax. The minimum gradient value gmin is the lower limit of the gradient setting range. The maximum gradient value gmax is the upper limit of the gradient setting range.

[0151] After reading the gradient setting range, the measuring device 10 determines in step S507 whether the gradients of two adjacent grid points 131a are within the gradient setting range. The gradients of two adjacent grid points 131a are represented by the following equation (13).

[0152]

[0153] Here, gΔn is the gradient between grid point 131a at coordinates Cn of the nth grid point and the adjacent grid point 131a at coordinates Cn+1. zn+1 is the depth component contained in the grid point Cn+1 of the (n+1)th grid point. dΔn is the distance between grid point 131a at coordinates Cn of the nth grid point and the grid point 131a at coordinates Cn+1 of the (n+1)th grid point.

[0154] The data processing unit 35 compares the gradients and gradient setting ranges of two adjacent grid points 131a. The data processing unit 35 determines whether the gradients of two adjacent grid points 131a are related by the following formula (14).

[0155]

[0156] At the edges of the face, the tip of the nose, and the periphery of the bridge of the nose, the gradient between two adjacent grid points 131a becomes a value outside the gradient setting range. By setting grid points 131a within the gradient setting range as measurement points, the possibility of detecting output values ​​within a predetermined range is increased. This can suppress the reduction in the detection accuracy of pulse wave signals.

[0157] When the gradient between a grid point 131a and an adjacent grid point 131a is in the relationship of equation (14), the data processing unit 35 identifies that a grid point 131a is a grid point coordinate within the gradient setting range relative to the adjacent grid point 131a. The data processing unit 35 identifies that a grid point 131a can be set as a measurement point. The measuring device 10 proceeds to step S509 (step S507: Yes). When the gradient between a grid point 131a and an adjacent grid point 131a is not in the relationship of equation (14), the data processing unit 35 identifies that a grid point 131a is not within the gradient setting range relative to the adjacent grid point 131a. The data processing unit 35 identifies that a grid point 131a is not extracted as a measurement point. The measuring device 10 proceeds to step S511 (step S507: No).

[0158] In step S509, the measuring device 10 sets a grid point 131a as a measuring point. The data processing unit 35 acquires the time-series data of facial feature points of the grid point 131a set as the measuring point, and uses it as the time-series data of the measuring point. The time-series data of the measuring point includes data of multiple measuring points. The data processing unit 35 uses the time-series data of the measuring point including the measuring point data to... Figure 7 The pulse wave signal is detected in step S119 shown.

[0159] In step S511, the measuring device 10 excludes one grid point 131a from the measuring points. The data processing unit 35 uses the time series data of the facial feature points of the grid point 131a excluded from the measuring points for pulse wave signal detection.

[0160] Figure 14 An example of a control flow executed by the measuring device 10 is shown. Figure 14 An example of how to set up measurement points is shown. Figure 14 It shows in Figure 7 An example of the control flow executed in step S117 is shown. Figure 14 The control flow for setting measurement points using the width, height, and depth components contained in the grid point coordinates is shown.

[0161] In step S601, the measuring device 10 acquires facial feature point data. The data processing unit 35 acquires facial feature point data contained in the facial feature point time series data. The data processing unit 35 acquires zn contained in the nth facial feature point data Dn.

[0162] After acquiring facial feature point data, the measuring device 10 reads the depth setting range in step S603. The data processing unit 35 reads the depth setting range stored in the storage unit 41.

[0163] After reading the depth setting range, the measuring device 10 determines in step S605 whether the depth component of the grid point coordinates is within the depth setting range. The data processing unit 35 compares the depth component of the grid point coordinates with the depth setting range. The data processing unit 35 determines whether the depth component contained in the grid point coordinates of each grid point 131a is related to equation (9).

[0164] When the zn contained in the coordinates Cn of the nth grid point is in the relationship of equation (9), the data processing unit 35 determines that the depth component is within the depth setting range. The measuring device 10 proceeds to step S607 (step S605: Yes). When the zn contained in the coordinates Cn of the nth grid point is not in the relationship of equation (9), the data processing unit 35 determines that the depth component is not within the depth setting range. The measuring device 10 proceeds to step S617 (step S605: No).

[0165] In step S607, the measuring device 10 reads the grid point association information MT from the storage unit 41. As an example, the data processing unit 35 reads... Figure 6 The grid point association information MT is shown.

[0166] After reading the grid point association information MT, the measuring device 10 calculates the distance between grid points in step S609. The data processing unit 35 uses the grid point association information MT to identify adjacent grid points 131a that are adjacent to the grid points 131a of the depth component within the depth setting range. The data processing unit 35 uses the grid point coordinates of the grid points 131a of the depth component within the depth setting range and the grid point coordinates of the adjacent grid points 131a to calculate the distance between grid points. The distance between grid points is the distance between the grid point 131a of the depth component within the depth setting range and the adjacent grid point 131a. Here, as an example, the grid point 131a of the depth component within the depth setting range is represented as the i-th grid point. The adjacent grid point 131a adjacent to the i-th grid point is represented as the j-th grid point. The i-th grid point coordinate Ci of the i-th grid point and the j-th grid point coordinate Cj of the j-th grid point are represented by the following equations (15) and (16), respectively.

[0167]

[0168] The data processing unit 35 uses the coordinates Ci of the i-th grid point and the coordinates Cj of the j-th grid point to calculate the distance dij between the i-th grid points. The distance dij between the i-th grid points is an example of the distance between grid points. The distance dij between the i-th grid points is represented by equation (17).

[0169] Mathematical Formula 1

[0170] Where i and j are two distinct points from 1 to N. In step S611, the measuring device 10 reads the set distance range. The data processing unit 35 reads the set distance range from the storage unit 41. The set distance range corresponds to an example of a planar coordinate threshold. The set distance range is pre-stored in the storage unit 41. The set distance range is information that specifies the planar range involved in the width and height components of the grid point coordinates. The planar range corresponds to an example of a planar coordinate range. The set distance range is compared with the distance between grid points calculated using the width and height components contained in the grid point coordinates.

[0171] After reading the set distance range, the measuring device 10 determines in step S613 whether the distance between grid points is within the set distance range. When the distance between grid points is within the set distance range, two adjacent grid points 131a are located in a position that can be visually confirmed when viewed from the plane of the shooting unit 11. The data processing unit 35 determines that the two adjacent grid points 131a have not moved to a position that cannot be visually confirmed due to factors such as the orientation of the face. When the distance dij between grid points ij is within the set distance range, the data processing unit 35 determines that the i-th grid point can be set as a measurement point. The measuring device 10 proceeds to step S615 (step S613: Yes). If the distance between grid points is not within the set distance range, one of the two adjacent grid points 131a is not in a position that can be visually confirmed when viewed from the plane of the shooting unit 11. The data processing unit 35 determines that one of the two adjacent grid points 131a has moved to a position that cannot be visually confirmed due to factors such as the orientation of the face. When the distance dij between grid points ij is not within the set distance range, the data processing unit 35 determines that the i-th grid point cannot be set as a measurement point. The measuring device 10 proceeds to step S617 (step S613: No).

[0172] In step S615, the measuring device 10 sets the i-th grid point as a measuring point. The data processing unit 35 acquires the time-series data of facial feature points of the i-th grid point set as a measuring point, and uses it as the time-series data of the measuring point. The data processing unit 35 uses the time-series data of the measuring point containing data from multiple measuring points to... Figure 7 The pulse wave signal is detected in step S119 shown.

[0173] In step S617, the measuring device 10 excludes grid points 131a and the i-th grid point whose depth components are not within the depth setting range from the measuring points. The data processing unit 35 does not use the facial feature point time series data of grid point 131a whose depth components are not within the depth setting range and the facial feature point time series data of the i-th grid point for pulse wave signal detection.

[0174] exist Figure 14 In the control flow shown, after determining whether the depth component is within the depth setting range, it determines whether the distance between grid points is within the set distance range, but it is not limited to this. The measuring device 10 may also perform the determination of whether the depth component is within the depth setting range and the determination of whether the distance between grid points is within the set distance range at the same timing. The measuring device 10 may also determine whether the depth component is within the depth setting range after determining whether the distance between grid points is within the set distance range.

[0175] The storage unit 41 has a set distance range that specifies the planar range involved in the width and height components. Preferably, the data processing unit 35 uses the set distance range to extract measurement point data.

[0176] The measuring device 10 can exclude grid points 131a that are not visually identifiable when viewed from the plane of the imaging unit 11 from the measurement points. The detection accuracy of the pulse wave signal is improved.

[0177] Figure 15 An example of a control flow executed by the measuring device 10 is shown. Figure 15 An example of how to set up measurement points is shown. Figure 15 It shows in Figure 7 An example of the control flow executed in step S117 is shown. Figure 15 The control flow for extracting measurement point data using facial feature point time series data is shown.

[0178] In step S701, the measuring device 10 acquires time-series data of facial feature points. The data processing unit 35 acquires time-series data of facial feature points including facial feature point data. The data processing unit 35 acquires the nth facial feature point time-series data Dn(t).

[0179] After acquiring the time-series data of facial feature points, the measuring device 10 calculates the grid point variation in step S703. The grid point variation corresponds to an example of variation. The grid point variation shows the change in grid point coordinates over the measurement time. The grid point variation is a body movement index representing the magnitude of the body movement of the measurer M. As the body movement increases, the measurement accuracy of the pulse wave signal decreases. As an example, the grid point variation is represented by equation (18). The RMS in equation (18) represents the root mean square.

[0180] Mathematical formula 2

[0181] Where n = 1 to N Here, xave is the average of the width components contained in the nth facial feature point data Dn. yave is the average of the height components contained in the nth facial feature point data Dn. zave is the average of the depth components contained in the nth facial feature point data Dn.

[0182] The variation in grid points can be calculated using either the root mean square (RMS) or a statistical value different from the RMS. The variation in grid points can also be calculated using variance, standard deviation, or coefficient of variation. The method for calculating the variation in grid points is appropriately defined.

[0183] In step S705, the measuring device 10 reads the variation threshold from the storage unit 41. The variation threshold is pre-stored in the storage unit 41. The variation threshold is compared with the variation of the grid points. The variation threshold is preset to a predetermined value. When the variation of the grid points is less than the variation threshold, it indicates that the body movements of the measurer M are within the range where the pulse wave signal can be measured. When the variation of the grid points is greater than the variation threshold, it indicates that the body movements of the measurer M are within the range where the pulse wave signal is difficult to measure.

[0184] After reading the variation threshold, the measuring device 10 determines in step S707 whether the variation of the grid points is less than the variation threshold. The data processing unit 35 compares the variation of the grid points with the variation threshold. If the data processing unit 35 determines that the variation of the grid points is less than the variation threshold, the measuring device 10 proceeds to step S709 (step S707: Yes). If the data processing unit 35 determines that the variation of the grid points is greater than the variation threshold, the measuring device 10 proceeds to step S711 (step S707: No).

[0185] In step S709, the measuring device 10 sets grid points 131a whose grid point variation is less than the variation threshold as measuring points. The data processing unit 35 acquires the facial feature point time series data of the grid points 131a set as measuring points, and uses it as the measuring point time series data. The data processing unit 35 uses the measuring point time series data containing multiple measuring point data to... Figure 7 The pulse wave signal is detected in step S119 shown.

[0186] In step S711, the measuring device 10 excludes grid points 131a whose grid point variation exceeds a variation threshold from the measuring points. The data processing unit 35 does not use the facial feature point time series data of the grid points 131a excluded from the measuring points for pulse wave signal detection. By excluding the facial feature point time series data of grid points 131a whose grid point variation exceeds the variation threshold, the detection accuracy of the pulse wave signal is improved.

[0187] The data processing unit 35 generates time-series data of facial feature points by tracking the grid points 131a contained in the facial image in a time series.

[0188] The data processing unit 35 improves the detection accuracy of the pulse wave signal by using the time series data of facial feature points tracked by the grid points 131a.

[0189] Preferably, the data processing unit 35 calculates the grid point variation of the grid point coordinates within the multiple facial feature point data contained in the facial feature point time series data, and extracts the measurement point data based on the grid point variation.

[0190] The measuring device 10 can improve the detection accuracy of pulse wave signals by excluding grid points 131a whose grid point variation is greater than the variation threshold.

[0191] Figure 16 An example of a control flow executed by the measuring device 10 is shown. Figure 16 An example of how to set up measurement points is shown. Figure 16 It shows in Figure 7 An example of the control flow executed in step S117 is shown. Figure 16 The control flow for extracting measurement point data using the grid point coordinates contained in facial feature point data is shown.

[0192] In step S801, the measuring device 10 acquires facial feature point data. The data processing unit 35 acquires the facial feature point data contained in the facial feature point time series data. The data processing unit 35 acquires the coordinates Cn of the nth grid point contained in the nth facial feature point data Dn.

[0193] After acquiring facial feature point data, the measuring device 10 normalizes the grid point coordinates in step S803. The data processing unit 35 normalizes the coordinates Cn of the nth grid point where n is 1 to N. The calculation formula for normalizing the coordinates Cn of the nth grid point is represented by equation (19) as an example. The normalized coordinates Cn of the nth grid point are shown in the facial feature point data Dn of the nth grid point in equation (19).

[0194] Mathematical Formula 3

[0195] Where n = 1 to N Dnorm n Let Dn represent the normalized facial feature point data of the nth grid point. Here, xmax is the maximum value of xn among n = 1 to N. xmin is the minimum value of xn among n = 1 to N. ymax is the maximum value of yn among n = 1 to N. ymin is the minimum value of yn among n = 1 to N. zmax is the maximum value of zn among n = 1 to N. zmin is the minimum value of zn among n = 1 to N.

[0196] After normalizing the grid point coordinates, the measuring device 10 reads the position setting range in step S805. The data processing unit 35 reads the position setting range from the storage unit 41. The position setting range is pre-stored in the storage unit 41. The position setting range is information that specifies the setting range related to the width component, height component, and depth component of the normalized grid point coordinates. As an example, the position setting range is a range of 25% above and below the center value of each of the normalized width component, height component, and depth component. The position setting range is a setting range that sets the value of at least one of the width component, height component, and depth component. By using the position setting range, the data processing unit 35 can exclude grid points 131a in areas where pulse wave signals are difficult to detect, such as the periphery of the face, the area around the nose, and the eye sockets, from the measurement points.

[0197] After reading the position setting range, the measuring device 10, in step S807, determines whether the normalized grid point coordinates are within the position setting range. The measuring device 10 compares the normalized grid point coordinates with the position setting range. If the data processing unit 35 determines that the normalized grid point coordinates are within the position setting range, the measuring device 10 proceeds to step S809 (step S807: Yes). If the data processing unit 35 determines that the normalized grid point coordinates are not within the position setting range, the measuring device 10 proceeds to step S815 (step S807: No).

[0198] In step S809, the measuring device 10 reads the output value setting range. The data processing unit 35 reads the output value setting range from the storage unit 41. The output value setting range is pre-stored in the storage unit 41. The output value setting range is information that specifies the range of output values ​​corresponding to the grid point coordinates. The output value setting range includes information that specifies the lower limit of at least one of the red grayscale values, green grayscale values, and blue grayscale values ​​contained in the facial feature point data. The output value setting range is compared with the output values ​​corresponding to the grid point coordinates.

[0199] After reading the output value setting range, the measuring device 10 determines whether the output value is within the output value setting range in step S811. In step S807, the data processing unit 35 compares the output value of grid point 131a whose grid point coordinates are within the position setting range with the output value setting range. The data processing unit 35 compares the red grayscale value, green grayscale value, and blue grayscale value contained in the facial feature point data of grid point 131a whose grid point coordinates are within the position setting range with the output value setting range.

[0200] If the data processing unit 35 determines that the output value is within the output value setting range, the measuring device 10 proceeds to step S813 (step S811: Yes). If the data processing unit 35 determines that the output value is not within the output value setting range, the measuring device 10 proceeds to step S815 (step S811: No).

[0201] In step S813, the measuring device 10 sets grid points 131a whose grid point coordinates are within the position setting range and whose output values ​​are within the output value setting range as measuring points. The data processing unit 35 acquires the facial feature point time series data of the grid points 131a set as measuring points, and uses it as measuring point time series data. The data processing unit 35 uses the measuring point time series data containing data from multiple measuring points to... Figure 7 The pulse wave signal is detected in step S119 shown.

[0202] In step S815, the measuring device 10 excludes grid points 131a whose grid point coordinates are outside the position setting range and grid points 131a whose output values ​​are outside the output value setting range from the measurement points. The data processing unit 35 uses the time series data of facial feature points of grid points 131a excluded from the measurement points for pulse wave signal detection.

[0203] exist Figure 16In the control flow shown, a measurement point is set by determining whether the output value is within the output value setting range, but this is not the only method. The control flow for setting the measurement point may also exclude steps S809 and S811. The measuring device 10 may also determine whether the grid point coordinates are within the position setting range and set the measurement point based on the determination result.

[0204] Figure 12 , Figure 13 , Figure 14 , Figure 15 as well as Figure 16 The control flow shown can be modified appropriately. As an example, the measuring device 10 can also be... Figure 11 In the control flow shown, after the processing of step S303 is executed, the processing of step S301 is executed. The execution order of each step is set appropriately.

[0205] Symbol Explanation 10… Measuring device; 11… Imaging unit; 13… Display unit; 15… Input unit; 31… Control unit; 33… Image recognition processing unit; 35… Data processing unit; 37… Display control unit; 41… Storage unit; 100… Captured image; 121… Face image area; 131… Face grid information; 131a… Grid point; 131b… Grid line; Db… Blue grayscale detection value; Dg… Green grayscale detection value; Dr… Red grayscale detection value; M… Measurer; MT… Grid point association information; PG… Biological analysis program; S… Noise removal signal; S1… Body movement range; S2… Quiet range; TBL-01… First code name; TBL-02… Second code name; TBL-03… Third code name; TBL-04… Fourth code name.

Claims

1. A biological information acquisition device, comprising: The imaging unit takes pictures of living organisms and generates imaging data containing image data; A face recognition unit identifies the face image contained in the image data and obtains three-dimensional coordinate data representing the positions of multiple feature points contained in the face image. The detection unit generates multiple feature point data that establishes a correlation between the three-dimensional coordinate data and the light intensity values ​​of the pixels contained in the image data. It uses the three-dimensional coordinate data to extract measurement feature point data for detecting the pulse wave signal of the organism from the multiple feature point data, and uses the measurement feature point data to detect the pulse wave signal.

2. The biological information acquisition device as described in claim 1, wherein, The system includes a storage unit that stores depth range thresholds for specifying the depth coordinate range based on the depth component of the three-dimensional coordinate data. The detection unit extracts the measurement feature point data by comparing the depth component contained in the three-dimensional coordinate data with the depth range threshold.

3. The biological information acquisition device as described in claim 2, wherein, The storage unit has a planar coordinate threshold that specifies the planar coordinate range for the width component and the height component of the three-dimensional coordinate data. The detection unit uses the planar coordinate threshold to extract the measurement feature point data.

4. The biological information acquisition device as described in claim 2 or 3, wherein, The storage unit stores light intensity thresholds that specify the light intensity range for the light intensity value. The detection unit extracts the measured feature point data by comparing the light intensity value contained in each of the plurality of feature point data with the light intensity threshold.

5. The biological information acquisition device as described in claim 1, wherein, The detection unit generates a feature point data set by tracking the feature points contained in the face image in a time sequence.

6. The biological information acquisition device as described in claim 5, wherein, Calculate the variation in the three-dimensional coordinate data of each of the plurality of feature point data, and extract the measured feature point data from the plurality of feature point data based on the variation.

7. A biological information acquisition program product, comprising a biological information acquisition program, The biological information acquisition program causes the computer connected to the imaging unit that generates image data by photographing the biological organism to perform the following processing, namely, Identify the facial images contained in the image data. Obtain three-dimensional coordinate data representing the positions of each of the multiple feature points contained in the facial image. Multiple feature point data are generated that establish a correlation between the three-dimensional coordinate data and the light intensity values ​​of the pixels contained in the image data. Using the three-dimensional coordinate data, measurement feature point data for detecting the pulse wave signal of the organism is extracted from the multiple feature point data. The pulse wave signal is detected using the measured feature point data.

Images