Arithmetic device, arithmetic method, and program

Abstract

To detect biological information in a non-contact manner.SOLUTION: An arithmetic device includes: a video information acquisition part for acquiring video information including a plurality of continuing frame images where a person to be measured is photographed; an area specification part for specifying an area to be measured of areas of the person photographed in the frame image; a statistic arithmetic part for performing statistical arithmetic of at least any two pixel values of pixel values of RGB of a plurality of continuing frame images, for the specified area; a difference calculation part for calculating a difference for any two pixel values to which the statistical arithmetic is performed; an operation part for converting information about a difference value every time calculated by the difference calculation part into a frequency axis; and an output part for outputting a pulse wave amplitude value and a heart rate about a person to be measured on the basis of the result converted by the operation part.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a computing device, a computing method, and a program. [Background technology]

[0002] Modern society is often called a stressful society, and mental illness caused by stress is a growing problem. Mental illness can be caused by a variety of factors, including interpersonal relationships at work, the workplace environment, and excessive work, as well as lack of exercise, sleep, and an unbalanced diet. In order to prevent stress-related mental illness, there is a demand for stress assessment tools that can easily measure stress levels at home and allow for lifelogging.

[0003] In recent years, psychological assessment questionnaires, which are subjective assessment methods, have become commonplace for stress assessment. However, stress assessments using psychological assessment questionnaires have the drawback of being easily influenced by the subject's physical condition and mood at the time. Therefore, as an objective assessment method, research has been conducted into methods for assessing stress based on physiological information obtained from the subject, such as respiration, heart rate, saliva, and pulse wave. In particular, among these physiological information, pulse waves require inexpensive measurement equipment, and the measurement equipment can be implemented in wearable devices such as smartwatches. Therefore, even though pulse waves are relatively difficult to obtain, they can be easily measured. A biometric information detection device, for example, is known as an example of a device that embodies this technology (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-147746 Summary of the Invention [Problem to be solved by the invention]

[0005] The biometric information detection devices described in the above-mentioned documents can detect biometric information, such as pulse waves, based on scattered light obtained by flashing a sensor. For example, such biometric information detection devices can be used to detect changes in a user's biometric information while the user is browsing content, such as an e-book, on a smartphone. However, the penetration rate of wearable devices, such as smartwatches, equipped with such biometric information detection devices is not necessarily higher than that of smartphones. Therefore, to detect changes in a user's biometric information using such technology, the user must wear a wearable device, such as a smartwatch, equipped with the biometric information detection device, in addition to a smartphone for browsing content, such as an e-book. If biometric information could be detected non-contact using a smartphone or the like, the user would not need to wear the wearable device, and biometric information could be obtained even from users who are not wearing the wearable device.

[0006] Therefore, an object of the present invention is to provide a computing device, a computing method, and a program that are capable of detecting biometric information in a non-contact manner. [Means for solving the problem]

[0007] [1] In order to solve the above problem, one aspect of the present invention includes a video information acquisition unit that acquires video information including a plurality of consecutive frame images of a person to be measured, the frame images being frame images of the person; a region identification unit that identifies the earlobe of the person as a region to be measured from among the regions of the person captured in the frame images; a statistical calculation unit that performs statistical calculations on at least two of the RGB pixel values ​​of the plurality of consecutive frame images for the identified region; a difference calculation unit that calculates the difference between at least two of the pixel values ​​for which statistical calculations have been performed; a calculation unit that converts information about the difference values ​​for each time calculated by the difference calculation unit into a frequency axis; and an output unit that outputs a pulse wave amplitude value and a heart rate for the person to be measured based on the results of the conversion by the calculation unit. The region identification unit identifies a plurality of regions in addition to the earlobe of the person captured in the frame image as regions to be measured, the statistical calculation unit performs statistical calculation for each of the plurality of regions identified by the region identification unit, the difference calculation unit calculates a difference for each of the plurality of regions identified by the region identification unit, the calculation unit converts each of the plurality of regions identified by the region identification unit into a frequency axis, and the output unit outputs a pulse wave amplitude value and a heart rate for the person to be measured based on at least one of the plurality of regions identified by the region identification unit. It is a computing device.

[0009] [ 2 ] Furthermore, one aspect of the present invention is the above [ 1 In the calculation device described in [2], a priority order is assigned to each of the multiple areas identified by the area identification unit in advance, the calculation unit calculates a pulse wave amplitude value and a heart rate for each of the multiple areas identified by the area identification unit, and if the calculation result is equal to or less than a threshold, determines that the calculation result is incorrect, and the output unit outputs the result measured at the location with the highest priority among the results for which the calculation result is greater than the threshold.

[0010] [ 3 ] Also, one aspect of the present invention is the above [1] or [ 2] In the calculation device described in the above, the statistical calculation unit performs statistical calculations on at least the R and G pixel values ​​among the pixel values ​​of each of the RGB of the consecutive frame images, and the difference calculation unit calculates the difference between the R and G pixel values ​​for which statistical calculations have been performed.

[0011] [ 4 ] Furthermore, one aspect of the present invention is the above [1] to [ 3In the calculation device described in any one of the above, the statistical calculation unit calculates the average value of at least any two pixel values ​​of each of the RGB pixel values ​​of a plurality of consecutive frame images for the identified area.

[0012] [ 5 ] Furthermore, one aspect of the present invention is the above [1] to [ 4 ] The calculation device according to any one of the preceding claims, further comprising an inverse Fourier calculation unit that calculates a pulse wave by performing an inverse Fourier transform based on the result of the conversion by the calculation unit, and the output unit further outputs the pulse wave calculated by the inverse Fourier calculation unit.

[0013] [ 6 Another aspect of the present invention is a method for measuring a pulse wave amplitude and a heart rate of a person to be measured, the method comprising: a video information acquisition step of acquiring video information including a plurality of consecutive frame images of a person to be measured; a region identification step of identifying an earlobe of the person from among the regions of the person captured in the frame images as a region to be measured; a statistical calculation step of performing statistical calculation on at least two of the RGB pixel values ​​of the plurality of consecutive frame images for the identified region; a difference calculation step of calculating a difference between at least two of the pixel values ​​for which statistical calculation has been performed; a calculation step of converting information on the difference values ​​for each time calculated by the difference calculation step into a frequency axis; and an output step of outputting a pulse wave amplitude and a heart rate of the person to be measured based on the results of the conversion by the calculation step. The region specifying step specifies a plurality of regions in addition to the earlobe of the person captured in the frame image as regions to be measured, the statistical calculation step performs a statistical calculation for each of the plurality of regions specified by the region specifying step, the difference calculation step calculates a difference for each of the plurality of regions specified by the region specifying step, the calculation step converts each of the plurality of regions specified by the region specifying step into a frequency axis, and the output step outputs a pulse wave amplitude value and a heart rate for the person to be measured based on at least one of the plurality of regions specified by the region specifying step. It is a calculation method.

[0014] [ 7Another aspect of the present invention is a method for making a computer execute the following steps: a video information acquisition step of acquiring video information including a plurality of consecutive frame images of a person to be measured, the frame images being frame images of the person; a region identification step of identifying an earlobe of the person from among the regions of the person captured in the frame images as a region to be measured; a statistical calculation step of performing statistical calculations on at least two of the RGB pixel values ​​of the plurality of consecutive frame images for the identified region; a difference calculation step of calculating the difference between at least two of the pixel values ​​for which statistical calculations have been performed; a calculation step of converting information about the difference values ​​for each time calculated in the difference calculation step into a frequency axis; and an output step of outputting a pulse wave amplitude value and a heart rate for the person to be measured based on the results converted in the calculation step. The region specifying step specifies a plurality of regions in addition to the earlobe of the person captured in the frame image as regions to be measured, the statistical calculation step performs a statistical calculation for each of the plurality of regions specified by the region specifying step, the difference calculation step calculates a difference for each of the plurality of regions specified by the region specifying step, the calculation step converts each of the plurality of regions specified by the region specifying step into a frequency axis, and the output step outputs a pulse wave amplitude value and a heart rate for the person to be measured based on at least one of the plurality of regions specified by the region specifying step. It is a program. [Effects of the Invention]

[0015] According to the present invention, it is possible to provide a computing device, a computing method, and a program that are capable of detecting biometric information in a non-contact manner. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 2 is a functional configuration diagram showing an example of the functional configuration of the content information providing device according to the present embodiment. [Figure 2] FIG. 2 is a functional configuration diagram showing an example of the functional configuration of a calculation device according to the present embodiment. [Figure 3] 10A and 10B are diagrams for explaining an example of region specification performed by the calculation device according to the present embodiment. [Figure 4] 10A and 10B are diagrams for explaining a method for acquiring pulse wave components by a calculation device according to the present embodiment. [Figure 5] FIG. 10 is a diagram showing an example of RGB difference values ​​according to the embodiment. [Figure 6] 4A and 4B are diagrams showing an example of pulse wave amplitude values ​​and heart rates obtained by the calculation device according to the present embodiment. [Figure 7]FIG. 10 is a diagram showing a comparison result between the change over time of the pulse wave amplitude value obtained based on the RG difference value according to the present embodiment and the change over time of the pulse wave amplitude value obtained by a photoplethysmograph. [Figure 8] FIG. 10 is a diagram showing the results of a comparison between the heart rate obtained by the calculation device according to the present embodiment and the heart rate obtained by a photoplethysmography system. [Figure 9] 1 is a flowchart showing the flow of a series of processes performed by a calculation method according to the present embodiment. [Figure 10] FIG. 10 is a functional configuration diagram showing a modified example of the functional configuration of the arithmetic device according to the present embodiment. [Figure 11] FIG. 2 is a block diagram showing an example of the internal configuration of a calculation device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] [Embodiment] Below, preferred embodiments of a computing device, a computing method, and a program according to the present invention will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments and includes various modifications and improvements. In other words, the components described below include those that would be easily conceivable to a person skilled in the art or that are substantially identical, and the components described below can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the present invention. Furthermore, in the drawings, the scale and number of components may differ from the scale and number of the actual components to make each configuration easier to understand.

[0018] The arithmetic device according to this embodiment detects biometric information of a user U in a non-contact manner. The biometric information detected from the user U can be used for various analyses. The arithmetic device is not limited to being used for browsing content, and can be widely used. For example, the arithmetic device may be used for tests such as psychological evaluations or stress evaluations. Furthermore, analyses may be performed using the results of these evaluations, and the analysis results may be used when providing various services. Furthermore, the arithmetic device may be used in place of a pulse wave measuring device in situations where a pulse wave measuring device has conventionally been used.

[0019] Hereinafter, as an example, a case will be described in which the arithmetic device is used to select content to be presented to a user when the user browses content, adjust the method of presenting the content, etc. In this case, the arithmetic device is implemented in, for example, the content information providing device 1.

[0020] FIG. 1 is a functional configuration diagram showing an example of the functional configuration of a content information providing device according to this embodiment. First, the functional configuration of the content information providing device 1 according to this embodiment will be described with reference to the same diagram. The content information providing device 1 is a device used by a user U. The user U browses content by operating the content information providing device 1. The content provided by the content information providing device 1 may be e-books including manga and novels, videos including movies, images including photographs, music, services provided by SNS (Social Networking Service), etc. The content provided by the content information providing device 1 may be content stored in the device itself, or may be web content stored in a server device (not shown). The content information providing device 1 may be a general-purpose device such as a smartphone, a tablet terminal, or smart glasses.

[0021] The content information providing device 1 is configured to include at least an imaging unit 2, a calculation unit 3, a control unit 4, and a display unit 5 as functional components. Each of these functional units may be realized using, for example, electronic circuits. Each functional unit may also include internal storage means such as a semiconductor memory or a magnetic hard disk drive, as necessary. Each function may also be realized by a computer having a CPU (Central Processing Unit) and software. All or part of each functional unit may also be realized by hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field-Programmable Gate Array). All or part of each functional unit may also be realized by a combination of software and hardware.

[0022] The imaging unit 2 acquires video information obtained by capturing an image of the user U. The video information includes a plurality of consecutive frame images. The frame images may be images captured by a CCD (Charge Coupled Devices) camera using a CCD image sensor for capturing video, or may be images captured by a CMOS (Complementary Metal Oxide Semiconductor) camera using a CMOS image sensor. The video information captured by the imaging unit 2 is preferably a color video including RGB pixel information. Furthermore, the video information captured by the imaging unit 2 is preferably an image of the user U's skin, and more preferably an image of the user U's face.

[0023] The calculation unit 3 analyzes the video information acquired by the imaging unit 2. The calculation unit 3 detects biometric information of the user U as a result of analyzing the video information. In other words, the calculation unit 3 can be said to detect biometric information of the user U based on video information acquired without contacting the user U (non-contact). The biometric information detected by the calculation unit 3 may include information indicating the stress state of the user U. In the following description, the calculation unit 3 may be referred to as a calculation device.

[0024] The control unit 4 acquires biometric information of the user U from the calculation unit 3. The control unit 4 controls the content to be presented to the user U based on the acquired biometric information of the user U. The control unit 4 can also provide suitable content depending on the stress state of the user U. For example, the content to be presented to the user U may be determined in advance based on the operation of the user U, and the control unit 4 may adjust the content to be presented to the user U based on the biometric information of the user U. This adjustment may include adjusting the timing of presenting the content, screen settings, etc. Note that the control unit 4 may select content to be presented to the user U based on the biometric information of the user U after the presentation of the content desired by the user U has been completed, and present the selected content to the user U.

[0025] The display unit 5 presents content to the user U in accordance with the control of the control unit 4. The content presented by the display unit 5 may include image information, video information, sound information, etc. The display unit 5 may be, for example, a liquid crystal display, an organic EL (Electroluminescence) display, etc.

[0026] FIG. 2 is a functional configuration diagram showing an example of the functional configuration of a calculation device according to this embodiment. An example of the functional configuration of the calculation device will be described with reference to the same diagram. The calculation device is the calculation unit 3 described above. The calculation unit 3 includes at least an image information acquisition unit 31, a region identification unit 32, a statistical calculation unit 33, a difference calculation unit 34, a Fourier calculation unit 35, and an output unit 36. Each of these functional units may be implemented using, for example, an electronic circuit. Each functional unit may also include internal storage means such as a semiconductor memory or a magnetic hard disk drive, as necessary. Each function may also be implemented by a computer having a CPU and software. All or part of each functional unit may also be implemented using hardware such as an ASIC, PLD, or FPGA. All or part of each functional unit may also be implemented by a combination of software and hardware.

[0027] The video information acquisition unit 31 acquires video information from the imaging unit 2. The video information includes a series of frame images. Each of the frame images captures an image of a person whose biometric information is to be measured. The person whose biometric information is to be measured may be a person to whom content information is to be provided by the content information providing device 1.

[0028] The area specifying unit 32 specifies an area to be measured from the area of ​​a person depicted in a frame image included in the video information acquired by the video information acquiring unit 31. Here, the area to be measured is the skin of the person depicted in the frame image. Preferably, the area to be measured may be the face of the person depicted in the frame image. In other words, the area specifying unit 32 specifies a part of the face of the person depicted in the frame image as the area to be measured.

[0029] It should be noted that the processing according to this embodiment does not need to be performed on all frame images included in the frame images contained in the video information acquired by the video information acquisition unit 31. The processing according to this embodiment may be performed on some of the frame images included in the video information. For example, if the frame rate of the video information is 60 [FPS (Frames Per Second)], the processing may be performed on each frame (i.e., at a granularity of 30 [FPS]).

[0030] 3 is a diagram illustrating an example of region identification performed by the arithmetic device according to this embodiment. Here, an example of a region identified by the region identification unit 32 will be described with reference to the same figure. The same figure shows a frame image FI included in video information acquired by the video information acquisition unit 31. First, the region identification unit 32 detects a face portion FP from the frame image FI. Next, the region identification unit 32 identifies a predetermined region from the face portion FP.

[0031] Here, it is preferable that the region identification unit 32 identify, as the region to be measured, a region of the face of the person captured in the frame image FI that has less fat than the surrounding area. The region that has less fat than the surrounding area may be predetermined, such as the cheek area or nose area. In the illustrated example, the region identification unit 32 identifies, as the regions to be measured, region AR1, which is the nose area, region AR2, which is the right cheek area, and region AR3, which is the left cheek area. Note that the region identification unit 32 may identify the forehead area as the region to be measured, but the forehead may be hidden by bangs, a hat, or the like depending on the user. Therefore, it is preferable that the region identification unit 32 identify a region that can be measured regardless of the user.

[0032] Note that depending on the user's appearance, what the user is wearing, and the like, the region identification unit 32 may be unable to identify the predetermined region. Therefore, the region identification unit 32 may be predetermined to identify multiple regions. If it is predetermined to identify multiple regions, the subsequent processing may be performed for each of the multiple regions, or processing may be performed for any of the regions based on a predetermined priority for each region (each body part). Returning to FIG. 2, the description of the functional configuration of the calculation unit 3 continues.

[0033] The statistical calculation unit 33 acquires information about the region identified by the region identification unit 32. The statistical calculation unit 33 preferably acquires pixel information about the region identified by the region identification unit 32. The pixel information may include pixel values ​​for R (red), G (green), and B (blue) independently. The pixel information does not need to include pixel information for all RGB, and preferably includes pixel information for two of RGB (R and G, G and B, or R and B). The statistical calculation unit 33 performs statistical calculation on at least two of the pixel values ​​of RGB (e.g., R and G) for the identified region. The statistical calculation may, for example, be calculating the average value of multiple pixel values ​​included in the identified region. Note that the statistical calculation performed by the statistical calculation unit 33 is not limited to calculating the average value, and may also be calculating statistical values ​​such as the mode, maximum value, and minimum value.

[0034] The difference calculation unit 34 calculates the difference between at least two pixel values ​​for which statistical calculations such as calculation of an average value have been performed by the statistical calculation unit 33. For example, when the statistical calculation unit 33 calculates the average value of R and the average value of G for pixels included in a region identified by the region identification unit 32, the difference calculation unit 34 calculates the difference between the average value of R and the average value of G. Note that the region identification unit 32 may calculate the difference between the average value of G and the average value of B, or may calculate the difference between the average value of R and the average value of B.

[0035] FIG. 4 is a diagram illustrating a method for acquiring pulse wave components using a calculation device according to this embodiment. As shown in the figure, the outermost layer of human skin is covered by the epidermis. Human skin has a three-layer structure consisting of the epidermis, dermis, and subcutaneous tissue, with arteries and veins running through the subcutaneous tissue. Capillaries run through the area close to the epidermis. Here, the R, G, and B components of light reflected by human skin are reflected to different depths (or it can be said that they reach different depths subcutaneously). Specifically, the B component is reflected by the epidermis, which is the shallowest layer. The G component is reflected by a layer deeper than the epidermis where capillaries and arteries run. The R component is reflected by a layer deeper than the epidermis where arteries and veins run.

[0036] This embodiment focuses on these characteristics and calculates the difference between any two of the RGB components to extract blood components from image information and further acquire pulse wave components. As described above, by determining areas with less fat than the surrounding area, rather than areas covered with fat, it becomes possible to extract blood components more accurately. Note that some users may wear makeup on their cheeks or nose. When makeup is applied, it may be difficult to extract blood components from image information. Therefore, the region identification unit 32 may identify areas where makeup is usually difficult to apply, such as the earlobes.

[0037] FIG. 5 is a diagram showing an example of RGB difference values ​​according to this embodiment. In the example shown, the horizontal axis represents time and the vertical axis represents RGB difference values, showing changes in the difference values ​​over time. Specifically, waveform W1 shows changes in the difference value between RG. Furthermore, waveform W2 shows changes in the difference value between RB. Waveform W3 shows changes in the difference value between GB. In this way, the difference values ​​change over time (as the stress state of the subject changes). Returning to FIG. 2, the functional configuration of calculation unit 3 will be continued.

[0038] The Fourier calculation unit 35 converts information about the difference values ​​for each time period calculated by the difference calculation unit 34 into the frequency axis. Specifically, the Fourier calculation unit 35 may convert into the frequency axis by performing an overlapping FFT (Fast Fourier Transformation) every second. In the following description, the Fourier calculation unit 35 may be simply referred to as the calculation unit. The calculation unit may also perform a process of converting the difference values ​​for each time period into the frequency axis using a known method other than Fourier analysis.

[0039] Based on the results of the conversion by the Fourier calculation unit 35, the output unit 36 ​​outputs the pulse wave amplitude value and the heart rate of the person being measured.

[0040] At least some of the functions of the calculation unit 3 may be configured using a neural network. For example, it is conceivable to replace the above-described calculation unit with a neural network. In this case, the calculation unit may be a machine learning model that has been trained in advance to infer the pulse wave amplitude value (or heart rate) from the difference value using a combination of the difference value and the pulse wave amplitude value (or heart rate) as training data. Furthermore, the function inferred by the machine learning model is not limited to the function of the calculation unit, and may further include, for example, at least one of the functions of the difference calculation unit 34, the statistical calculation unit 33, and the region identification unit 32.

[0041] FIG. 6 shows an example of pulse wave amplitude and heart rate obtained by the calculation device according to this embodiment. The figure shows the results of overlap FFT performed every second by the Fourier calculation unit 35. For each waveform shown, the horizontal axis represents frequency, and the vertical axis represents amplitude. The figure shows a spectrum ranging from 0.7 Hz (Hertz) to 4.0 Hz. As shown in the figure, the calculation device according to this embodiment can obtain the pulse wave amplitude and heart rate from peak values ​​between 0.7 Hz and 4.0 Hz. The maximum value of the spectrum is the pulse wave amplitude, and the heart rate [bpm (beats per minute)] is obtained by multiplying the frequency at which the pulse wave amplitude is measured by 60.

[0042] Note that, for example, when makeup is applied to an area identified by the area identification unit 32, the pulse wave amplitude and heart rate may not be accurately detected from the image information. Therefore, the calculation unit 3 may perform processing based on the results obtained from each of the multiple areas. Specifically, the area identification unit 32 identifies multiple areas, such as the nose, cheeks, and earlobes, using a predetermined algorithm. The multiple areas are assigned a predetermined priority order. The calculation unit 3 calculates the pulse wave amplitude and heart rate for each area, and if the calculation result is equal to or less than a threshold, determines that the calculation result is incorrect and excludes it from the list of output result candidates. The calculation unit 3 may output the measurement result for the highest-ranked area among the calculation results greater than the threshold.

[0043] FIG. 7 shows a comparison of the time-dependent change in pulse wave amplitude obtained based on the RG difference value according to this embodiment with the time-dependent change in pulse wave amplitude obtained using a photoplethysmograph. The horizontal axis of the figure represents time, and the vertical axis represents standardized pulse wave amplitude. To obtain the results shown in the figure, a photoplethysmograph (1 kHz) was attached to the subject's earlobe. At the same time, an image of the subject's face was captured using the content information providing device 1 according to this embodiment, and the pulse wave amplitude was measured using both the photoplethysmograph and a computing device included in the content information providing device 1. Waveform W21 represents the pulse wave amplitude measured using the photoplethysmograph, and waveform W22 represents the pulse wave amplitude measured using a computing device included in the content information providing device 1 according to this embodiment.

[0044] As is clear from the figure, when the pulse wave amplitude value increases over time in the measurement results of the photoplethysmograph, the pulse wave amplitude value also increases in the measurement results of the calculation device. In other words, the calculation device according to this embodiment can obtain measurement results similar to those of a contact-type photoplethysmograph without contact.

[0045] Alternatively, pulse wave amplitude values ​​may be standardized by first subtracting the average value of the pulse wave amplitude values ​​from the pulse wave amplitude data, and then dividing by the standard deviation. For example, if pulse wave amplitude data is X, μ is the average value of X, and σ is the standard deviation, the standardized pulse wave amplitude value z can be expressed by the following equation (1):

[0046] z=(X-μ) / σ (1)

[0047] FIG. 8 is a diagram showing the results of comparing the heart rate obtained by the calculation device according to this embodiment with the heart rate obtained by a photoplethysmography system. The figure shows the error obtained as a result of comparing the heart rate obtained by the calculation device provided in the content information providing device 1 according to this embodiment with the heart rate obtained by a photoplethysmograph. The figure also shows the results obtained by the method according to this embodiment and the results obtained by a conventional method. The method according to this embodiment shows the results obtained based on the difference values ​​between RG, RB, and GB. The heart rate shown in the figure is in units of [bpm].

[0048] The conventional technique involves acquiring pulse waves by performing independent component analysis (ICA) on images captured by a webcam. However, this technique has the drawback of sometimes being unable to separate pulse wave components.

[0049] It was found that the calculation method according to this embodiment, when based on the difference value between RG and GB, resulted in smaller errors than the conventional techniques. In particular, when based on the difference value between RG, the error was about half or less than that of the conventional techniques. Therefore, it can be said that this embodiment makes it possible to accurately extract reflected components that contain a large amount of pulse wave information. In other words, the calculation method according to this embodiment makes it possible to obtain pulse wave information with higher accuracy than the conventional techniques.

[0050] According to this embodiment, it is also possible to use a combination of the difference value between RG, the difference value between RB, and the difference value between GB. In this case, the average value of each difference value may be used, or weighting may be performed based on accuracy, etc. The weight may be different based on the region identified by the region identification unit 32, and the combination of each difference value may be different for each region.

[0051] 9 is a flowchart showing the flow of a series of processes performed by the calculation method according to this embodiment. With reference to this figure, an example of the calculation method performed by the calculation device according to this embodiment will be described.

[0052] (Step S11) First, video information is acquired by the video information acquisition unit 31. The video information is acquired, for example, from the imaging unit 2. The video information includes a plurality of frame images. The frame rate of the video information may be, for example, 30 [FPS] or 30 [FPS].

[0053] (Step S12) Next, the region identification unit 32 identifies a region to be measured for each frame image included in the video information. The region to be measured may be the nose, right cheek, left cheek, etc. It is preferable that the region to be measured is predetermined. Furthermore, depending on the imaging angle, the user's appearance, etc., the predetermined measurement region may not necessarily be displayed in the frame image, and analysis of the region may not be easy. Therefore, it is preferable that there are multiple regions to be measured. In this case, subsequent processing may be performed based on one of the multiple regions, or a combination of the multiple regions, etc.

[0054] (Step S13) Next, the statistical calculation unit 33 performs statistical calculations on the RGB pixel values. The statistical calculations performed by the statistical calculation unit 33 may be, for example, calculation of the average value of pixel values ​​in the region identified by the region identification unit 32. The region on which the statistical calculation unit 33 performs calculations may be multiple. Furthermore, the statistical calculation unit 33 does not necessarily have to perform statistical calculations on all RGB pixel values, but may perform statistical calculations on at least any two of the RGB pixel values.

[0055] (Step S14) Next, the difference calculation unit 34 calculates a difference value between any two statistically calculated values ​​(for example, average values) calculated by the statistical calculation unit 33. The difference value calculated by the difference calculation unit 34 may be, for example, a difference value between RGs, a difference value between RBs, or a difference value between GBs. Furthermore, the difference value calculated by the difference calculation unit 34 may be a combination of these difference values, or a combination of these difference values ​​weighted by a predetermined weight.

[0056] (Step S15) Next, Fourier calculation unit 35 converts the difference value calculated by difference calculation unit 34 into a frequency axis. Based on the result of frequency conversion by Fourier calculation unit 35, the pulse wave amplitude value and the heart rate can be calculated.

[0057] (Step S16) Finally, the output unit 36 ​​outputs the pulse wave amplitude value and the heart rate obtained in step S15. The output destination of the output unit 36 ​​may be, for example, the control unit 4 or a storage device (not shown).

[0058] 10 is a functional configuration diagram showing a modified example of the functional configuration of the arithmetic device according to this embodiment. Referring to the same figure, a arithmetic unit 3A, which is a modified example of the arithmetic unit 3, will be described. In the description of the arithmetic unit 3A, the same components as those in the arithmetic unit 3 will be denoted by the same reference numerals, and the description thereof may be omitted. The arithmetic unit 3A differs from the arithmetic unit 3 in that it further includes an inverse Fourier calculation unit 37.

[0059] The inverse Fourier calculation unit 37 performs an inverse Fourier transform based on the result of the transformation performed by the Fourier calculation unit 35. A pulse wave can be obtained by performing the inverse Fourier transform by the inverse Fourier calculation unit 37. The output unit 36 ​​further outputs the pulse wave calculated by the inverse Fourier calculation unit 37 instead of or in addition to the pulse wave amplitude value and heart rate calculated by the Fourier calculation unit 35.

[0060] FIG. 11 is a block diagram showing an example of the internal configuration of a computing device according to this embodiment. An example of the internal configuration of the computing device will be described with reference to this diagram. At least some of the functions of the computing device can be implemented using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be implemented using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices. The input / output devices 904 and 905 are input / output devices. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to a RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses an input / output port via the bus 906. All or part of the functional units of the arithmetic device according to this embodiment may be realized using hardware such as an ASIC, a PLD, or an FPGA. All or part of the functional units may be realized by a combination of software and hardware. The content information providing device 1 may also have an internal configuration as shown in the same figure.

[0061] [Summary of this embodiment] According to the embodiment described above, the computing device according to this embodiment includes the video information acquisition unit 31, which acquires video information including a plurality of consecutive frame images of a person to be measured; the region identification unit 32, which identifies the region of the person captured in the frame images as the measurement target; the statistical calculation unit 33, which performs statistical calculations on at least two of the RGB pixel values ​​of the consecutive frame images for the identified region; the difference calculation unit 34, which calculates the difference between at least two of the statistically calculated pixel values; the Fourier calculation unit 35, which converts information about the time-dependent difference values ​​calculated by the difference calculation unit 34 into a frequency axis; and the output unit 36, which outputs the pulse wave amplitude and heart rate of the person to be measured based on the results of the conversion by the Fourier calculation unit 35. In other words, the computing device according to this embodiment detects biometric information based on image information. Therefore, according to this embodiment, biometric information can be detected without contact.

[0062] Furthermore, according to this embodiment, the region identification unit 32 identifies a part of the face of a person captured in a frame image as the region to be measured. Here, a user of this embodiment often uses a content information providing device 1 equipped with an imaging unit 2 that captures an image of the user's face, such as a smartphone, a tablet terminal, or smart glasses. Therefore, according to this embodiment, by using the imaging unit 2 equipped in the content information providing device 1, it is possible to detect biometric information without providing new hardware used for biometric information detection. Therefore, according to this embodiment, it is possible to detect biometric information at low cost and without imposing a burden on the user.

[0063] Furthermore, according to this embodiment, the region specifying unit 32 specifies, as a region to be measured, a region of a part of the face of a person captured in a frame image that has less fat than the surrounding area. Because the region specifying unit 32 specifies, as a region to be measured, a region that has less fat than the surrounding area, the arithmetic device according to this embodiment can accurately obtain changes in blood vessels present in the dermis and subcutaneous tissue.

[0064] Furthermore, according to this embodiment, the statistical calculation unit 33 performs statistical calculations on at least the R and G pixel values ​​among the pixel values ​​of each of the R, G, and B components of a plurality of consecutive frame images, and the difference calculation unit 34 calculates the difference between the R and G pixel values ​​for which the statistical calculations have been performed. That is, according to this embodiment, biometric information is detected based on the R and G difference values. Here, referring to FIG. 8, among the R, G, and B difference values, the difference value between R and G had the smallest error. Therefore, according to this embodiment, biometric information can be detected with higher accuracy.

[0065] Furthermore, according to this embodiment, the statistical calculation unit 33 calculates the average value of at least any two of the RGB pixel values ​​of a plurality of consecutive frame images for the region identified by the region identification unit 32. In other words, the statistical calculation performed by the statistical calculation unit 33 is the calculation of an average value. Therefore, according to this embodiment, biometric information can be easily detected.

[0066] Furthermore, the calculation device according to this embodiment further includes an inverse Fourier calculation unit 37, which performs an inverse Fourier transform based on the result of the transformation performed by the Fourier calculation unit 35 to calculate a pulse wave. The output unit 36 ​​further outputs the pulse wave calculated by the inverse Fourier calculation unit 37. Therefore, according to this embodiment, a pulse wave can be detected without contact.

[0067] Note that all or part of the functions of each unit of the arithmetic device according to the above-described embodiment may be realized by recording a program for realizing these functions on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.

[0068] Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage units such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the above-mentioned programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.

[0069] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the spirit of the present invention. [Explanation of symbols]

[0070] 1...content information providing device, 2...imaging unit, 3...calculation unit, 4...control unit, 5...display unit, U...user, 31...video information acquisition unit, 32...area identification unit, 33...statistical calculation unit, 34...difference calculation unit, 35...Fourier calculation unit, 36...output unit, 37...inverse Fourier calculation unit

Claims

1. a video information acquisition unit that acquires video information including a plurality of consecutive frame images of a person being measured; an area specifying unit that specifies an earlobe of a person as a measurement target area from among areas of the person captured in the frame image; a statistical calculation unit that performs statistical calculations on at least two pixel values ​​of RGB of the plurality of consecutive frame images for the identified region; a difference calculation unit that calculates a difference between at least two pixel values ​​for which statistical calculation has been performed; a calculation unit for converting information about the difference value for each time calculated by the difference calculation unit into information on a frequency axis; an output unit that outputs the pulse wave amplitude value and the heart rate of the person being measured based on the results of the conversion by the calculation unit; Equipped with the region specifying unit specifies a plurality of regions as measurement target regions in addition to an earlobe of the person photographed in the frame image; the statistical calculation unit performs statistical calculations for each of the plurality of regions identified by the region identification unit; the difference calculation unit calculates a difference for each of the plurality of regions identified by the region identification unit; the calculation unit converts each of the plurality of regions identified by the region identification unit into a frequency axis; the output unit outputs a pulse wave amplitude value and a heart rate of the person to be measured based on at least one of the plurality of regions identified by the region identification unit. Computing device.

2. a priority order is set in advance for each of the plurality of regions identified by the region identification unit; the calculation unit calculates a pulse wave amplitude value and a heart rate for each of the plurality of regions identified by the region identification unit, and determines that the calculation result is incorrect when the calculation result is equal to or less than a threshold value; the output unit outputs the measurement result at the location with the highest priority among the results for which the calculation result is greater than the threshold. The computing device of claim 1 .

3. the statistical calculation unit performs statistical calculations on at least R and G pixel values ​​among the R, G, and B pixel values ​​of the plurality of consecutive frame images; The difference calculation unit calculates the difference between the R and G pixel values ​​on which the statistical calculation has been performed. The computing device of claim 1 .

4. The statistical calculation unit calculates an average value of at least two pixel values ​​of RGB of the plurality of consecutive frame images for the specified region. The computing device of claim 1 .

5. An inverse Fourier calculation unit is further provided which calculates a pulse wave by performing an inverse Fourier transform based on the result of the conversion by the calculation unit, The output unit further outputs the pulse wave calculated by the inverse Fourier calculation unit. The computing device according to any one of claims 1 to 4.

6. a video information acquisition step of acquiring video information including a plurality of consecutive frame images of a person to be measured; an area specifying step of specifying an earlobe of the person as an area to be measured from an area of ​​the person captured in the frame image; a statistical calculation step of performing statistical calculations on at least two pixel values ​​of RGB of the plurality of consecutive frame images for the identified region; a difference calculation step of calculating a difference between at least two pixel values ​​for which statistical calculation has been performed; a calculation step of converting information about the difference values ​​for each time calculated in the difference calculation step into a frequency axis; an output step of outputting the pulse wave amplitude value and the heart rate of the person being measured based on the results converted by the calculation step; and the region specifying step specifies a plurality of regions as measurement target regions in addition to the earlobe of the person photographed in the frame image; the statistical calculation step performs a statistical calculation for each of the plurality of regions identified by the region identification step; the difference calculation step calculates a difference for each of the plurality of regions identified by the region identification step; The calculation step converts each of the plurality of regions identified by the region identification step into a frequency axis, The output step outputs a pulse wave amplitude value and a heart rate of the person to be measured based on at least one of the plurality of regions identified by the region identification step. Calculation method.

7. On the computer, a video information acquisition step of acquiring video information including a plurality of consecutive frame images of a person to be measured; an area specifying step of specifying an earlobe of the person as an area to be measured from an area of ​​the person captured in the frame image; a statistical calculation step of performing statistical calculations on at least two pixel values ​​of RGB of the plurality of consecutive frame images for the identified region; a difference calculation step of calculating a difference between at least two pixel values ​​for which statistical calculation has been performed; a calculation step of converting information about the difference values ​​for each time calculated in the difference calculation step into a frequency axis; an output step of outputting the pulse wave amplitude value and the heart rate of the person being measured based on the results converted by the calculation step; Execute the region specifying step specifies a plurality of regions as measurement target regions in addition to an earlobe of the person photographed in the frame image; the statistical calculation step performs a statistical calculation for each of the plurality of regions identified by the region identification step, the difference calculation step calculates a difference for each of the plurality of regions identified by the region identification step; The calculation step converts each of the plurality of regions identified by the region identification step into a frequency axis, The output step outputs a pulse wave amplitude value and a heart rate of the person to be measured based on at least one of the plurality of regions identified by the region identification step. program.

Images