Computation device, computation method, and program

Abstract

This computation device comprises: a video information acquisition unit that acquires video information including a plurality of continuous frame images in which a person subjected to measurement is captured; a region identification unit that identifies a region subjected to measurement among regions of the person captured in the frame images; a statistics computation unit that performs, regarding the identified region, statistics computation on at least two pixel values among RGB pixel values of the plurality of continuous frame images; a difference calculation unit that calculates a difference between the at least two pixel values on which the statistics computation has been performed; a computation unit that transforms information about a time-based difference value, which has been calculated by the difference calculation unit, onto the frequency axis; and an output unit that outputs, on the basis of a result having been transformed by the computation unit, a pulse wave amplitude value and a heart rate of the person subjected to measurement.

Description

Arithmetic device, arithmetic method, and program

[0001] This application claims priority to Japanese Patent Application No. 2023-178767, filed on October 17, 2023, the contents of which are incorporated herein by reference.

[0002] Modern society is often called a stressful society, and mental illness caused by stress is a growing problem. Mental illness can be caused by a variety of factors, including interpersonal relationships at work, the workplace environment, and excessive work, as well as lack of exercise, sleep, and an unbalanced diet. In order to prevent stress-related mental illness, there is a demand for stress assessment tools that can easily measure stress levels at home and allow for lifelogging.

[0003] In recent years, psychological assessment questionnaires, which are subjective assessment methods, have been commonly used to assess stress. However, stress assessments using psychological assessment questionnaires have the problem that the results are easily influenced by the subject's physical condition and mood at the time. Therefore, as an objective assessment method, research has been conducted on methods that acquire physiological information from the subject, such as respiration, heart rate, saliva, and pulse wave, and perform stress assessment based on the physiological information. In particular, among these physiological information, pulse waves can be measured using inexpensive devices and can be implemented in wearable devices such as smart watches. Therefore, even though physiological information is relatively difficult to obtain, pulse waves can be easily measured. A biometric information detection device, for example, is known as an example of a device that embodies this technology (see, for example, Patent Document 1).

[0004] JP 2011-147746 A

[0005] The biometric information detection devices described in the above-mentioned documents can detect biometric information, such as pulse waves, based on scattered light obtained by flashing a sensor. For example, such biometric information detection devices can be used to detect changes in a user's biometric information while the user is browsing content, such as an e-book, on a smartphone. However, the penetration rate of wearable devices, such as smartwatches, equipped with such biometric information detection devices is not necessarily higher than that of smartphones. Therefore, to detect changes in a user's biometric information using such technology, the user must wear a wearable device, such as a smartwatch, equipped with the biometric information detection device, in addition to a smartphone for browsing content, such as an e-book. If biometric information could be detected non-contact using a smartphone or the like, the user would not need to wear the wearable device, and biometric information could be obtained even from users who are not wearing the wearable device.

[0006] Therefore, an object of the present invention is to provide a computing device, a computing method, and a program that are capable of detecting biometric information in a non-contact manner.

[0007] [1] In order to solve the above problem, one aspect of the present invention is a calculation device comprising: a video information acquisition unit that acquires video information including a plurality of consecutive frame images in which a person to be measured is captured; an area identification unit that identifies an area to be measured from among the areas of the person captured in the frame images; a statistical calculation unit that performs statistical calculations on at least two of the RGB pixel values ​​of the plurality of consecutive frame images for the identified area; a difference calculation unit that calculates the difference between at least two of the pixel values ​​for which the statistical calculation was performed; a calculation unit that converts information about the difference values ​​for each time period calculated by the difference calculation unit into a frequency axis; and an output unit that outputs a pulse wave amplitude value and a heart rate for the person to be measured based on the results of the conversion by the calculation unit.

[0008] [2] Furthermore, in one aspect of the present invention, in the calculation device described in [1] above, the area identification unit identifies a part of a face of a person captured in the frame image as the area to be measured.

[0009] [3] Furthermore, in one aspect of the present invention, in the computing device described in [2] above, the area identification unit identifies an area of ​​a part of the face of a person captured in the frame image that has less fat than the surrounding area as the area to be measured.

[0010] [4] Furthermore, one aspect of the present invention is a calculation device according to any one of [1] to [3] above, wherein the statistical calculation unit performs statistical calculations on at least the R and G pixel values ​​among the RGB pixel values ​​of a plurality of consecutive frame images, and the difference calculation unit calculates the difference between the R and G pixel values ​​for which the statistical calculations have been performed.

[0011] [5] Furthermore, one aspect of the present invention is a calculation device according to any one of [1] to [4] above, wherein the statistical calculation unit calculates the average value of at least any two pixel values ​​of RGB of a plurality of consecutive frame images for a specified region.

[0012] [6] Furthermore, in one aspect of the present invention, the calculation device according to any one of [1] to [5] above further comprises an inverse Fourier calculation unit that calculates a pulse wave by performing an inverse Fourier transform based on the result of the conversion by the calculation unit, and the output unit further outputs the pulse wave calculated by the inverse Fourier calculation unit.

[0013] [7] Also, one aspect of the present invention is a calculation method having a video information acquisition step of acquiring video information including a plurality of consecutive frame images in which a person to be measured is captured; an area identification step of identifying an area to be measured from within the area of ​​the person captured in the frame images; a statistical calculation step of performing statistical calculations on at least any two of the RGB pixel values ​​of the plurality of consecutive frame images for the identified area; a difference calculation step of calculating the difference between at least any two of the pixel values ​​for which the statistical calculation was performed; a calculation step of converting information about the difference values ​​for each time period calculated by the difference calculation step into a frequency axis; and an output step of outputting a pulse wave amplitude value and a heart rate for the person to be measured based on the results converted by the calculation step.

[0014] [8] Also, one aspect of the present invention is a program that causes a computer to execute the following steps: a video information acquisition step of acquiring video information including a plurality of consecutive frame images in which a person to be measured is captured; an area identification step of identifying an area to be measured from among the areas of the person captured in the frame images; a statistical calculation step of performing statistical calculations on at least two of the RGB pixel values ​​of the plurality of consecutive frame images for the identified area; a difference calculation step of calculating the difference between at least two of the pixel values ​​for which the statistical calculations were performed; a calculation step of converting information about the difference values ​​for each time period calculated by the difference calculation step into a frequency axis; and an output step of outputting a pulse wave amplitude value and a heart rate for the person to be measured based on the results converted by the calculation step.

[0015] According to the present invention, it is possible to provide a computing device, a computing method, and a program that are capable of detecting biometric information in a non-contact manner.

[0016] FIG. 1 is a functional configuration diagram showing an example of the functional configuration of a content information providing device according to the present embodiment. FIG. 1 is a functional configuration diagram showing an example of the functional configuration of a calculation device according to the present embodiment. FIG. 2 is a diagram for explaining an example of region specification performed by the calculation device according to the present embodiment. FIG. 3 is a diagram for explaining a method for acquiring pulse wave components by the calculation device according to the present embodiment. FIG. 4 is a diagram showing an example of RGB difference values ​​according to the present embodiment. FIG. 5 is a diagram showing an example of pulse wave amplitude values ​​and heart rates obtained by the calculation device according to the present embodiment. FIG. 6 is a diagram showing a comparison result between the change over time of pulse wave amplitude values ​​obtained based on RG difference values ​​according to the present embodiment and the change over time of pulse wave amplitude values ​​obtained by a photoplethysmograph. FIG. 7 is a diagram showing a comparison result between the heart rate obtained by the calculation device according to the present embodiment and the heart rate obtained by a photoplethysmograph. FIG. 8 is a flowchart showing the flow of a series of processes performed by a calculation method according to the present embodiment. FIG. 9 is a functional configuration diagram showing a modified example of the functional configuration of the calculation device according to the present embodiment. FIG. 10 is a block diagram showing an example of the internal configuration of the calculation device according to the present embodiment.

[0017] [Embodiments] The following describes in detail preferred embodiments of a computing device, a computing method, and a program according to aspects of the present invention, with reference to the accompanying drawings. Note that the aspects of the present invention are not limited to these embodiments and include various modifications or improvements. In other words, the components described below include those that would be easily conceivable to a person skilled in the art or that are substantially identical, and the components described below can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the present invention. Furthermore, in the drawings, the scale and number of components may differ from the scale and number of the actual structures to make each configuration easier to understand.

[0018] The arithmetic device according to this embodiment detects biometric information of a user U in a non-contact manner. The biometric information detected from the user U can be used for various analyses. The arithmetic device is not limited to being used for browsing content, and can be widely used. For example, the arithmetic device may be used for tests such as psychological evaluations or stress evaluations. Furthermore, analyses may be performed using the results of these evaluations, and the analysis results may be used when providing various services. Furthermore, the arithmetic device may be used in place of a pulse wave measuring device in situations where a pulse wave measuring device has traditionally been used.

[0019] Hereinafter, as an example, a case will be described in which the arithmetic device is used to select content to be presented to a user when the user browses content, adjust the method of presenting the content, etc. In this case, the arithmetic device is implemented in, for example, a content information providing device 1.

[0020] FIG. 1 is a functional configuration diagram showing an example of the functional configuration of a content information providing device according to this embodiment. First, the functional configuration of the content information providing device 1 according to this embodiment will be described with reference to the same diagram. The content information providing device 1 is a device used by a user U. The user U browses content by operating the content information providing device 1. The content provided by the content information providing device 1 may be e-books including manga and novels, videos including movies, images including photographs, music, services provided by SNS (Social Networking Service), etc. The content provided by the content information providing device 1 may be content stored in the device itself or may be web content stored in a server device (not shown). The content information providing device 1 may also be a general-purpose device such as a smartphone, a tablet terminal, or smart glasses.

[0021] The content information providing device 1 includes at least an imaging unit 2, a calculation unit 3, a control unit 4, and a display unit 5 as functional components. Each of these functional units may be implemented using, for example, electronic circuits. Each functional unit may also include internal storage means such as semiconductor memory or a magnetic hard disk drive, as necessary. Each function may also be implemented using software and a computer having a central processing unit (CPU). All or part of each functional unit may also be implemented using hardware such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a field-programmable gate array (FPGA). All or part of each functional unit may also be implemented using a combination of software and hardware.

[0022] The imaging unit 2 acquires video information of the user U. The video information includes a plurality of consecutive frame images. The frame images may be images captured by a CCD (Charge Coupled Device) camera using a CCD image sensor for video capture, or may be images captured by a CMOS (Complementary Metal Oxide Semiconductor) camera using a CMOS image sensor. The video information captured by the imaging unit 2 is preferably a color video including RGB pixel information. Furthermore, the video information captured by the imaging unit 2 is preferably an image of the user U's skin, and more preferably an image of the user U's face.

[0023] The calculation unit 3 analyzes the video information acquired by the imaging unit 2. The calculation unit 3 detects biometric information of the user U as a result of analyzing the video information. In other words, the calculation unit 3 can be said to detect biometric information of the user U based on video information obtained without contacting the user U (non-contact). The biometric information detected by the calculation unit 3 may include information indicating the stress state of the user U. In the following description, the calculation unit 3 may be referred to as a calculation device.

[0024] The control unit 4 acquires biometric information of the user U from the calculation unit 3. The control unit 4 controls the content to be presented to the user U based on the acquired biometric information of the user U. The control unit 4 can also provide suitable content depending on the stress state of the user U. For example, the content to be presented to the user U may be determined in advance based on the operation of the user U, and the control unit 4 may adjust the content to be presented to the user U based on the biometric information of the user U. This adjustment may include adjusting the timing of presenting the content, screen settings, etc. Note that the control unit 4 may select content to be presented to the user U based on the biometric information of the user U after the presentation of the content desired by the user U has been completed, and present the selected content to the user U.

[0025] The display unit 5 presents content to the user U in accordance with the control of the control unit 4. The content presented by the display unit 5 may include image information, video information, sound information, etc. The display unit 5 may be, for example, a liquid crystal display, an organic EL (Electroluminescence) display, etc.

[0026] FIG. 2 is a functional configuration diagram showing an example of the functional configuration of a calculation device according to this embodiment. An example of the functional configuration of the calculation device will be described with reference to the same diagram. The calculation device is the calculation unit 3 described above. The calculation unit 3 includes at least an image information acquisition unit 31, a region identification unit 32, a statistical calculation unit 33, a difference calculation unit 34, a Fourier calculation unit 35, and an output unit 36. Each of these functional units may be implemented using, for example, an electronic circuit. Each functional unit may also include internal storage means such as a semiconductor memory or a magnetic hard disk drive, as necessary. Each function may also be implemented by a computer having a CPU and software. All or part of each functional unit may also be implemented using hardware such as an ASIC, a PLD, or an FPGA. All or part of each functional unit may also be implemented by a combination of software and hardware.

[0027] The video information acquisition unit 31 acquires video information from the imaging unit 2. The video information includes a series of frame images. Each of the frame images captures an image of a person whose biometric information is to be measured. The person whose biometric information is to be measured may be a person to whom content information is to be provided by the content information providing device 1.

[0028] The area identifying unit 32 identifies an area to be measured from the area of ​​a person depicted in a frame image included in the video information acquired by the video information acquiring unit 31. Here, the area to be measured is the skin of the person depicted in the frame image. Preferably, the area to be measured may be the face of the person depicted in the frame image. In other words, the area identifying unit 32 identifies a part of the face of the person depicted in the frame image as the area to be measured.

[0029] It should be noted that the processing according to this embodiment does not need to be performed on all frame images included in the frame images contained in the video information acquired by the video information acquisition unit 31. The processing according to this embodiment may be performed on some of the frame images included in the video information. For example, if the frame rate of the video information is 60 FPS (Frames Per Second), the processing may be performed on each frame (i.e., at a granularity of 30 FPS).

[0030] 3 is a diagram illustrating an example of region identification performed by the arithmetic device according to this embodiment. Here, an example of a region identified by the region identification unit 32 will be described with reference to the same figure. The figure shows a frame image FI included in video information acquired by the video information acquisition unit 31. First, the region identification unit 32 detects a face portion FP from the frame image FI. Next, the region identification unit 32 identifies a predetermined region from the face portion FP.

[0031] Here, it is preferable that the region identification unit 32 identify, as the region to be measured, a region of the face of the person captured in the frame image FI that has less fat than the surrounding area. The region that has less fat than the surrounding area may be predetermined, and may be the cheek area, nose area, or the like. In the example shown in the figure, the region identification unit 32 identifies, as the regions to be measured, region AR1, which is the nose area, region AR2, which is the right cheek area, and region AR3, which is the left cheek area. Note that the region identification unit 32 may identify the forehead area as the region to be measured, but the forehead may be hidden by bangs, a hat, or the like depending on the user. Therefore, it is preferable that the region identification unit 32 identify a region that can be measured regardless of the user.

[0032] Note that depending on the user's appearance, what the user is wearing, and the like, the region identification unit 32 may be unable to identify the predetermined region. Therefore, the region identification unit 32 may be predetermined to identify multiple regions. If multiple regions are predetermined to be identified, the subsequent processing may be performed for each of the multiple regions, or processing may be performed for any of the regions based on a priority order or the like predetermined for each region (each body part). Returning to FIG. 2 , the description of the functional configuration of the calculation unit 3 continues.

[0033] The statistical calculation unit 33 acquires information about the region identified by the region identification unit 32. The statistical calculation unit 33 preferably acquires pixel information about the region identified by the region identification unit 32. The pixel information may include pixel values ​​for R (red), G (green), and B (blue) independently. Furthermore, the pixel information does not need to include pixel information for all RGB colors; preferably, it only needs to include pixel information for two of the RGB colors (R and G, G and B, or R and B). The statistical calculation unit 33 performs statistical calculations on at least two of the RGB pixel values ​​(e.g., R and G) for the identified region. The statistical calculation may, for example, be calculating the average value of multiple pixel values ​​included in the identified region. Note that the statistical calculation performed by the statistical calculation unit 33 is not limited to calculating an average value, and may also be calculating statistical values ​​such as the mode, maximum value, and minimum value.

[0034] The difference calculation unit 34 calculates the difference between at least two pixel values ​​for which statistical calculations such as calculation of an average value have been performed by the statistical calculation unit 33. For example, when the statistical calculation unit 33 calculates the average value of R and the average value of G for pixels included in a region identified by the region identification unit 32, the difference calculation unit 34 calculates the difference between the average value of R and the average value of G. Note that the region identification unit 32 may calculate the difference between the average value of G and the average value of B, or may calculate the difference between the average value of R and the average value of B.

[0035] FIG. 4 is a diagram illustrating a method for acquiring pulse wave components using a computing device according to this embodiment. As shown in the figure, the outermost layer of human skin is covered by the epidermis. Human skin has a three-layer structure consisting of the epidermis, dermis, and subcutaneous tissue, with arteries and veins running through the subcutaneous tissue. Furthermore, capillaries run through the area close to the epidermis. Here, the R, G, and B components of light reflected by human skin have different reflection depths (or, alternatively, they have different penetration depths into the subcutaneous tissue). Specifically, the B component is reflected by the epidermis, which is the shallowest layer. Furthermore, the G component is reflected by a layer deeper than the epidermis, where capillaries and arteries run. Furthermore, the R component is reflected by a layer deeper than the epidermis, where arteries and veins run.

[0036] This embodiment focuses on these characteristics and calculates the difference between any two of the RGB components to extract blood components from image information and further acquire pulse wave components. As described above, by determining areas with less fat than the surrounding area, rather than areas covered with fat, blood components can be extracted more accurately. Note that some users may wear makeup on their cheeks or nose. When makeup is applied, it may be difficult to extract blood components from image information. Therefore, the region identification unit 32 may identify areas where makeup is normally difficult to apply, such as the earlobes.

[0037] FIG. 5 is a diagram showing an example of RGB difference values ​​according to this embodiment. In the illustrated example, the horizontal axis represents time and the vertical axis represents the RGB difference values, showing changes in the difference values ​​over time. Specifically, waveform W1 shows changes in the difference value between R and G. Waveform W2 shows changes in the difference value between R and B. Waveform W3 shows changes in the difference value between G and B. In this way, the difference values ​​change over time (as the subject's stress state changes). Returning to FIG. 2, we will continue to explain the functional configuration of the calculation unit 3.

[0038] The Fourier calculation unit 35 converts information about the difference values ​​for each time period calculated by the difference calculation unit 34 into the frequency axis. Specifically, the Fourier calculation unit 35 may perform an overlapping FFT (Fast Fourier Transformation) every second to perform the conversion into the frequency axis. In the following description, the Fourier calculation unit 35 may be simply referred to as the calculation unit. The calculation unit may perform a process of converting the difference values ​​for each time period into the frequency axis using a known method other than Fourier analysis.

[0039] Based on the results of the conversion by the Fourier calculation unit 35, the output unit 36 ​​outputs the pulse wave amplitude value and the heart rate of the person being measured.

[0040] At least some of the functions of the calculation unit 3 may be configured using a neural network. For example, the above-described calculation unit may be replaced with a neural network. In this case, the calculation unit may be a machine learning model that has been trained in advance to infer the pulse wave amplitude value (or heart rate) from the difference value using a combination of the difference value and the pulse wave amplitude value (or heart rate) as training data. Furthermore, the function inferred by the machine learning model is not limited to the function of the calculation unit, and may further include, for example, at least one of the functions of the difference calculation unit 34, the statistical calculation unit 33, and the region identification unit 32.

[0041] FIG. 6 shows an example of pulse wave amplitude and heart rate obtained by the calculation device according to this embodiment. This figure shows the results of overlap FFT performed every second by the Fourier calculation unit 35. For each waveform shown, the horizontal axis represents frequency, and the vertical axis represents amplitude. This figure shows a spectrum ranging from 0.7 Hz (Hertz) to 4.0 Hz. As shown in the figure, the calculation device according to this embodiment can obtain the pulse wave amplitude and heart rate from peak values ​​between 0.7 Hz and 4.0 Hz. The maximum value of the spectrum is the pulse wave amplitude, and the heart rate [bpm (beats per minute)] is obtained by multiplying the frequency at which the pulse wave amplitude is measured by 60.

[0042] Note that, for example, when makeup is applied to an area identified by the area identification unit 32, the pulse wave amplitude and heart rate may not be accurately detected from the image information. Therefore, the calculation unit 3 may perform processing based on the results obtained from each of the multiple areas. Specifically, the area identification unit 32 identifies multiple areas, such as the nose, cheeks, and earlobes, using a predetermined algorithm. The multiple areas are prioritized in advance. The calculation unit 3 calculates the pulse wave amplitude and heart rate for each area, and if the calculation result is equal to or less than a threshold, determines that the calculation result is incorrect and excludes it from the list of output result candidates. The calculation unit 3 may output the measurement result for the highest-ranked area among the calculation results greater than the threshold.

[0043] FIG. 7 shows a comparison of the time-dependent change in pulse wave amplitude obtained based on the RG difference value according to the present embodiment with the time-dependent change in pulse wave amplitude obtained using a photoplethysmograph. The horizontal axis of the figure represents time, and the vertical axis represents standardized pulse wave amplitude. To obtain the results shown in the figure, a photoplethysmograph (1 kHz) was attached to the subject's earlobe. At the same time, the content information providing device 1 according to the present embodiment captured an image of the subject's face, and the pulse wave amplitude was measured using both the photoplethysmograph and a computing device included in the content information providing device 1. Waveform W21 represents the pulse wave amplitude measured using the photoplethysmograph, and waveform W22 represents the pulse wave amplitude measured using a computing device included in the content information providing device 1 according to the present embodiment.

[0044] As is clear from the figure, when the pulse wave amplitude value increases over time in the measurement results of the photoplethysmograph, the pulse wave amplitude value also increases in the measurement results of the calculation device. In other words, the calculation device according to this embodiment can obtain measurement results similar to those of a contact-type photoplethysmograph without contact.

[0045] Alternatively, the pulse wave amplitude may be standardized by first subtracting the average value of the pulse wave amplitude from the pulse wave amplitude data and then dividing by the standard deviation. For example, if the pulse wave amplitude data is X, μ is the average value of X, and σ is the standard deviation, the standardized pulse wave amplitude z can be expressed by the following equation (1):

[0046] z=(X-μ) / σ...(1)

[0047] FIG. 8 is a diagram showing the results of comparing the heart rate obtained by the calculation device according to this embodiment with the heart rate obtained by the photoplethysmography system. This figure shows the error obtained as a result of comparing the heart rate obtained by the calculation device included in the content information providing device 1 according to this embodiment with the heart rate obtained by the photoplethysmography system. This figure also shows the results obtained by the method according to this embodiment and the results obtained by a conventional method. The method according to this embodiment shows the results obtained based on the difference value between R and G, the difference value between R and B, and the difference value between G and B. The heart rate shown in this figure is in units of [bpm].

[0048] The conventional technique involves performing independent component analysis (ICA) on images captured by a webcam to obtain a pulse wave. However, the conventional technique has a problem in that it may not be possible to separate the pulse wave components.

[0049] It was found that the calculation method according to this embodiment, when based on the difference value between R and G and when based on the difference value between G and B, resulted in smaller errors than conventional techniques. In particular, when based on the difference value between R and G, the error was approximately half or less compared to conventional techniques. Therefore, it can be said that this embodiment makes it possible to accurately extract reflected components that contain a large amount of pulse wave information. In other words, the calculation method according to this embodiment makes it possible to obtain pulse wave information with higher accuracy than conventional techniques.

[0050] According to this embodiment, it is also possible to use a combination of the difference value between R and G, the difference value between R and B, and the difference value between G and B. In this case, the average value of each difference value may be used, or weighting may be performed based on accuracy or the like. The weighting may be different based on the region identified by the region identification unit 32, and the combination of each difference value may be different for each region.

[0051] 9 is a flowchart showing the flow of a series of processes performed by the calculation method according to this embodiment. With reference to this figure, an example of the calculation method performed by the calculation device according to this embodiment will be described.

[0052] (Step S11) First, video information is acquired by the video information acquisition unit 31. The video information is acquired, for example, from the imaging unit 2. The video information includes a plurality of frame images. The frame rate of the video information may be, for example, 30 [FPS] or 30 [FPS].

[0053] (Step S12) Next, the region identification unit 32 identifies a region to be measured for each frame image included in the video information. The region to be measured may be the nose, right cheek, left cheek, etc. It is preferable that the region to be measured is predetermined. Furthermore, depending on the imaging angle, the user's appearance, etc., the predetermined measurement region may not necessarily be displayed in the frame image, and analysis of the region may not be easy. Therefore, it is preferable that there are multiple regions to be measured. In this case, subsequent processing may be performed based on one of the multiple regions, or a combination of the multiple regions.

[0054] (Step S13) Next, the statistical calculation unit 33 performs statistical calculations on the RGB pixel values. The statistical calculations performed by the statistical calculation unit 33 may be, for example, calculation of the average value of pixel values ​​in the region identified by the region identification unit 32. The region on which the statistical calculation unit 33 performs calculations may be multiple. Furthermore, the statistical calculation unit 33 does not necessarily need to perform statistical calculations on all RGB pixel values, but may perform statistical calculations on at least any two of the RGB pixel values.

[0055] (Step S14) Next, the difference calculation unit 34 calculates a difference value between any two statistically calculated values ​​(e.g., average values) calculated by the statistical calculation unit 33. The difference value calculated by the difference calculation unit 34 may be, for example, a difference value between R and G, a difference value between R and B, or a difference value between G and B. Furthermore, the difference value calculated by the difference calculation unit 34 may be a combination of these difference values, or a combination of these difference values ​​weighted in a predetermined manner.

[0056] (Step S15) Next, the Fourier calculation unit 35 converts the difference value calculated by the difference calculation unit 34 into a frequency axis. Based on the result of the frequency conversion by the Fourier calculation unit 35, the pulse wave amplitude value and the heart rate can be calculated.

[0057] (Step S16) Finally, the output unit 36 ​​outputs the pulse wave amplitude value and the heart rate obtained in step S15. The output destination of the output unit 36 ​​may be, for example, the control unit 4 or a storage device (not shown).

[0058] 10 is a functional configuration diagram showing a modified example of the functional configuration of the arithmetic device according to this embodiment. Referring to the same figure, a arithmetic unit 3A, which is a modified example of the arithmetic unit 3, will be described. In the description of the arithmetic unit 3A, components similar to those of the arithmetic unit 3 will be denoted by similar reference numerals, and their description may be omitted. The arithmetic unit 3A differs from the arithmetic unit 3 in that it further includes an inverse Fourier calculation unit 37.

[0059] The inverse Fourier calculation unit 37 performs an inverse Fourier transform based on the result of the transform performed by the Fourier calculation unit 35. A pulse wave can be obtained by performing the inverse Fourier transform by the inverse Fourier calculation unit 37. The output unit 36 ​​further outputs the pulse wave calculated by the inverse Fourier calculation unit 37 instead of or in addition to the pulse wave amplitude value and heart rate calculated by the Fourier calculation unit 35.

[0060] FIG. 11 is a block diagram showing an example of the internal configuration of a computing device according to this embodiment. An example of the internal configuration of the computing device will be described with reference to this diagram. At least some of the functions of the computing device can be implemented using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be implemented using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices, etc. The input / output devices 904 and 905 are input / output devices. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906. Furthermore, all or part of the functional units provided in the arithmetic device according to this embodiment may be realized using hardware such as an ASIC, a PLD, or an FPGA. Furthermore, all or part of the functional units may be realized by a combination of software and hardware. Note that the content information providing device 1 may also have an internal configuration as shown in the same figure.

[0061] Summary of the Present Embodiment According to the embodiment described above, the arithmetic device according to the present embodiment includes a video information acquisition unit 31 for acquiring video information including a plurality of consecutive frame images of a person to be measured, a region identification unit 32 for identifying a region of the person captured in the frame images as a measurement target, a statistical calculation unit 33 for performing statistical calculations on at least two of the RGB pixel values ​​of the plurality of consecutive frame images for the identified region, a difference calculation unit 34 for calculating the difference between at least two of the pixel values ​​subjected to statistical calculation, a Fourier calculation unit 35 for converting information on the time-dependent difference values ​​calculated by the difference calculation unit 34 into a frequency axis, and an output unit 36 ​​for outputting a pulse wave amplitude value and a heart rate for the person to be measured based on the results of the conversion by the Fourier calculation unit 35. In other words, the arithmetic device according to the present embodiment detects biometric information based on image information. Therefore, according to the present embodiment, biometric information can be detected without contact.

[0062] Furthermore, according to this embodiment, the region identification unit 32 identifies a portion of a person's face captured in the frame image as the region to be measured. Here, a user of this embodiment often uses a content information providing device 1 equipped with an imaging unit 2 that captures an image of the user's face, such as a smartphone, tablet device, or smart glasses. Therefore, according to this embodiment, by using the imaging unit 2 equipped in the content information providing device 1, biometric information can be detected without providing new hardware for biometric information detection. Therefore, according to this embodiment, biometric information can be detected at low cost and without imposing a burden on the user.

[0063] Furthermore, according to this embodiment, the region identifying unit 32 identifies, as a region to be measured, a region of a part of the face of a person captured in a frame image that has less fat than the surrounding area. Because the region identifying unit 32 identifies, as a region to be measured, a region that has less fat than the surrounding area, the arithmetic device according to this embodiment can accurately obtain changes in blood vessels present in the dermis and subcutaneous tissue.

[0064] Furthermore, according to this embodiment, the statistical calculation unit 33 performs statistical calculations on at least the R and G pixel values ​​among the RGB pixel values ​​of a plurality of consecutive frame images, and the difference calculation unit 34 calculates the difference between the R and G pixel values ​​for which the statistical calculations have been performed. That is, according to this embodiment, biometric information is detected based on the R and G difference values. Here, referring to FIG. 8 , among the RGB difference values, the difference value between R and G had the smallest error. Therefore, according to this embodiment, biometric information can be detected with greater accuracy.

[0065] Furthermore, according to this embodiment, the statistical calculation unit 33 calculates the average value of at least any two of the RGB pixel values ​​of a plurality of consecutive frame images for the region identified by the region identification unit 32. In other words, the statistical calculation performed by the statistical calculation unit 33 is the calculation of an average value. Therefore, according to this embodiment, biometric information can be easily detected.

[0066] Furthermore, the calculation device according to this embodiment further includes an inverse Fourier calculation unit 37, which performs an inverse Fourier transform based on the result of the transformation performed by the Fourier calculation unit 35 to calculate a pulse wave. The output unit 36 ​​further outputs the pulse wave calculated by the inverse Fourier calculation unit 37. Therefore, according to this embodiment, a pulse wave can be detected in a non-contact manner.

[0067] Note that all or part of the functions of each unit of the arithmetic device according to the above-described embodiment may be realized by recording a program for realizing these functions on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.

[0068] Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage units such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the above-mentioned programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.

[0069] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the spirit of the present invention.

[0070] According to the present invention, biological information can be detected in a non-contact manner.

[0071] 1...Content information providing device, 2...imaging unit, 3...calculation unit, 4...control unit, 5...display unit, U...user, 31...video information acquisition unit, 32...area identification unit, 33...statistical calculation unit, 34...difference calculation unit, 35...Fourier calculation unit, 36...output unit, 37...inverse Fourier calculation unit

Claims

1. A calculation device comprising: a video information acquisition unit which acquires video information including a series of frame images in which a person to be measured is captured; an area identification unit which identifies the area to be measured from among the areas of the person captured in the frame images; a statistical calculation unit which performs statistical calculations on at least any two of the RGB pixel values ​​of the series of frame images for the identified area; a difference calculation unit which calculates the difference between at least any two pixel values ​​for which statistical calculations have been performed; a calculation unit which converts information on the difference values ​​for each time period calculated by the difference calculation unit into a frequency axis; and an output unit which outputs a pulse wave amplitude value and a heart rate for the person to be measured based on the results converted by the calculation unit.

2. The computing device according to claim 1, wherein the area specifying section specifies a part of a face of a person captured in the frame image as the area to be measured.

3. The computing device according to claim 2, wherein the region specifying unit specifies, as the region to be measured, a region of the face of the person captured in the frame image that has less fat compared to the surrounding area.

4. The calculation device according to claim 1, wherein the statistical calculation unit performs statistical calculations on at least R and G pixel values ​​among the pixel values ​​of RGB of a plurality of consecutive frame images, and the difference calculation unit calculates the difference between the R and G pixel values ​​for which the statistical calculations have been performed.

5. The computing device according to claim 1, wherein the statistical computing unit computes an average value for at least any two pixel values ​​of RGB of a plurality of consecutive frame images for the identified region.

6. The calculation device according to any one of claims 1 to 5, further comprising an inverse Fourier calculation unit that calculates a pulse wave by performing an inverse Fourier transform based on the result of the conversion by said calculation unit, and said output unit further outputs the pulse wave calculated by said inverse Fourier calculation unit.

7. A calculation method comprising: a video information acquisition step of acquiring video information including a plurality of consecutive frame images in which a person to be measured is captured; an area identification step of identifying an area to be measured from within the area of ​​the person captured in the frame images; a statistical calculation step of performing statistical calculations on at least any two of the RGB pixel values ​​of the plurality of consecutive frame images for the identified area; a difference calculation step of calculating the difference between at least any two pixel values ​​for which statistical calculations have been performed; a calculation step of converting information on the difference values ​​for each time period calculated by the difference calculation step into a frequency axis; and an output step of outputting a pulse wave amplitude value and a heart rate for the person to be measured based on the results converted by the calculation step.

8. A program causing a computer to execute the following steps: a video information acquisition step of acquiring video information including a plurality of consecutive frame images in which a person to be measured is captured; an area identification step of identifying an area to be measured from among the areas of the person captured in the frame images; a statistical calculation step of performing a statistical calculation on at least any two of the RGB pixel values ​​of the plurality of consecutive frame images for the identified area; a difference calculation step of calculating the difference between at least any two pixel values ​​for which the statistical calculation has been performed; a calculation step of converting information about the difference values ​​for each time period calculated by the difference calculation step into a frequency axis; and an output step of outputting a pulse wave amplitude value and a heart rate for the person to be measured based on the results converted by the calculation step.

Images