Arithmetic device, arithmetic method, and program

Abstract

To detect biological information in a non-contact manner.SOLUTION: An arithmetic device includes: a video information acquisition part for acquiring video information including a plurality of continuing frame images where a person to be measured is photographed; an area specification part for specifying an area to be measured of areas of the person photographed in the frame image; a statistic arithmetic part for performing statistical arithmetic of at least any two pixel values of pixel values of RGB of a plurality of continuing frame images, for the specified area; a difference calculation part for calculating a difference for any two pixel values to which the statistical arithmetic is performed; an operation part for converting information about a difference value every time calculated by the difference calculation part into a frequency axis; and an output part for outputting a pulse wave amplitude value and a heart rate about a person to be measured on the basis of the result converted by the operation part.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a computing device, a computing method, and a program. [Background technology]

[0002] Modern society is often called a stressful society, and mental illness caused by stress is considered a problem. Mental illness is caused by a variety of factors, including interpersonal relationships at work, the work environment, and excessive work, as well as lack of exercise, sleep, and dietary balance. In order to prevent mental illness caused by stress, there is a demand for stress assessment that allows the degree of stress to be easily measured at home and allows life logging.

[0003] In recent years, as a method for performing stress evaluation, a psychological evaluation questionnaire, which is a subjective evaluation method, is generally used. However, there is a problem in that the results of stress evaluation using a psychological evaluation questionnaire are easily influenced by the physical condition and mood at that time. Therefore, as an objective evaluation method, a method of acquiring physiological information such as breathing, heart rate, saliva, and pulse wave from a subject and performing stress evaluation based on the physiological information has been studied. In particular, among these physiological information, a device for measuring pulse wave is inexpensive and the measuring device can be implemented in a wearable device such as a smart watch. Therefore, even among physiological information that is relatively difficult to acquire, pulse wave can be easily measured. As a device that embodies such a technology, for example, a biological information detection device is known (see Patent Document 1, for example). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2011-147746 A Summary of the Invention [Problem to be solved by the invention]

[0005] According to the bioinformation detection device described in the above-mentioned document, it is possible to blink a sensor and detect bioinformation such as a pulse wave based on the scattered light obtained by blinking the sensor. For example, it is possible to detect a change in a user's bioinformation when browsing content such as an e-book on a smartphone by using such a bioinformation detection device. However, the penetration rate of wearable devices such as smart watches equipped with such a bioinformation detection device is not necessarily high compared to the penetration rate of smartphones. Therefore, when trying to detect a change in a user's bioinformation using such a technology, the user needs to wear a wearable device such as a smart watch equipped with the bioinformation detection device in addition to a smartphone for browsing content such as an e-book. If it becomes possible to detect bioinformation in a non-contact manner using a smartphone or the like, the user does not need to wear the wearable device, and it is also possible to obtain bioinformation from a user who is not wearing the wearable device.

[0006] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to provide a computing device, a computing method, and a program capable of detecting biological information in a non-contact manner. [Means for solving the problem]

[0007] [1] In order to solve the above problems, one aspect of the present invention is a calculation device including: a video information acquisition unit that acquires video information including a plurality of consecutive frame images in which a person to be measured is captured; an area identification unit that identifies an area to be measured from among the areas of the person captured in the frame images; a statistical calculation unit that performs statistical calculations on at least any two pixel values ​​of the RGB pixel values ​​of the plurality of consecutive frame images for the identified area; a difference calculation unit that calculates the difference between at least any two pixel values ​​for which the statistical calculations were performed; a calculation unit that converts information about the difference values ​​for each time period calculated by the difference calculation unit into a frequency axis; and an output unit that outputs a pulse wave amplitude value and a heart rate for the person to be measured based on the results converted by the calculation unit.

[0008] [2] Furthermore, in one aspect of the present invention, in the calculation device described in [1] above, the area identification unit identifies a part of a face of a person captured in the frame image as the area to be measured.

[0009] [3] Moreover, one aspect of the present invention is that in the computing device described in [2] above, the area identification unit identifies, as the area to be measured, an area of ​​a part of a face of a person captured in the frame image that has less fat compared to the surrounding area.

[0010] [4] Furthermore, in one aspect of the present invention, in the calculation device described in any one of [1] to [3] above, the statistical calculation unit performs statistical calculation on at least the R and G pixel values ​​among the R, G and B pixel values ​​of a plurality of consecutive frame images, and the difference calculation unit calculates the difference between the R and G pixel values ​​for which the statistical calculation was performed.

[0011] [5] Furthermore, in one aspect of the present invention, in the calculation device described in any one of [1] to [4] above, the statistical calculation unit calculates an average value for at least any two pixel values ​​of each of the RGB pixel values ​​of a plurality of consecutive frame images for a specified area.

[0012] [6] Furthermore, one aspect of the present invention is a calculation device according to any one of [1] to [5] above, further comprising an inverse Fourier calculation unit that calculates a pulse wave by performing an inverse Fourier transform based on the result of the conversion by the calculation unit, and the output unit further outputs the pulse wave calculated by the inverse Fourier calculation unit.

[0013] [7] Moreover, one aspect of the present invention is a calculation method having a video information acquisition step of acquiring video information including a plurality of consecutive frame images in which a person to be measured is captured; an area identification step of identifying an area to be measured among the areas of the person captured in the frame images; a statistical calculation step of performing a statistical calculation on at least any two pixel values ​​of the RGB pixel values ​​of the plurality of consecutive frame images for the identified area; a difference calculation step of calculating the difference between at least any two pixel values ​​for which the statistical calculation was performed; a calculation step of converting information about the difference values ​​for each time calculated by the difference calculation step into a frequency axis; and an output step of outputting a pulse wave amplitude value and a heart rate for the person to be measured based on the results converted by the calculation step.

[0014] [8] Also, one aspect of the present invention is a program that causes a computer to execute the following steps: a video information acquisition step of acquiring video information including a plurality of consecutive frame images in which a person to be measured is captured; an area identification step of identifying an area to be measured from among the areas of the person captured in the frame images; a statistical calculation step of performing a statistical calculation on at least any two of the RGB pixel values ​​of the plurality of consecutive frame images for the identified area; a difference calculation step of calculating the difference between at least any two pixel values ​​for which the statistical calculation was performed; a calculation step of converting information about the difference values ​​for each time period calculated by the difference calculation step into a frequency axis; and an output step of outputting a pulse wave amplitude value and a heart rate for the person to be measured based on the results converted by the calculation step. Effect of the Invention

[0015] According to the present invention, it is possible to provide a calculation device, a calculation method, and a program capable of detecting biological information in a non-contact manner. [Brief description of the drawings]

[0016] [Figure 1] FIG. 2 is a functional configuration diagram showing an example of a functional configuration of the content information providing device according to the present embodiment. [Diagram 2] FIG. 2 is a functional configuration diagram showing an example of the functional configuration of a calculation device according to the present embodiment. [Diagram 3] 11 is a diagram for explaining an example of region specification performed by the calculation device according to the embodiment; FIG. [Figure 4] 10 is a diagram for explaining a method for acquiring a pulse wave component by a calculation device according to the present embodiment. FIG. [Diagram 5] 6 is a diagram showing an example of RGB difference values ​​according to the embodiment; FIG. [Figure 6] 4A to 4C are diagrams illustrating an example of a pulse wave amplitude value and a heart rate obtained by the calculation device according to the present embodiment. [Figure 7] FIG. 13 is a diagram showing a comparison result between the change over time of the pulse wave amplitude value obtained based on the RG difference value in this embodiment and the change over time of the pulse wave amplitude value obtained by a photoplethysmograph. [Figure 8] FIG. 13 is a diagram showing a comparison result between the heart rate obtained by the calculation device according to the present embodiment and the heart rate obtained by a photoplethysmography system. [Figure 9] 4 is a flowchart showing the flow of a series of processes performed by a calculation method according to the present embodiment. [Figure 10] FIG. 11 is a functional configuration diagram showing a modified example of the functional configuration of the arithmetic device according to the present embodiment. [Figure 11] 2 is a block diagram showing an example of the internal configuration of a calculation device according to the present embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] [Embodiment] Hereinafter, the arithmetic device, the arithmetic method, and the program according to the aspects of the present invention will be described in detail with reference to the accompanying drawings, showing preferred embodiments. Note that the aspects of the present invention are not limited to these embodiments, and include various modifications or improvements. In other words, the components described below include those that a person skilled in the art can easily imagine, and those that are substantially the same, and the components described below can be appropriately combined. In addition, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the present invention. In addition, in the following drawings, the scale and number of each structure may be different from the scale and number of the actual structure in order to make each structure easier to understand.

[0018] The arithmetic device according to this embodiment detects biometric information of a user U in a non-contact manner. The biometric information detected from the user U can be used for various analyses. The arithmetic device is not limited to being used for browsing content, and can be widely used. For example, the arithmetic device may be used for tests such as psychological evaluations, or for stress evaluations. Furthermore, analyses may be performed using the results of these evaluations, and the analysis results may be used when providing various services. Furthermore, the arithmetic device may be used in place of a pulse wave measuring device in situations where a pulse wave measuring device has conventionally been used.

[0019] Hereinafter, as an example, a case will be described in which the arithmetic device is used to select content to be presented to a user when the user browses content, adjust the method of presenting the content, etc. In this case, the arithmetic device is implemented in, for example, the content information providing device 1.

[0020] FIG. 1 is a functional configuration diagram showing an example of the functional configuration of the content information providing device according to this embodiment. First, the functional configuration of the content information providing device 1 according to this embodiment will be described with reference to the same figure. The content information providing device 1 is a device used by a user U. The user U browses content by operating the content information providing device 1. The content provided by the content information providing device 1 may be electronic books including comics and novels, videos including movies, images including photos, music, services provided by SNS (Social Networking Service), and the like. The content provided by the content information providing device 1 may be stored in the device itself, or may be Web content stored in a server device (not shown). The content information providing device 1 may be a general-purpose device such as a smartphone, a tablet terminal, or smart glasses.

[0021] The content information providing device 1 is configured to include at least an imaging unit 2, a calculation unit 3, a control unit 4, and a display unit 5 as functional components. Each of these functional units may be realized, for example, by using electronic circuits. Each functional unit may also include internal storage means such as a semiconductor memory or a magnetic hard disk device, as necessary. Each function may also be realized by a computer having a CPU (Central Processing Unit) and software. All or a part of each functional unit may also be realized by hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field-Programmable Gate Array). All or a part of each functional unit may also be realized by a combination of software and hardware.

[0022] The imaging unit 2 acquires video information of the user U. The video information includes a plurality of consecutive frame images. The frame images may be images captured by a CCD camera using a CCD (Charge Coupled Devices) image sensor for video capture, or may be images captured by a CMOS camera using a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The video information captured by the imaging unit 2 is preferably a color video including each pixel information of RGB. Moreover, the video information captured by the imaging unit 2 is preferably an image of the skin of the user U, and more preferably an image of the face of the user U.

[0023] The calculation unit 3 analyzes the video information acquired by the imaging unit 2. The calculation unit 3 detects bio-information of the user U as a result of analyzing the video information. In other words, it can be said that the calculation unit 3 detects bio-information of the user U based on video information acquired without contacting the user U (non-contact). The bio-information detected by the calculation unit 3 may include information indicating the stress state of the user U. In the following description, the calculation unit 3 may be referred to as a calculation device.

[0024] The control unit 4 acquires biometric information of the user U from the calculation unit 3. The control unit 4 controls the content to be presented to the user U based on the acquired biometric information of the user U. The control unit 4 can also provide suitable content according to the stress state of the user U. For example, the content to be presented to the user U is determined in advance based on the operation of the user U, and the control unit 4 may adjust the content to be presented to the user U based on the biometric information of the user U. The adjustment may be to adjust the timing of presenting the content, the screen settings, etc. Note that the control unit 4 may select the content to be presented to the user U based on the biometric information of the user U after the presentation of the content desired by the user U is completed, and present the selected content to the user U.

[0025] The display unit 5 presents content to the user U in response to control by the control unit 4. The content presented by the display unit 5 may include image information, video information, sound information, etc. The display unit 5 may be, for example, a liquid crystal display, an organic EL (Electroluminescence) display, etc.

[0026] FIG. 2 is a functional configuration diagram showing an example of the functional configuration of the arithmetic device according to this embodiment. With reference to the same figure, an example of the functional configuration of the arithmetic device will be described. The arithmetic device is, that is, the arithmetic unit 3 described above. The arithmetic unit 3 is configured to include at least an image information acquisition unit 31, an area identification unit 32, a statistical calculation unit 33, a difference calculation unit 34, a Fourier calculation unit 35, and an output unit 36. Each of these functional units may be realized, for example, using an electronic circuit. Furthermore, each functional unit may have a storage means such as a semiconductor memory or a magnetic hard disk device inside as necessary. Furthermore, each function may be realized by a computer having a CPU and software. Furthermore, all or a part of each functional unit may be realized by hardware such as ASIC, PLD, or FPGA. Furthermore, all or a part of each functional unit may be realized by a combination of software and hardware.

[0027] The video information acquiring unit 31 acquires video information from the imaging unit 2. The video information includes a series of multiple frame images. In each of the multiple frame images, an image of a person who is a measurement target of biometric information is captured. The person who is a measurement target of biometric information may be a person to whom content information is provided by the content information providing device 1.

[0028] The area identifying unit 32 identifies an area to be measured from the area of ​​a person captured in a frame image included in the video information acquired by the video information acquiring unit 31. Here, the area to be measured is the skin of the person captured in the frame image. Preferably, the area to be measured may be the face of the person captured in the frame image. In other words, the area identifying unit 32 identifies a part of the face of the person captured in the frame image as the area to be measured.

[0029] It should be noted that the processing according to this embodiment does not need to be performed on all frame images included in the frame images contained in the video information acquired by the video information acquisition unit 31. The processing according to this embodiment may be performed on some of the frame images included in the video information. For example, if the frame rate of the video information is 60 [FPS (Frames Per Seconds)], the processing may be performed on each image (i.e., at a granularity of 30 [FPS]).

[0030] 3 is a diagram for explaining an example of region specification performed by the arithmetic device according to this embodiment. Here, an example of a region specified by the region specification unit 32 will be described with reference to the same figure. The same figure shows a frame image FI included in video information acquired by the video information acquisition unit 31. First, the region specification unit 32 detects a face portion FP from the frame image FI. Next, the region specification unit 32 specifies a predetermined region from the face portion FP.

[0031] Here, it is preferable that the region specifying unit 32 specifies, as the region to be measured, a region of the face of the person captured in the frame image FI that has less fat compared to the surrounding area. The region that has less fat compared to the surrounding area may be predetermined, and may be a cheek area, a nose area, or the like. In the illustrated example, the region specifying unit 32 specifies, as the regions to be measured, an area AR1 that is the nose area, an area AR2 that is the right cheek area, and an area AR3 that is the left cheek area. Note that the region specifying unit 32 may specify the forehead area as the region to be measured, but the forehead area may be hidden by bangs, a hat, or the like, depending on the user. Therefore, it is preferable that the region specifying unit 32 specifies a region that can be measured regardless of the user.

[0032] Depending on the user's appearance, what the user is wearing, etc., the region identification unit 32 may not be able to identify the predetermined region. Therefore, the region identification unit 32 may be predetermined to identify a plurality of regions. When it is predetermined to identify a plurality of regions, the subsequent processing may be performed for each of the plurality of regions, or processing may be performed for any of the regions based on a priority order, etc., predetermined for each region (each part). Returning to FIG. 2, the explanation of the functional configuration of the calculation unit 3 will be continued.

[0033] The statistical calculation unit 33 acquires information about the region specified by the region specification unit 32. The statistical calculation unit 33 preferably acquires pixel information about the region specified by the region specification unit 32. The pixel information may include pixel values ​​of R (red), G (green), and B (blue) independently. The pixel information does not need to include all pixel information of RGB, and preferably includes pixel information of two of RGB (R and G, G and B, or R and B). The statistical calculation unit 33 performs statistical calculation on at least any two pixel values ​​(e.g., R and G) of the pixel values ​​of RGB for the specified region. The statistical calculation may be, for example, calculating the average value of a plurality of pixel values ​​included in the specified region. Note that the statistical calculation performed by the statistical calculation unit 33 is not limited to calculating the average value, and may be, for example, calculating statistical values ​​such as the mode, maximum value, and minimum value.

[0034] The difference calculation unit 34 calculates the difference between at least any two pixel values ​​for which statistical calculation such as calculation of an average value has been performed by the statistical calculation unit 33. For example, when the statistical calculation unit 33 calculates the average value of R and the average value of G for pixels included in a region identified by the region identification unit 32, the difference calculation unit 34 calculates the difference between the average value of R and the average value of G. Note that the region identification unit 32 may calculate the difference between the average value of G and the average value of B, or may calculate the difference between the average value of R and the average value of B.

[0035] FIG. 4 is a diagram for explaining a method of acquiring a pulse wave component by a calculation device according to this embodiment. Here, as shown in the figure, the outermost layer of human skin is covered by the epidermis. Human skin has a three-layer structure of the epidermis, dermis, and subcutaneous tissue, and arteries and veins run through the subcutaneous tissue. Capillaries run through the area close to the epidermis. Here, the light reflected by human skin has different reflection depths for the R component, G component, and B component (it can also be said that the depths that reach the subcutaneous tissue are different). Specifically, the B component is reflected by the shallowest epidermis. The G component is reflected by a layer deeper than the epidermis where capillaries and arteries run. The R component is reflected by a layer where the deepest arteries and veins run.

[0036] According to the present embodiment, by focusing on such characteristics, the difference between any two components of RGB is calculated to extract blood components from image information, and furthermore, to obtain pulse wave components. As described above, by determining an area with less fat compared to the surrounding area, rather than a part covered with fat, it becomes possible to extract blood components with higher accuracy. Note that, depending on the user, makeup may be applied to the cheeks or nose. When makeup is applied, it may be difficult to extract blood components from image information. Therefore, the area identification unit 32 may identify an area where makeup is usually difficult to apply, such as an earlobe.

[0037] FIG. 5 is a diagram showing an example of RGB difference values ​​according to this embodiment. In the illustrated example, the horizontal axis represents time and the vertical axis represents RGB difference values, showing changes in difference values ​​over time. Specifically, waveform W1 shows changes in difference values ​​between RG. Waveform W2 shows changes in difference values ​​between RB. Waveform W3 shows changes in difference values ​​between GB. In this way, the difference values ​​change over time (as the stress state of the subject changes). Returning to FIG. 2, the explanation of the functional configuration of the calculation unit 3 will be continued.

[0038] The Fourier calculation unit 35 converts information on the difference value for each time calculated by the difference calculation unit 34 into the frequency axis. Specifically, the Fourier calculation unit 35 may convert into the frequency axis by performing overlap FFT (Fast Fourier Transformation) every second. In the following description, the Fourier calculation unit 35 may be simply referred to as the calculation unit. The calculation unit may perform a process of converting the difference value for each time into the frequency axis by a known method other than Fourier analysis.

[0039] Based on the results of the conversion by Fourier calculation section 35, output section 36 outputs the pulse wave amplitude value and the heart rate of the person being measured.

[0040] At least some of the functions of the calculation unit 3 may be configured using a neural network. For example, it is possible to replace the above-mentioned calculation unit with a neural network. In this case, the calculation unit may be a machine learning model that has been trained in advance to infer the pulse wave amplitude value (or heart rate) from the difference value using a combination of the difference value and the pulse wave amplitude value (or heart rate) as teacher data. In addition, the function inferred by the machine learning model is not limited to the function of the calculation unit, and may further include at least one of the functions of the difference calculation unit 34, the statistical calculation unit 33, and the area identification unit 32, for example.

[0041] FIG. 6 is a diagram showing an example of a pulse wave amplitude value and a heart rate obtained by the calculation device according to this embodiment. The diagram shows the result of overlap FFT performed every second by the Fourier calculation unit 35. For each of the illustrated waveforms, the horizontal axis indicates the frequency, and the vertical axis indicates the amplitude. The diagram shows a spectrum in the range of 0.7 [Hz (Hertz)] to 4.0 [Hz]. As shown in the diagram, the calculation device according to this embodiment can obtain the pulse wave amplitude value and the heart rate from the peak value between 0.7 [Hz] and 4.0 [Hz]. Note that the maximum value of the spectrum is the pulse wave amplitude value, and the heart rate [bpm (beats per minute)] is obtained by multiplying the frequency at which the pulse wave amplitude value is measured by 60.

[0042] In addition, when makeup is applied to the area specified by the area specifying unit 32, the pulse wave amplitude value and the heart rate may not be detected accurately from the image information. Therefore, the calculation unit 3 may perform processing based on the results obtained from each of the multiple areas. Specifically, the area specifying unit 32 specifies multiple areas, such as the nose, cheeks, and earlobes, using a predetermined algorithm. The multiple areas are assumed to have a predetermined priority order. The calculation unit 3 calculates the pulse wave amplitude value and the heart rate for each area, and if the calculation result is equal to or less than a threshold, the calculation result is determined to be incorrect and is excluded from the candidates for the output result. The calculation unit 3 may output the result measured at the location with the highest priority among the results for which the calculation result is greater than the threshold.

[0043] FIG. 7 is a diagram showing a comparison result between the change over time of the pulse wave amplitude value obtained based on the RG difference value according to this embodiment and the change over time of the pulse wave amplitude value obtained by a photoplethysmograph. The horizontal axis of the figure shows time, and the vertical axis shows the standardized pulse wave amplitude value. To obtain the results shown in the figure, a photoplethysmograph (1 kHz) was attached to the earlobe of the subject, and at the same time, the subject's face was imaged by the content information providing device 1 according to this embodiment, and the pulse wave amplitude value was measured by both the photoplethysmograph and the calculation device provided in the content information providing device 1. The waveform W21 shows the pulse wave amplitude value measured by the photoplethysmograph, and the waveform W22 shows the pulse wave amplitude value measured by the calculation unit device provided in the content information providing device 1 according to this embodiment.

[0044] As is clear from the figure, when the pulse wave amplitude value increases over time in the measurement results of the photoplethysmograph, the pulse wave amplitude value also increases in the measurement results of the calculation device. In other words, the calculation device according to this embodiment makes it possible to obtain measurement results similar to those of a contact-type photoplethysmograph without contact.

[0045] The pulse wave amplitude may be standardized by first subtracting the average value of the pulse wave amplitude from the pulse wave amplitude data, and then dividing by the standard deviation. For example, if the pulse wave amplitude data is X, μ is the average value of X, and σ is the standard deviation, the standardized pulse wave amplitude z can be expressed by the following formula (1).

[0046] z = (X - μ) / σ (1)

[0047] FIG. 8 is a diagram showing a comparison result between the heart rate obtained by the calculation device according to this embodiment and the heart rate obtained by the photoplethysmograph. The figure shows an error obtained as a result of comparing the heart rate obtained by the calculation device included in the content information providing device 1 according to this embodiment and the heart rate obtained by the photoplethysmograph. The figure also shows the results obtained by the method according to this embodiment and the results obtained by the conventional method. In the method according to this embodiment, the results obtained based on the difference value between RG, the difference value between RB, and the difference value between GB are shown. The unit of the heart rate shown in the figure is [bpm].

[0048] The conventional technique is a method for acquiring a pulse wave by performing independent component analysis (ICA) based on an image captured by a web camera. However, the conventional technique has a problem in that it is sometimes not possible to separate the pulse wave components.

[0049] According to the calculation method of this embodiment, it was found that the error was smaller than that of the conventional technique when based on the difference value between RG and when based on the difference value between GB. In particular, when based on the difference value between RG, the error was about half or less than that of the conventional technique. Therefore, according to this embodiment, it can be said that it is possible to extract the reflected component containing a large amount of pulse wave information with high accuracy. In other words, the calculation method of this embodiment is capable of acquiring pulse wave information with high accuracy compared to the conventional technique.

[0050] According to the present embodiment, it is also possible to use a combination of the difference value between RG, the difference value between RB, and the difference value between GB. In this case, the average value of each difference value may be used, or weighting may be performed based on accuracy or the like. The weighting may be different based on the region specified by the region specifying unit 32, and the combination of each difference value may be different for each region.

[0051] 9 is a flow chart showing the flow of a series of processes performed by the calculation method according to this embodiment. With reference to this figure, an example of the calculation method performed by the calculation device according to this embodiment will be described.

[0052] (Step S11) First, video information is acquired by the video information acquisition unit 31. The video information is acquired, for example, from the imaging unit 2. The video information includes a plurality of frame images. The frame rate of the video information may be, for example, 30 [FPS] or 30 [FPS], etc.

[0053] (Step S12) Next, the area to be measured is identified by the area identification unit 32 for each frame image included in the video information. The area to be measured may be the nose area, the right cheek area, the left cheek area, etc. It is preferable that the area to be measured is determined in advance. Also, depending on the imaging angle and the user's appearance, etc., the predetermined measurement area is not necessarily shown in the frame image, and analysis of the area may not be easy. Therefore, it is preferable that there are multiple areas to be measured. In this case, the subsequent processing may be performed based on any one of the multiple areas, or a combination of the multiple areas.

[0054] (Step S13) Next, the statistical calculation unit 33 performs statistical calculation on the RGB pixel values. The statistical calculation performed by the statistical calculation unit 33 may be, for example, calculation of an average value of pixel values ​​in the area identified by the area identification unit 32. The area on which the statistical calculation unit 33 performs calculation may be multiple. Furthermore, the statistical calculation unit 33 does not necessarily need to perform statistical calculation on all RGB pixel values, but may perform statistical calculation on at least any two pixel values ​​of the RGB pixel values.

[0055] (Step S14) Next, the difference calculation unit 34 calculates a difference value between any two statistical calculation values ​​(for example, average values) calculated by the statistical calculation unit 33. The difference value calculated by the difference calculation unit 34 may be, for example, a difference value between RG, a difference value between RB, or a difference value between GB. The difference value calculated by the difference calculation unit 34 may be a combination of these difference values, or a combination of these difference values ​​weighted by a predetermined weighting.

[0056] (Step S15) Next, Fourier calculation unit 35 converts the difference value calculated by difference calculation unit 34 into a frequency axis. Based on the result of frequency conversion by Fourier calculation unit 35, a pulse wave amplitude value and a heart rate can be calculated.

[0057] (Step S16) Finally, the output unit 36 ​​outputs the pulse wave amplitude value and the heart rate obtained in step S15. The destination of the output by the output unit 36 ​​may be, for example, the control unit 4 or a storage device (not shown) or the like.

[0058] 10 is a functional configuration diagram showing a modified example of the functional configuration of the arithmetic device according to this embodiment. With reference to the same figure, a arithmetic unit 3A which is a modified example of the arithmetic unit 3 will be described. In the description of the arithmetic unit 3A, the same components as those of the arithmetic unit 3 are denoted by the same reference numerals and description thereof may be omitted. The arithmetic unit 3A differs from the arithmetic unit 3 in that it further includes an inverse Fourier calculation unit 37.

[0059] The inverse Fourier calculation unit 37 performs an inverse Fourier transform based on the result of the transform performed by the Fourier calculation unit 35. A pulse wave can be obtained by performing the inverse Fourier transform by the inverse Fourier calculation unit 37. The output unit 36 ​​further outputs the pulse wave calculated by the inverse Fourier calculation unit 37 instead of or in addition to the pulse wave amplitude value and heart rate calculated by the Fourier calculation unit 35.

[0060] FIG. 11 is a block diagram showing an example of the internal configuration of the arithmetic device according to this embodiment. An example of the internal configuration of the arithmetic device will be described with reference to the same figure. At least some of the functions of the arithmetic device can be realized using a computer. As shown in the figure, the computer is configured to include a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, etc., and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902, etc. The central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations according to each instruction. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with an external input / output device, etc. The input / output devices 904 and 905 are input / output devices. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used inside the computer. For example, the central processing unit 901 reads and writes data in the RAM 902 via the bus 906. For example, the central processing unit 901 accesses the input / output port via the bus 906. In addition, all or part of the functional units of the arithmetic device according to this embodiment may be realized using hardware such as an ASIC, a PLD, or an FPGA. In addition, all or part of the functional units may be realized by a combination of software and hardware. The content information providing device 1 may also have an internal configuration as shown in the figure.

[0061] [Summary of this embodiment] According to the above-described embodiment, the arithmetic device according to this embodiment includes the image information acquisition unit 31, which acquires image information including a plurality of consecutive frame images in which a person to be measured is captured, the region identification unit 32, which identifies the region of the person captured in the frame images to be measured, the statistical calculation unit 33, which performs statistical calculation on at least two pixel values ​​of the RGB pixel values ​​of the plurality of consecutive frame images for the identified region, the difference calculation unit 34, which calculates the difference between at least two pixel values ​​for which statistical calculation has been performed, the Fourier calculation unit 35, which converts information on the difference value calculated for each time by the difference calculation unit 34 into a frequency axis, and the output unit 36, which outputs the pulse wave amplitude value and the heart rate of the person to be measured based on the result converted by the Fourier calculation unit 35. That is, the arithmetic device according to this embodiment detects biological information based on image information. Therefore, according to this embodiment, biological information can be detected without contact.

[0062] Also, according to this embodiment, the region identification unit 32 identifies a part of the face of the person captured in the frame image as the region to be measured. Here, the user of this embodiment often uses a content information providing device 1 equipped with an imaging unit 2 that images the user's face, such as a smartphone, a tablet terminal, or smart glasses. Therefore, according to this embodiment, by using the imaging unit 2 equipped in the content information providing device 1, it is possible to detect biometric information without providing new hardware used for biometric information detection. Therefore, according to this embodiment, it is possible to detect biometric information at low cost and without imposing a burden on the user.

[0063] According to this embodiment, the region specifying unit 32 specifies, as a region to be measured, a region having less fat than the surrounding area among a part of the face of a person captured in a frame image. Since the region specifying unit 32 specifies, as a region to be measured, a region having less fat than the surrounding area, the arithmetic device according to this embodiment can obtain changes in blood vessels present in the dermis and subcutaneous tissue with high accuracy.

[0064] Furthermore, according to this embodiment, the statistical calculation unit 33 performs statistical calculations on at least the R and G pixel values ​​among the pixel values ​​of each of the R, G and B pixels of a plurality of consecutive frame images, and the difference calculation unit 34 calculates the difference between the R and G pixel values ​​for which the statistical calculations have been performed. That is, according to this embodiment, biometric information is detected based on the R and G difference values. Here, referring to FIG. 8, among the R, G and B difference values, the difference value between R and G has the smallest error. Therefore, according to this embodiment, biometric information can be detected with higher accuracy.

[0065] Furthermore, according to this embodiment, the statistical calculation unit 33 calculates the average value of at least any two pixel values ​​of each of the RGB pixel values ​​of a plurality of consecutive frame images for the region identified by the region identification unit 32. That is, the statistical calculation performed by the statistical calculation unit 33 is the calculation of an average value. Therefore, according to this embodiment, biometric information can be easily detected.

[0066] Moreover, the calculation device according to this embodiment further includes an inverse Fourier calculation unit 37, which performs an inverse Fourier transform based on the result of the transformation by the Fourier calculation unit 35 to calculate a pulse wave. The output unit 36 ​​further outputs the pulse wave calculated by the inverse Fourier calculation unit 37. Therefore, according to this embodiment, the pulse wave can be detected in a non-contact manner.

[0067] In addition, all or part of the functions of each unit of the arithmetic device according to the above-mentioned embodiment may be realized by recording a program for realizing these functions on a computer-readable recording medium, reading the program recorded on the recording medium into a computer system, and executing it. Note that the "computer system" referred to here includes hardware such as an OS and peripheral devices.

[0068] In addition, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage units such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" may also include those that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and those that hold a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0069] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the spirit of the present invention. [Explanation of symbols]

[0070] 1...content information providing device, 2...imaging unit, 3...calculation unit, 4...control unit, 5...display unit, U...user, 31...video information acquisition unit, 32...area identification unit, 33...statistical calculation unit, 34...difference calculation unit, 35...Fourier calculation unit, 36...output unit, 37...inverse Fourier calculation unit

Claims

1. an image information acquisition unit that acquires image information including a plurality of consecutive frame images of a person to be measured; an area specifying unit that specifies an area to be measured among the areas of a person photographed in the frame image; a statistical calculation unit that performs statistical calculations on at least any two pixel values ​​of RGB of the plurality of consecutive frame images for the identified region; a difference calculation unit that calculates a difference between at least two pixel values ​​for which statistical calculation has been performed; a calculation unit for converting information on the difference value for each time calculated by the difference calculation unit into information on a frequency axis; an output unit that outputs a pulse wave amplitude value and a heart rate of the person being measured based on the results of the conversion by the calculation unit; A computing device comprising:

2. The area specifying unit specifies a part of a face of a person captured in the frame image as an area to be measured. The computing device of claim 1 .

3. The region specifying unit specifies, as a measurement target region, a region having less fat than the surrounding area among a part of a face of a person captured in the frame image. The computing device according to claim 2 .

4. the statistical calculation unit performs statistical calculations on at least R and G pixel values ​​among the R, G and B pixel values ​​of the plurality of consecutive frame images; The difference calculation unit calculates the difference between the R and G pixel values ​​on which the statistical calculation has been performed. The computing device of claim 1 .

5. The statistical calculation unit calculates an average value of at least any two pixel values ​​of RGB of the consecutive plurality of frame images for the specified region. The computing device of claim 1 .

6. An inverse Fourier calculation unit is further provided for calculating a pulse wave by performing an inverse Fourier transform based on the result of the conversion by the calculation unit, The output unit further outputs the pulse wave calculated by the inverse Fourier calculation unit. A computing device according to any one of claims 1 to 5.

7. an image information acquiring step of acquiring image information including a plurality of consecutive frame images of a person to be measured; an area specifying step of specifying an area to be measured among the areas of the person photographed in the frame image; a statistical calculation step of performing a statistical calculation on at least any two pixel values ​​of RGB of the plurality of consecutive frame images for the identified region; a difference calculation step of calculating a difference between at least two pixel values ​​for which the statistical calculation has been performed; a calculation step of converting information about the difference value for each time calculated by the difference calculation step into a frequency axis; an output step of outputting the pulse wave amplitude value and the heart rate of the person to be measured based on the results converted by the calculation step; The calculation method has the following steps.

8. On the computer, a video information acquiring step of acquiring video information including a plurality of consecutive frame images of a person to be measured; an area specifying step of specifying an area to be measured among the areas of the person photographed in the frame image; a statistical calculation step of performing statistical calculations on at least any two pixel values ​​of RGB of the plurality of consecutive frame images for the identified region; a difference calculation step of calculating a difference between at least two pixel values ​​for which the statistical calculation has been performed; a calculation step of converting information about the difference value for each time calculated in the difference calculation step into a frequency axis; an output step of outputting the pulse wave amplitude value and the heart rate of the person to be measured based on the results converted by the calculation step; A program that executes the following.

Images