Imaging device, information acquisition device, and imaging method

The imaging device allows for a simple switch to a high-resolution imaging mode for pulse wave detection, enhancing the capture and analysis of pulse wave information.

JP7590896B2Active Publication Date: 2024-11-27SHARP KK
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2021037363
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-03-09
Publication Date
2024-11-27
Estimated Expiration
2041-03-09

AI Technical Summary

Technical Problem

There is a demand for a simple method to switch camera settings to settings suitable for pulse wave detection.

Method used

An imaging device with an imaging unit that can switch between a first and second imaging mode, where the second mode captures moving images with higher resolution to detect pulse wave information, utilizing a changeover switch to facilitate this transition.

Benefits of technology

Enables the capture of high-resolution images for accurate pulse wave detection, improving user convenience and accuracy in acquiring pulse wave information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007590896000001
    Figure 0007590896000001
  • Figure 0007590896000002
    Figure 0007590896000002
  • Figure 0007590896000003
    Figure 0007590896000003
Patent Text Reader

Abstract

To detect a pulse wave highly accurately by detecting the pulse wave with high resolution.SOLUTION: An imaging device has: an imaging part having a resolution of 4K or higher, for imaging a moving image; and a change-over switch for switching an imaging mode for imaging the moving image by the imaging part from a first imaging mode to a second imaging mode. The second imaging mode is the imaging mode for imaging the moving image having a higher resolution, than the first imaging mode, for detecting a variation of pulse wave information for showing a pulse wave of an organism in the moving image.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] One aspect of the present disclosure relates to an imaging device, an information acquisition device, and an imaging method. [Background technology]

[0002] Patent Document 1 discloses a pulse wave detection device that detects pulse waves from a captured image. In the pulse wave detection device of Patent Document 1, the aperture of the camera is changed to adjust the amount of light from the camera so that the ratio of pixels with brightness values ​​below a predetermined value becomes equal to or greater than a threshold value, based on a histogram of pixel brightness values ​​created from the captured image. According to Patent Document 1, this makes it possible to improve brightness even if the amount of light from the camera capturing the image is insufficient. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-220915 A Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, there has been a demand for a simple method for switching the camera settings to settings suitable for detecting pulse waves. An object of one aspect of the present disclosure is to provide an imaging device, an information acquisition device, and an imaging method that are capable of switching the imaging mode to settings suitable for acquiring pulse wave information in a simple manner. [Means for solving the problem]

[0005] An imaging device according to one embodiment of the present disclosure has a resolution of 4K or more and includes an imaging unit that captures moving images, and a changeover switch for switching an imaging mode in which the imaging unit captures the moving images from a first imaging mode to a second imaging mode, wherein the second imaging mode is an imaging mode in which the moving images are captured with a higher resolution for detecting an amount of change in pulse wave information indicating the pulse waves of a living body shown in the moving images than the first imaging mode. [Brief description of the drawings]

[0006] [Figure 1] FIG. 2 is a front view of the mobile terminal according to the first embodiment. [Diagram 2] FIG. 2 is a functional block diagram of the mobile terminal according to the first embodiment. [Diagram 3] 3 is a diagram showing a first color gamut represented by first color gamut data stored in a storage unit according to the embodiment, and a second color gamut represented by second color gamut data. FIG. [Figure 4] FIG. 2 is a diagram illustrating a first dynamic range indicated by first dynamic range data stored in a storage unit according to the embodiment, and a second dynamic range indicated by second dynamic range data. [Diagram 5] 5 is a schematic diagram of a moving image created by an imaging control unit according to the embodiment. FIG. [Figure 6] 10 is a diagram illustrating a state in which a biological information acquisition unit according to an embodiment acquires pulse wave information that indicates a pulse wave captured in an image. FIG. [Figure 7] 4 is a diagram showing an example of a waveform of a pulse wave indicated by pulse wave information acquired by a biological information acquisition unit according to the embodiment; FIG. [Figure 8] 10 is a flowchart illustrating an example of an operation of a control unit according to the embodiment. [Figure 9] FIG. 13 is a diagram showing a result of comparing S / N ratios for each pixel resolution according to Experimental Example 1 of the embodiment. [Figure 10] FIG. 13 is a diagram showing the results of comparing the number of pixels having a predetermined luminance value or more for each pixel resolution in Experimental Example 1 of the embodiment. [Figure 11]13 is a diagram showing a comparison result of pulse wave information when various parameters of a camera that captures moving images are changed, in Experimental Example 2 according to the embodiment. FIG. [Figure 12] FIG. 11 is a diagram illustrating a comparison result of the S / N ratio for each parameter when various parameters are changed in a camera that captures moving images, in accordance with an experimental example 2 according to the embodiment. [Figure 13] FIG. 4 is a functional block diagram of a mobile terminal according to a first modified example of the embodiment. [Figure 14] 13 is a flowchart illustrating an example of an operation of a control unit in the first modified example of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] [Embodiment] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the drawings, the same or equivalent elements are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0008] Fig. 1 is a front view showing the configuration of a mobile terminal (biometric information acquisition device) 1 according to an embodiment. That is, Fig. 1 shows a plan view of the mobile terminal 1. The mobile terminal 1 is an example of a biometric information acquisition device capable of contactlessly acquiring pulse wave information that is information representing the pulse wave of a living body such as a human.

[0009] For example, the mobile terminal 1 includes an imaging device 40 capable of capturing a video or a series of still images showing the body surface of a living body. When the body surface of a living body is captured in the video or a series of still images captured by the imaging device 40, the mobile terminal 1 acquires pulse wave information that is information that represents the pulse wave of the captured body. For example, since the video or a series of still images showing the body surface show a change in color of the body surface based on the blood flow of the subject for each frame (or for each of the still images), the mobile terminal 1 acquires the pulse wave information of the subject based on the change in color of the body surface for each frame (or for each of the still images).

[0010] In this embodiment, the mobile terminal 1 is described as being a smartphone. For example, the mobile terminal 1 has a size and weight that allows a person to carry it around. For example, the mobile terminal 1 is small enough to be held in one hand. Note that the mobile terminal 1 is not limited to a smartphone, and may be, for example, a tablet, a notebook computer, or an electronic device such as a digital camera that can capture a video or a series of still images.

[0011] The mobile terminal 1 includes, for example, a touch panel unit 2, a housing 5, an imaging device 40 including an imaging unit 6, a speaker 7, and a microphone 8. For example, in the mobile terminal 1, the touch panel unit 2, the imaging device 40 including the imaging unit 6, the speaker 7, and the microphone 8 are provided in, for example, the same housing 5.

[0012] The touch panel unit 2 is a display that displays moving images and still images, and also an input device that receives input from a user such as a subject. The touch panel unit 2 has a display unit 3 and an input unit 4 that is provided on top of the display unit 3.

[0013] The display unit 3 is a display for displaying images such as moving images and still images. The display unit 3 has a display area for displaying images such as moving images and still images. The display unit 3 may be any display capable of displaying images such as moving images and still images, such as a liquid crystal display or an organic EL (electro-luminescence) display.

[0014] For example, a changeover switch 10 is displayed as an image in the display area of ​​the display unit 3 (in other words, the display area of ​​the touch panel unit 2). The changeover switch 10 is a switch for switching the imaging mode of the imaging device 40 from a first imaging mode, which is a normal imaging mode for capturing high-quality images, to a second imaging mode, which is an imaging mode for capturing images suitable for acquiring pulse waves. By providing the changeover switch 10, a user can switch the imaging mode of the imaging device 40 from the first imaging mode to the second imaging mode simply by selecting the changeover switch 10. In this way, the imaging device 40 can switch the imaging mode to a setting suitable for acquiring pulse wave information in a simple manner.

[0015] In this embodiment, the changeover switch 10 will be described as a soft key displayed as an image in the display area of ​​the touch panel unit 2. The changeover switch 10, which is a soft key, accepts contact with a finger or a touch pen as an input and outputs an input signal. However, the changeover switch 10 is not limited to a soft key, and may be, for example, a hardware key provided in an area other than the display area of ​​the touch panel unit 2 (for example, the housing 5) and configured with a hardware structure. If the changeover switch 10 is a hardware key, for example, when it is pressed or slid, it accepts it as an input and outputs an input signal.

[0016] The input unit 4 is an input device that receives touch from a finger or a touch pen as an input from a user, and generates and outputs an input signal based on the received input. The input unit 4 is, for example, a transparent touch sensor, and is provided so as to overlap the display area of ​​the display unit 3. That is, for example, the display unit 3 and the input unit 4 are integrated to form the touch panel unit 2. Note that the input unit 4 is not limited to a touch sensor, and may be, for example, a switch having a hardware structure that receives input from a user when pressed by the user.

[0017] The speaker 7 is a sound output device that generates sound based on a sound signal. The microphone 8 is a sound input device that receives external sound from the mobile terminal 1, such as the user's voice, as an input and converts it into a sound signal.

[0018] The imaging device 40 is an RGB camera capable of capturing a moving image or a series of still images showing the body surface of the subject from which pulse wave information is to be acquired. The imaging device 40 captures a moving image at a predetermined frame rate (e.g., 10 to 300 fps (frames per second)). Alternatively, the imaging device 40 captures a series of still images at a predetermined imaging interval. The imaging device 40 is, for example, an RGB camera having color channels that capture the luminance values ​​(also referred to as pixel values) of R (red), G (green), and B (blue).

[0019] The imaging device 40 is preferably a camera capable of capturing a moving image having a resolution of 4K (2160 pixels x 3840 pixels = 8294400 pixels) or more or a plurality of continuous still images. For example, the imaging device 40 may be a camera capable of capturing a moving image having a resolution of DCI 4K (2160 pixels x 4096 pixels = 8847360 pixels). For example, the imaging device 40 may be a camera capable of capturing a moving image having a resolution of 8K (4320 pixels x 7680 pixels = 33177600 pixels) or a plurality of continuous still images. The imaging device 40 may be a camera capable of capturing a moving image having a resolution less than 4K, such as full high-definition (1920 pixels x 1080 pixels = 2073600 pixels), or a plurality of continuous still images.

[0020] The colors acquired by the imaging device 40 are not limited to RGB, and may be other colors such as cyan, magenta, and orange. The light from which the imaging device 40 acquires luminance values ​​is not limited to visible light, and may be invisible light such as infrared light. The number of color channels acquired by the imaging device 40 is not limited to three, and may be one.

[0021] The imaging unit 6 is a sensor device including a light receiving unit of a camera in the imaging device 40. The light receiving unit of the imaging unit 6 is covered with a transparent cover exposed from the mobile terminal 1. The transparent cover covering the light receiving unit of the imaging unit 6 may be provided so as to be exposed on the front surface of the mobile terminal 1 (the surface on the side where an image is displayed on the touch panel unit 2) or on the opposite back surface.

[0022] For example, the imaging unit 6 may use a COMS (Complementary MOS) sensor device. The imaging unit 6 is preferably a sensor device having a pixel count of 4K (2160 pixels x 3840 pixels = 8294400 pixels) or more. For example, the imaging unit 6 may be a sensor device having a pixel count of DCI 4K (2160 pixels x 4096 pixels = 8847360 pixels). For example, the imaging unit 6 may be a sensor device having a pixel count of 8K (4320 pixels x 7680 pixels = 33177600 pixels). Note that the imaging unit 6 may be a sensor device having a pixel count of less than 4K, such as full high-definition (1920 pixels x 1080 pixels = 2073600 pixels).

[0023] Fig. 2 is a functional block diagram of a mobile terminal 1 according to an embodiment. As shown in Fig. 2, the mobile terminal 1 has a control unit 20 and a storage unit 30 in addition to a touch panel unit 2, an imaging unit 6, a speaker 7, a microphone 8, and a changeover switch 10. The control unit 20 has, for example, an imaging control unit 21, a compression unit 22, and a biometric information acquisition unit 23. Note that the compression unit 22 may be omitted when the moving image or multiple consecutive still images created by the imaging control unit 21 is not compressed.

[0024] The touch panel unit 2, the imaging unit 6, the speaker 7, the microphone 8, the changeover switch 10, the storage unit 30, and the control unit 20a including the imaging control unit 21 and the compression unit 22 constitute an imaging device 40.

[0025] For example, the information acquiring device according to the present embodiment may include an imaging device 40, a control unit 20b including a biometric information acquiring unit 23, and an information terminal having a storage unit (not shown). In this case, the imaging device 40 may be configured as an independent device, and the information terminal may be configured as an independent device separated from the imaging device 40. The information terminal is, for example, a personal computer.

[0026] The control unit 20 is configured with a processor such as a CPU (Central Processing Unit), and comprehensively controls various functions of the information terminal 1. When the control unit 20 is configured with the control unit 20a and the control unit 20b, each of the control unit 20a and the control unit 20b is configured with a processor such as a CPU.

[0027] The storage unit 30 can be realized by, for example, a hard disk, a solid state drive (SSD), or a semiconductor memory. The storage unit 30 may be a storage medium fixed inside the mobile terminal 1 as a part of the mobile terminal 1, or may be a storage medium removable from the mobile terminal 1 such as a SIM card. The storage unit 30 stores various parameter data, an imaging program, an information acquisition program, and the like used when creating images such as a moving image or a series of multiple still images, which are pre-installed or installed from a server capable of communicating with the mobile terminal 1. Examples of the various parameter data include color gamut data D1 and dynamic range data D2.

[0028] The color gamut data D1 is data for setting the color gamut of the image to be created, and includes first color gamut data D1a, which is a relatively wide color gamut, and second color gamut data D1b, which is a relatively narrower color gamut than the first color gamut data D1a. The dynamic range data D2 is data for setting the dynamic range of the image to be created, and includes first dynamic range data D2a, which is a relatively wide dynamic range, and second dynamic range data D2b, which is a relatively narrower dynamic range than the first dynamic range data D2a.

[0029] The imaging program causes the control unit 20, which is a computer, to function as an imaging control unit 21 and a compression unit 22. The information acquisition program causes the control unit 20, which is a computer, to function as a biological information acquisition unit 23.

[0030] The imaging control unit 21 captures a moving image or a series of multiple still images together with the imaging unit 6. For example, when the imaging control unit 21 acquires an input signal from the input unit 4, the imaging control unit 21 drives the imaging unit 6 by outputting a drive signal to the imaging unit 6 based on the input signal. Then, when the imaging control unit 21 acquires an electric signal photoelectrically converted from the imaging unit 6, the imaging control unit 21 creates a moving image or a series of multiple still images based on various parameter data stored in the storage unit 30.

[0031] The imaging control unit 21 switches, in response to an input signal from the input unit 4, an imaging mode for capturing a moving image or a series of still images using the imaging unit 6 between a first imaging mode and a second imaging mode.

[0032] The first imaging mode is an imaging mode for capturing a moving image or a plurality of continuous still images with relatively higher image quality than the second imaging mode. The second imaging mode is an imaging mode for capturing a moving image or a plurality of continuous still images with relatively higher resolution for detecting the amount of change in pulse wave information indicating the pulse wave of a living body captured in the moving image or the plurality of continuous still images than the first imaging mode.

[0033] That is, the first imaging mode is an imaging mode for capturing normal moving images or still images, not limited to moving images or still images for the purpose of acquiring pulse wave information of a living body, and the second imaging mode is an imaging mode for capturing moving images or a series of multiple still images that can acquire pulse wave information of a living body with high accuracy.

[0034] Specifically, the imaging control unit 21 acquires an imaging instruction signal, which is an input signal that instructs the start of imaging of a moving image or a series of still images, from the input unit 4, and starts imaging in the first imaging mode when it does not acquire a switching signal, which is an input signal indicating that the changeover switch 10 is selected. Imaging in the first imaging mode means, for example, that the imaging control unit 21 drives the imaging unit 6 when it acquires an imaging instruction signal. Then, when the imaging control unit 21 acquires a photoelectrically converted electrical signal from the imaging unit 6, it refers to the storage unit 30 and selects the first color gamut data D1a and the first dynamic range data D2a that can create a moving image or a still image with higher image quality from the first color gamut data D1a and the second color gamut data D2b in the color gamut data D1 and from the first dynamic range data D2a and the second dynamic range data D2b in the dynamic range data D2, and creates a moving image or a still image.

[0035] Moreover, the imaging control unit 21 acquires an imaging instruction signal, which is an input signal that instructs the start of imaging of a moving image or a plurality of continuous still images, from the input unit 4, and when it acquires a switching signal, which is an input signal indicating that the changeover switch 10 has been selected, it switches from the first imaging mode to the second imaging mode and starts imaging in the second imaging mode. Imaging in the second imaging mode means, for example, that the imaging control unit 21 drives the imaging unit 6 when it acquires the imaging instruction signal and the switching signal. Then, when the imaging control unit 21 acquires the photoelectrically converted electric signal from the imaging unit 6, it refers to the storage unit 30, and selects the second color gamut data D1b and the second dynamic range data D2b from the first color gamut data D1a and the second color gamut data D2b in the color gamut data D1, and from the first dynamic range data D2a and the second dynamic range data D2b in the dynamic range data D2, which are capable of creating a moving image or a plurality of continuous still images with a higher resolution for detecting the amount of change in pulse wave information indicating the pulse wave of a living body, and creates a moving image or a still image.

[0036] In other words, if the relationship between the amount of light input to the imaging unit 6 and the luminance value of each pixel of the imaging unit 6 is defined as the input / output characteristics, when imaging in the second imaging mode, the imaging control unit 21 creates a moving image or still image by narrowing the input / output characteristics in both color gamut and dynamic range compared to when imaging in the first imaging mode.

[0037] Fig. 3 is a diagram showing a first color gamut D1A (shown by a dashed line in Fig. 3) represented by first color gamut data D1a stored in a storage unit 30 according to an embodiment, and a second color gamut D1B (shown by a dashed line in Fig. 3) represented by second color gamut data D1b. In Fig. 3, the first color gamut D1A and the second color gamut D1B are expressed using xy color space coordinates.

[0038] The first color gamut D1A is wider than the second color gamut D1B. For example, the first color gamut D1A is wide enough to cover almost BT.2020 and is wider than BT.709. As a result, when capturing an image in the first capturing mode, the imaging control unit 21 can create a high-quality video or still image with a wide color reproduction range and vivid colors expressed in the first color gamut D1A by creating a video or still image using the first color gamut data D1a from an electrical signal photoelectrically converted from the imaging unit 6 having a pixel count corresponding to 4K. As a result, the imaging control unit 21 can create a video or still image of higher quality in the first capturing mode than in the second capturing mode. Note that the first color gamut D1A only needs to be wider than the second color gamut D1B, and is not limited to the range of the above-mentioned color gamut.

[0039] The second color gamut D1B is narrower than the first color gamut D1A, and has a color gamut width equivalent to that of BT.709, for example. In other words, the second color gamut D1B is a color gamut narrowed to a color gamut in which the color of the body surface changes more than the first color gamut D1A. As a result, for example, when capturing an image in the second imaging mode, the imaging control unit 21 can create a moving image or a continuous plurality of still images expressed in the second color gamut D1B narrower than the first color gamut D1A and narrowed to a color gamut in which the color of the body surface changes more by creating a moving image or a continuous plurality of still images using the second color gamut data D1b based on an electrical signal photoelectrically converted from the imaging unit 6 having a pixel count corresponding to 4K.

[0040] The second color gamut D1B is not limited to the above-mentioned range of color gamut, as long as it is narrower than the first color gamut D1A and is limited to a color gamut in which the color of the body surface changes.

[0041] In this way, even in the second imaging mode, the imaging control unit 21 obtains an electrical signal from the imaging unit 6 that uses multiple pixels of the same high resolution (e.g., multiple pixels with a number of pixels equivalent to 4K) as the multiple pixels of the same high resolution used in the first imaging mode (e.g., multiple pixels with a number of pixels equivalent to 4K).

[0042] In the second imaging mode, the imaging control unit 21 creates a moving image or a series of still images expressed in a second color gamut D1B, which is narrower than the first color gamut D1A in the first imaging mode and is narrowed to approach the color gamut in which the color of the body surface of the living body (for example, the color of the skin) changes, from the electrical signal obtained from the imaging unit 6. That is, in the second imaging mode, the imaging control unit 21 creates a moving image or a series of still images expressing finer changes in the color of the body surface of the living body than in the first imaging mode.

[0043] As a result, the imaging control unit 21 can create a moving image or a series of still images in the second imaging mode that has a higher resolution for detecting the amount of change in pulse wave information showing the pulse wave captured in the moving image or the series of still images than the moving image or still images represented by the first color gamut D1A. This makes it possible to obtain a moving image or series of still images from which pulse wave information that accurately represents the actual pulse wave can be obtained. As a result, the mobile terminal 1 can obtain pulse wave information that accurately represents the pulse wave of a living body.

[0044] 4 is a diagram illustrating a first dynamic range D2A (shown as D2A in FIG. 4) indicated by the first dynamic range data D2a stored in the storage unit 30 according to the embodiment, and a second dynamic range D2B (shown as D2B in FIG. 4) indicated by the second dynamic range data D2b. The luminance shown in FIG. 4 represents the range of luminance of light incident on the imaging unit 6.

[0045] As described above, the first dynamic range D2A represented by the first dynamic range data D2a is wider than the second dynamic range D2B represented by the second dynamic range data D2b.

[0046] For example, but not limited to, the first dynamic range D2A is assumed to be approximately the same as a high dynamic range (HDR), and for example, but not limited to, the second dynamic range D2B is assumed to be approximately the same as a standard dynamic range (SDR), which is narrower than HDR.

[0047] For example, when capturing an image in the first imaging mode, the imaging control unit 21 can create a moving image or still image with a wide luminance range and natural brightness expressed in the first dynamic range D2A by creating a moving image or still image using the first dynamic range data D2a from the photoelectrically converted electrical signal from the imaging unit 6 having a pixel count corresponding to 4K, for example. This allows the imaging control unit 21 to create a high-quality moving image or still image with a wider dynamic range in the first imaging mode than in the second imaging mode.

[0048] The second dynamic range D2B is a dynamic range narrower than the first dynamic range D2A. In other words, the second dynamic range D2B expresses luminance in a narrower range than the first dynamic range D2A with the same amount of data as the first dynamic range D2A. For example, the first dynamic range D2A and the second dynamic range D2B are both expressed with n bits from X1 bits to Xn bits. Although the second dynamic range D2B has the same number of bits as the first dynamic range D2A, the second dynamic range D2B intentionally narrows the range of luminance that can be expressed compared to the first dynamic range D2A.

[0049] In this way, in the second imaging mode, the imaging control unit 21 creates a moving image or a series of still images in which the narrower range of luminance changes are displayed in detail from the electrical signal obtained from the imaging unit 6 by deliberately narrowing the range of luminance displayed compared to the first dynamic range D2A in the first imaging mode. In this way, the moving image or the series of still images in which the luminance changes are displayed in detail also shows the fine color changes of the body surface (for example, changes in skin color) that indicate the captured pulse wave between frames of the moving image or between the series of still images.

[0050] In this way, the imaging control unit 21 can create a moving image or a series of still images in the second imaging mode that has a higher resolution for detecting the amount of change in pulse wave information showing the pulse wave captured in the moving image or a series of still images than the moving image or still images expressed by the first dynamic range D2A. This makes it possible to obtain a moving image or a series of still images from which pulse wave information that accurately represents the actual pulse wave can be obtained. As a result, the mobile terminal 1 can obtain the actual pulse wave with high accuracy.

[0051] In particular, when the imaging control unit 21 has a small number of gradation values ​​expressing pixel values ​​when creating a video or multiple still images, or when the compression unit 22 compresses a video or multiple consecutive still images, it is preferable that the luminance range other than the luminance range required to acquire color changes on the surface of the living body is as small as possible.

[0052] Therefore, when the number of gradation values ​​expressing pixel values ​​when the imaging control unit 21 creates a moving image or multiple still images is small, or when the compression unit 22 compresses a moving image or multiple consecutive still images, it is particularly effective to obtain highly accurate pulse wave information by expressing the moving image or multiple still images using a dynamic range narrower than the first dynamic range D2A, such as the second dynamic range D2B.

[0053] 5 is a schematic diagram of a moving image created by the imaging control unit 21 according to the embodiment. For example, the moving image created by the imaging control unit 21 includes a plurality of consecutive frames F1, F2, F3, F4, . . . frame Fm.

[0054] The moving image or a series of still images created by the imaging control unit 21 may be stored in the storage unit 30 without being compressed, and then the pulse wave information may be acquired by the biological information acquisition unit 23. Alternatively, the moving image created by the imaging control unit 21 may be compressed and then stored in the storage unit 30, and then the compressed moving image may be decompressed and the pulse wave information may be acquired by the biological information acquisition unit 23. For example, in the case of a high-resolution moving image such as 4K or 8K, the amount of information is large, so it is preferable that the moving image created by the imaging control unit 21 is compressed once and then stored in the storage unit 30.

[0055] When the imaging control unit 21 creates a moving image or a series of still images, the compression unit 22 compresses the created moving image or a series of still images and stores them in the storage unit 30. The compression method used by the compression unit 22 is preferably lossless compression, but lossy compression may also be used to further reduce the amount of information.

[0056] If the image to be compressed is an 8K video, the bit rate is preferably 200 Mbps or more. If the image to be compressed is a 4K video, the bit rate is preferably 50 Mbps or more.

[0057] Furthermore, it is preferable that the compression unit 22 compresses each of the multiple frames F1, F2, Fm included in the moving image captured by the imaging unit 6 and the imaging control unit 21, or each of the multiple consecutive still images. That is, when compressing a moving image, it is preferable that the compression unit 22 compresses only one frame (for example, compresses only frame F1, compresses only frame F2, etc.) rather than compressing across frames. Also, when compressing multiple consecutive still images, it is preferable that the compression unit 22 compresses only one still image rather than compressing across multiple still images.

[0058] This is because, as will be described later, pulse wave information is obtained based on changes in the body surface between frames (or between multiple still images), and therefore more accurate pulse wave information can be obtained by compressing only one frame (one still image) rather than compressing multiple frames (or multiple still images) together.

[0059] Although the compression unit 22 has been described as compressing a moving image or a series of consecutive still images before the biometric information acquisition unit 23 acquires pulse wave information, the present invention is not limited to this, and the timing at which the compression unit 22 compresses the moving image or a series of consecutive still images can be changed as desired. For example, the compression unit 22 may compress a moving image or a series of consecutive still images after the biometric information acquisition unit 23 acquires pulse wave information and store them in the storage unit 30, or may compress a moving image or a series of consecutive still images while the biometric information acquisition unit 23 is acquiring pulse wave information (for example, after detecting feature points such as a face) and store them in the storage unit 30.

[0060] FIG. 6 is a diagram illustrating a state in which the biological information acquiring unit 23 according to the embodiment is acquiring pulse wave information that indicates a pulse wave captured in an image 50. As shown in FIG.

[0061] The biometric information acquiring unit 23 reads in the moving image or a series of still images stored in the storage unit 30 and performs image processing to acquire pulse wave information indicating the pulse wave of the subject. If the moving image or a series of still images stored in the storage unit 30 is compressed, the biometric information acquiring unit 23 decompresses the compressed image and performs image processing to acquire pulse wave information indicating the pulse wave of the subject. The moving image or a series of still images is referred to as an image 50.

[0062] For example, the biometric information acquiring unit 23 identifies a partial area (e.g., a partial area such as the forehead, cheek, or chin) of the body surface of the living body shown in the image 50, and extracts a change in color of the body surface of the living body for each frame (or for each of a plurality of still images) in the identified partial area. Then, the biometric information acquiring unit 23 acquires pulse wave information indicating a pulse wave based on the extracted change in color of the body surface.

[0063] 6, for example, the faces of two people are shown in image 50. When multiple people are shown in image 50, biological information acquiring unit 23 may acquire pulse wave information of one person, or may acquire pulse wave information of each of the multiple people.

[0064] For example, the biometric information acquiring unit 23 identifies a predetermined feature point (e.g., nose) of the first face U1 shown in the image 50, and identifies an area A11 in the image 50 that includes the identified feature point (e.g., nose). Then, the biometric information acquiring unit 23 identifies an area of ​​interest A12 (e.g., forehead) from which pulse wave information can be easily acquired, which is a predetermined distance away in the image 50 from the area A11 that includes the identified feature point (e.g., nose). The area of ​​interest A12 is not limited to the forehead, and may be set arbitrarily, for example, to the entire face or the cheek.

[0065] Similarly to the first face U1, the biometric information acquiring unit 23 also identifies a predetermined feature point (e.g., nose) of the second face U2 appearing in the image 50, and identifies an area A21 in the image 50 that includes the identified feature point (e.g., nose).Then, it identifies an area of ​​interest A22 (e.g., forehead) that is a predetermined distance away in the image 50 from the area A21 that includes the identified feature point (e.g., nose) and from which pulse wave information can be easily acquired.

[0066] Then, the biometric information acquisition unit 23 acquires pulse wave information indicating the pulse wave based on the change in color of the body surface of each of the areas of interest A12-A22 for each frame if the image 50 is a moving image (or the change in color of the body surface of each of the areas of interest A12-A22 for each of the still images if the image 50 is a series of multiple still images).

[0067] Specifically, for example, the biometric information acquisition unit 23 calculates a representative value of a plurality of pixel values ​​included in the region of interest A12. For example, the representative value of the plurality of pixel values ​​of the region of interest A12 may be a statistical quantity such as an average value, a median value, or a mode value of each of the plurality of pixels included in the region of interest A12. The biometric information acquisition unit 23 calculates a representative value of the region of interest A12 for each color channel (for example, for each of R (red), G (green), and B (blue)). Then, the biometric information acquisition unit 23 calculates pulse wave information indicating a pulse wave using the calculated representative value for each color channel and various methods such as principal component analysis, independent component analysis, or pigment component separation. The biometric information acquisition unit 23 calculates pulse wave information for the region of interest A22 in the same manner. In this way, the biometric information acquisition unit 23 acquires biometric information from a moving image or a plurality of consecutive still images.

[0068] The living body is not limited to a human, and may be a non-human living body. The part from which the pulse wave information is obtained may be a part other than the face. The part from which the pulse wave information is obtained may be, for example, a part other than the face, such as the palm of the hand or the neck, as long as the body surface is exposed and captured.

[0069] 7 is a diagram showing an example of a waveform of a pulse wave indicated by pulse wave information acquired by the biological information acquiring unit 23 according to the embodiment. After that, the biological information acquiring unit 23 may acquire various biological information from the pulse wave indicated by the acquired pulse wave information by measuring, for example, a blood pressure value, a pulse rate, a stress level, a heart rate, and information indicating a health state of blood vessels.

[0070] The biometric information acquiring unit 23 may also detect respiration based on a time-dependent change in luminance value due to the body movement of a living body captured in a video or a series of still images. For example, the biometric information acquiring unit 23 may acquire the respiratory rate, exhalation, and inhalation by acquiring the movement of the shoulders or chest accompanying respiration.

[0071] The biological information such as the pulse wave information acquired by the biological information acquiring unit 23 may be displayed on the display unit 3 or another display. In addition, the biological information such as the pulse wave information acquired by the biological information acquiring unit 23 may be stored in a storage device other than the mobile terminal 1, such as a server.

[0072] In addition, when obtaining information that can be obtained regardless of the body part, such as heart rate, stress level, and respiratory rate, from the pulse wave indicated by the acquired pulse wave information, as described using Figure 6, it is expected that the accuracy of information detection will be improved by identifying multiple areas (multiple body parts) for each person shown in the image and identifying areas (multiple body parts) with high pixel values ​​from among the multiple areas (multiple body parts) and setting them as the areas of interest A12-A22, respectively.

[0073] Furthermore, for example, the biometric information acquisition unit 23 may detect a feature point (e.g., a nose) in one frame (one still image) among multiple frames (or multiple consecutive still images) in a video, track the coordinates of the detected feature point in the other multiple frames (or multiple still images), and set the regions of interest A12-A22 based on the tracked coordinates.

[0074] Also, for example, the biometric information acquisition unit 23 may detect a feature point (e.g., a nose) in one frame (one still image) out of multiple frames (or multiple consecutive still images) and set the region of interest A12-A22, and in other multiple frames (or multiple still images), set the region of interest A12-A22 to the same coordinates as the already set region of interest A12-A22.

[0075] In this way, the imaging unit 6 and imaging control unit 21 of the imaging device 40 capture, for example, a moving image or a series of multiple still images having a resolution of 4K or more. That is, the imaging unit 6 has a number of pixels equivalent to 4K. If the imaging device 40 has a number of pixels equivalent to 4K or more, it can capture a moving image or a series of multiple still images with high resolution due to the large number of pixels.

[0076] Therefore, biological information acquisition section 23 can acquire pulse wave information based on high-resolution regions of interest A12-A22 imaged using a large number of pixels, making it possible to acquire highly accurate pulse wave information.

[0077] Furthermore, since the imaging device 40 has a larger pixel count than that of pixels less than 4K, for example, if the pixel size is the same as that of pixels less than 4K, it can capture a moving image or a series of still images with a high S / N (signal / noise) ratio. Therefore, the biological information acquisition unit 23 can acquire highly accurate pulse wave information from the regions of interest A12-A22 with a high S / N (signal / noise) ratio.

[0078] Furthermore, even when there is little ambient light when capturing a moving image or a series of still images, imaging device 40 can capture a moving image or a series of still images with a high S / N ratio captured with a larger number of pixels than an imaging device having a pixel count equivalent to less than 4K. Therefore, biometric information acquisition unit 23 can acquire highly accurate pulse wave information based on regions of interest A12-A22 with a high S / N ratio, even when the moving image or a series of still images is captured in a dark environment with little ambient light.

[0079] Furthermore, since imaging device 40 can capture high-resolution video or multiple consecutive still images, even if multiple people are captured, the pulse wave information of each of the captured people can be obtained with high accuracy in a single capture. Thus, when obtaining pulse wave information of multiple people, imaging device 40 can obtain the pulse wave information efficiently.

[0080] Furthermore, because mobile terminal 1 can capture high-resolution video or a series of still images, it can accurately acquire pulse wave information about a small object captured at a relatively large distance from imaging device 40. Therefore, mobile terminal 1 can increase the distance at which pulse wave information can be acquired, improving convenience.

[0081] Fig. 8 is a flowchart showing an example of the operation of the control unit 20 according to the embodiment. As shown in Fig. 8, in step S11, the imaging control unit 21 drives the imaging unit 6 based on an imaging instruction signal, which is an input signal from the input unit 4 based on a user's operation, to start capturing a moving image or a series of still images.

[0082] Then, in step S12, the imaging control unit 21 judges whether the changeover switch 10 has been selected. Specifically, for example, when the imaging control unit 21 does not receive a switching signal, which is an input signal indicating that the changeover switch 10 has been selected, from the input unit 4, the imaging control unit 21 judges that the changeover switch 10 has not been selected (NO in step S12), and captures a moving image or a plurality of consecutive still images in the first imaging mode in step S13. Then, when the imaging control unit 21 receives an input signal from the input unit 4 indicating that imaging is to be ended, the imaging control unit 21 ends imaging in the first imaging mode.

[0083] Furthermore, when the imaging control unit 21 receives a switching signal, which is an input signal indicating that the changeover switch 10 has been selected, from the input unit 4, it determines that the changeover switch 10 has been selected (YES in step S12), and in step S14 switches to the second imaging mode to capture a moving image or a plurality of consecutive still images, and creates a moving image or a plurality of consecutive still images. Then, in step S15, the compression unit 22 compresses the moving image or the plurality of consecutive still images created by the imaging control unit 21 as necessary, and stores the compressed moving image or the plurality of consecutive still images in the storage unit 30.

[0084] Next, in step S16, the biometric information acquisition unit 23 acquires the video or multiple consecutive still images stored in the memory unit 30, and if they are compressed, it decompresses them and then acquires pulse wave information indicating the pulse waves captured in the video or multiple consecutive still images.

[0085] Then, when the imaging control unit 21 receives an input signal from the input unit 4 instructing to end imaging, it ends imaging in the second imaging mode.

[0086] Thus, the imaging device 40 has an imaging section 6 and an imaging control section 21. The imaging section 6 and the imaging control section 21 capture a moving image or a series of multiple still images having a resolution of 4K or higher. This makes it possible to capture a moving image or a series of multiple still images with a high resolution and a high S / N ratio. Therefore, the imaging section 6 and the imaging control section 21 can capture a moving image or a series of multiple still images from which highly accurate pulse wave information can be obtained.

[0087] The imaging device 40 also has a changeover switch 10 for switching the imaging mode in which the imaging unit 6 captures a moving image or a plurality of continuous still images from a first imaging mode to a second imaging mode. This allows the user to switch the imaging mode of the imaging device 40 from the first imaging mode to the second imaging mode in which the imaging unit 6 captures a moving image or a plurality of continuous still images with a high resolution for detecting the amount of change in pulse wave information indicating the pulse wave of a living body captured in the moving image or the plurality of continuous still images, simply by selecting the changeover switch 10. In this way, the imaging device 40 allows the imaging mode to be switched to a setting suitable for acquiring pulse wave information in a simple manner. This improves the convenience of the user.

[0088] Also, the moving image or the continuous still images captured in the second imaging mode are created based on a narrower color gamut than the moving image or the continuous still images captured in the first imaging mode. That is, as described with reference to FIG. 3 and the like, the imaging control unit 21 expresses the captured moving image or the continuous still images in the second imaging mode using a narrower second color gamut D1B than the first color gamut D1A in the first imaging mode. This makes it possible to capture a moving image or a continuous still image with a higher resolution for detecting the amount of change in pulse wave information indicating the pulse wave of a living body than in the first imaging mode. This makes it possible to create a moving image or a continuous still image from which pulse wave information can be obtained with high accuracy.

[0089] Moreover, the moving image or the continuous still images captured in the second imaging mode are created based on a narrower dynamic range than the moving image or the continuous still images captured in the first imaging mode. That is, as described with reference to FIG. 4 and the like, the imaging control unit 21 expresses the captured moving image or the continuous still images in the second imaging mode using a narrower second dynamic range D2B than the first dynamic range D2A in the first imaging mode. This makes it possible to capture a moving image or a continuous still image with a higher resolution for detecting the amount of change in pulse wave information indicating the pulse wave of a living body than in the first imaging mode. This makes it possible to create a moving image or a continuous still image from which pulse wave information can be obtained with high accuracy.

[0090] Moreover, the mobile terminal 1 includes an imaging device 40 and a biological information acquiring section 23. This allows the biological information acquiring section 23 to acquire highly accurate pulse wave information from a moving image or a series of still images captured by the imaging device 40.

[0091] Moreover, the mobile terminal 1 has a compression unit 22. The compression unit 22 compresses each of a plurality of frames included in a moving image or each of a plurality of consecutive still images captured by the imaging unit 6 and the imaging control unit 21. This enables the biological information acquisition unit 23 to acquire highly accurate pulse wave information based on the decompressed moving image or the plurality of consecutive still images.

[0092] 9 is a diagram showing a result of comparing S / N ratios for each pixel resolution in Experimental Example 1 of the embodiment. In Experimental Example 1, a video of a person standing at a point 5 meters away from the camera was captured using a camera with a number of pixels equivalent to 8K. A video equivalent to 4K and a video equivalent to 2K were created by extracting some pixels from the video equivalent to 8K in a thinned manner. Pulse wave information was obtained from the change in color of the person's face in each of the original video equivalent to 8K and the created video equivalent to 4K and video equivalent to 2K, and the S / N of each pulse wave information was calculated.

[0093] As shown in FIG. 9, the S / N ratio of the pulse wave information obtained from a video equivalent to 4K is higher than the S / N ratio of the pulse wave information obtained from a video equivalent to 2K, and the S / N ratio of the pulse wave information obtained from a video equivalent to 8K is even higher.

[0094] 10 is a diagram showing a result of comparing the number of pixels having a predetermined luminance value or more for each pixel resolution in Experimental Example 1 of the embodiment. The number of pixels having a predetermined luminance value or more among pixels constituting the facial area of ​​a person captured in the original video equivalent to 8K, the created video equivalent to 4K, and the created video equivalent to 2K was compared.

[0095] As shown in Figure 10, the number of pixels above a certain brightness value in a video equivalent to 4K is greater than the number of pixels above a certain brightness value in a video equivalent to 2K, and furthermore, the number of pixels above a certain brightness value in a video equivalent to 8K is greater.

[0096] It can be seen that the number of pixels above a certain brightness value is approximately four times as high when going from 2K to 4K, and is again approximately four times as high when going from 4K to 8K. In addition, for example, when the camera resolution goes from 2K to 4K, it is expected that pulse wave information with the same accuracy can be obtained when the distance between the camera and the subject is doubled.

[0097] 11 is a diagram showing a comparison result of pulse wave information when various parameters are changed in a camera that captures a moving image, in Experimental Example 2 according to the embodiment. In Experimental Example 2, various parameters were changed using a camera with a pixel count equivalent to 8K to capture a moving image showing a person standing 50 cm from the camera.

[0098] FIG. 12 is a diagram illustrating a comparison result of the S / N ratio for each parameter when various parameters are changed in a camera that captures moving images, in accordance with an experimental example 2 according to the embodiment.

[0099] In Fig. 11, SDR (Standard Dynamic Range), HLG (Hybrid Log Gamma), and Log represent dynamic range. Log gamma is capable of reproducing fine changes in a small pixel value range (dark environment). It can be seen that SDR represents the pulse wave waveform more accurately than Log gamma. Also, when comparing BT.709 and BT.2020, which represent the color gamut, it can be seen that BT.709 represents the pulse wave waveform more accurately than BT.2020.

[0100] As shown in FIG. 12, in Experimental Example 3, the S / N ratio is highest for “SDR BT.709”, followed by “Log BT.709”, then “HLG BT.2020”, and “Log BT.2020” is the lowest in Experimental Example 3.

[0101] [Variation 1] Fig. 13 is a functional block diagram of a mobile terminal 1 of the first modification of the embodiment. As shown in Fig. 13, the control unit 20 (control unit 20a) of the imaging device 40 may further include a determination unit 25. Fig. 14 is a flowchart showing an example of the operation of the control unit 20 of the first modification of the embodiment.

[0102] It is assumed that after the processes of step S11 and step S12 in FIG. 14 are NO, the imaging unit 6 and the imaging control unit 21 are capturing a moving image or a series of still images in the first imaging mode.

[0103] Next, in step S13A, the determination unit 25 determines whether or not a living body such as a person is captured in the moving image or the multiple consecutive still images when the imaging unit 6 and the imaging control unit 21 capture a moving image or multiple consecutive still images in the first imaging mode. Then, when the determination unit 25 determines that a living body is captured in the moving image or the multiple consecutive still images captured in the first imaging mode (YES in step S13A), it instructs the imaging unit 6 and the imaging control unit 21 to switch the imaging mode from the first imaging mode to the second imaging mode. As a result, the imaging unit 6 and the imaging control unit 21 switch the imaging mode from the first imaging mode to the second imaging mode and capture a moving image or multiple consecutive still images (step S14). After this, the processing of steps S15 and S16 is performed.

[0104] In this way, by providing determination unit 25, imaging device 40 switches from the first imaging mode to the second imaging mode and captures a moving image or a series of still images without waiting for a user to select changeover switch 10. This further eliminates the user's effort in acquiring pulse wave information, and makes it possible to obtain imaging device 40 that can switch the imaging mode to a setting suitable for acquiring pulse wave information in a simpler manner.

[0105] [Modification 2] The imaging device 40 or the imaging method according to the present disclosure is provided with a control unit 20a as a computer. The computer executes an imaging program to realize the functions of the imaging device 40 or the imaging method according to the present disclosure.

[0106] Moreover, the subject of the mobile terminal (information acquisition device) 1 or the information acquisition method in the present disclosure includes a control unit 20 as a computer. The computer executes an information acquisition program to realize the functions of the mobile terminal (information acquisition device) 1 or the subject of the information acquisition method in the present disclosure.

[0107] The computer includes, as its main hardware configuration, a processor that operates according to an imaging program and an information acquisition program. The type of the processor is not important as long as it can realize functions by executing the imaging program and the information acquisition program. The processor is configured with one or more electronic circuits including a semiconductor integrated circuit (IC) or an LSI (Large Scale Integration). The multiple electronic circuits may be integrated into one chip or may be provided on multiple chips. The multiple chips may be integrated into one device or may be provided on multiple devices. The imaging program and the information acquisition program are recorded in a non-transitory storage medium such as a computer-readable ROM, an optical disk, or a hard disk drive. The imaging program and the information acquisition program may be stored in the storage medium in advance or may be supplied to the storage medium via an electric communication line including the Internet.

[0108] The present invention is not limited to the above-described embodiments, and may be replaced with a configuration that is substantially the same as the configuration shown in the above-described embodiments, a configuration that has the same action and effect, or a configuration that can achieve the same purpose. [Explanation of symbols]

[0109] 1: mobile terminal (biometric information acquisition device), 2: touch panel unit, 3: display unit, 4: input unit, 5: housing, 6: imaging unit, 10: changeover switch, 20: control unit, 20a: control unit, 20b: control unit, 21: imaging control unit, 22: compression unit, 23: biometric information acquisition unit, 25: determination unit, 30: storage unit, 40: imaging device, A12: region of interest, A22: region of interest, D1A: first color gamut, D1B: second color gamut, D2A: first dynamic range, D2B: second dynamic range

Claims

1. An imaging unit having a resolution of 4K or more and capturing moving images; a changeover switch for changing an imaging mode in which the imaging unit captures the moving image from a first imaging mode to a second imaging mode, the second imaging mode is an imaging mode for capturing the moving image with a higher resolution for detecting an amount of change in pulse wave information indicating a pulse wave of a living body captured in the moving image than the first imaging mode; The imaging apparatus, wherein the moving image captured in the second imaging mode is created based on a narrower color gamut than the moving image captured in the first imaging mode.

2. An imaging unit having a resolution of 4K or more and capturing video; a changeover switch for changing an imaging mode in which the imaging unit captures the moving image from a first imaging mode to a second imaging mode, the second imaging mode is an imaging mode for capturing the moving image with a higher resolution for detecting an amount of change in pulse wave information indicating a pulse wave of a living body captured in the moving image than the first imaging mode; The imaging apparatus, wherein the moving image captured in the second imaging mode is created based on a narrower dynamic range than the moving image captured in the first imaging mode.

3. The imaging device according to claim 1 or 2, and a biometric information acquisition unit that acquires the pulse wave information based on the moving image captured in the second imaging mode.

4. a determination unit that determines whether the living body is captured in the moving image; The biometric information acquisition device according to claim 3 , further comprising: an imaging control unit that switches the imaging mode of the imaging unit from the first imaging mode to the second imaging mode when the living body is captured in the video.

5. The biometric information acquisition device according to claim 3 , further comprising a compression unit configured to compress each of a plurality of frames included in the moving image captured by the imaging unit.

6. obtaining an input signal; and when the input signal is acquired, switching an imaging mode in which the imaging unit captures a moving image to a second imaging mode in which the imaging unit captures the moving image with a higher resolution for detecting an amount of change in pulse wave information indicating a pulse wave of a living body shown in the moving image than the first imaging mode, The imaging method, wherein the moving image captured in the second imaging mode is created based on a narrower color gamut than the moving image captured in the first imaging mode.

7. A step of acquiring an input signal; and when the input signal is acquired, switching an imaging mode in which the imaging unit captures a moving image to a second imaging mode in which the imaging unit captures the moving image with a higher resolution for detecting an amount of change in pulse wave information indicating a pulse wave of a living body shown in the moving image than the first imaging mode, The imaging method, wherein the moving image captured in the second imaging mode is created based on a narrower dynamic range than the moving image captured in the first imaging mode.

Citation Information

Patent Citations

  • Pulse wave detection device, pulse wave detection method, pulse wave detection system, and program

    JP2016220915A

  • Biological information detection device

    JP2018008039A

  • On-vehicle display device, control method of on-vehicle display device, and program

    JP2018038553A

  • Information processor, information processing method, information processing program, television conference system, and biological information acquisition system

    JP2019017876A

  • Signal measurement method

    JP2019209042A