Measurement method, computer program product, measurement system, and recording medium

Abstract

The invention provides a measurement method, a computer program product, a measurement system, and a recording medium. A method for measuring biological information according to the present disclosure is a method for measuring biological information on the basis of a group of frame images acquired by capturing an image of a subject over time. In a biological information measurement method according to the present disclosure, a target region in each frame image constituting a frame image group is divided into a plurality of sub-regions, and the amount of variation in each of the plurality of sub-regions is extracted from the frame image group. Furthermore, in the biological information measurement method, sub-regions having similar variation amounts are identified from among the plurality of sub-regions, and biological information of the subject is acquired on the basis of the accumulated value of the variation amounts for each of the similar sub-regions.

Description

Technical Field

[0001] The present invention relates to a measurement method, a computer program product, a measurement system, and a recording medium, and more particularly to a technology for improving the measurement accuracy of biological information. Background Art

[0002] Biometric information is used to measure the current health status of the person being measured, diagnose diseases, and determine the effectiveness of treatment. In order to detect changes in physical conditions at an early stage and treat diseases at an early stage, there is an increasing demand for measuring biometric information in the daily life of the family.

[0003] In daily biological information measurement, a non-contact measurement method is preferred for the measured person without side effects such as discomfort and skin rashes caused by wearing a measuring device. In this regard, Japanese Patent Publication No. 2016-522027 discloses a technology for detecting vital signs from a frame image group obtained by photographing a measured person.

[0004] In addition, when acquiring biometric information based on a moving image obtained by photographing a subject, the movement derived from the biometric information is sometimes slight, so that the measurement sensitivity of the subject's biometric information is insufficient. In this regard, Shoichiro Takeda, Megumi Isogai, Shinya Shimizu, Hideaki Kimata, "Local Riesz Pyramid for Faster Phase-Based Video Magnification", June 2020, IEICE Transactions on Information and Systems disclosed a technology for enhancing subtle changes in images. Summary of the invention

[0005] By using the technology disclosed in Japanese Translation of National Publication No. 2016-522027, biological information can be acquired from a frame image group obtained by photographing a subject. However, there are cases where movement derived from the biological information is small and the biological information cannot be measured with sufficient measurement sensitivity.

[0006] In addition, by using the technology disclosed in Shoichiro Takeda, Megumi Isogai, Shinya Shimizu, Hideaki Kimata, "Local Riesz Pyramid for Faster Phase-Based Video Magnification", June 2020, IEICE Transactions on Information and Systems, it is possible to enhance subtle movements in the image, but sometimes not only the movement derived from the biological information set as the target is enhanced, but also the movement unrelated to the biological information set as the target is enhanced. In this case, the noise becomes larger and it may not be possible to obtain the biological information with sufficient measurement accuracy.

[0007] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a technology for improving the measurement accuracy of biological information.

[0008] A measurement method according to the first mode of the present disclosure is used to measure biological information based on a group of frame images obtained by photographing a subject over time, and the measurement method includes the following steps: dividing the object area in each frame image constituting the frame image group into multiple sub-areas; extracting the change amount of each sub-area in the multiple sub-areas from the frame image group; determining sub-areas with similar change amounts in the multiple sub-areas; and acquiring the biological information of the subject based on the accumulated value of the change amount of each sub-area in the similar sub-areas.

[0009] A computer program product according to a second aspect of the present disclosure includes a program, and when a processor mounted on a computer executes the program, the computer executes the above-mentioned measurement method.

[0010] A third-party measurement system according to the present disclosure comprises: a camera that photographs a subject over time to generate a frame image group; and a processing device that performs processing to measure the subject's biological information based on the frame image group generated by the camera, the processing device performing the following processing: dividing an object area in each frame image constituting the frame image group into a plurality of sub-areas; extracting a change amount of each sub-area in a plurality of sub-areas from the frame image group; determining sub-areas in a plurality of sub-areas with similar change amounts; and acquiring the subject's biological information based on the accumulated value of the change amounts of each sub-area in the similar sub-areas.

[0011] The recording medium according to the fourth aspect of the present disclosure stores the program in a non-transitory manner, and the processor mounted on the computer executes the program to cause the computer to execute the above-mentioned measuring method.

[0012] The above and other objects, features, aspects and advantages of the present invention will become more apparent from the following detailed description of the present invention when read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a schematic diagram of the measurement system.

[0014] Figure 2 is a diagram showing the hardware configuration of the processing system.

[0015] Figure 3 This is a diagram for explaining a method of identifying sub-regions having similar amounts of variation and acquiring the amount of variation of a target region.

[0016] Figure 4 This is a flowchart showing the process related to the measurement of biological information.

[0017] Figure 5 This is a diagram for explaining a method of setting a target area.

[0018] Figure 6 This is a diagram for explaining a method of taking a frame image as input and outputting the position of a landmark.

[0019] Figure 7 This is a diagram for explaining a method of setting a target area in a ring shape within a certain angle range with the position of a landmark as the rotation center. DETAILED DESCRIPTION

[0020] Embodiments of the present invention will be described in detail with reference to the accompanying drawings. In addition, the same reference numerals are attached to the same or corresponding parts in the drawings, and their description will not be repeated.

[0021] [Overall structure of the measurement system]

[0022] Figure 1 1 is a diagram schematically showing the overall structure of the measurement system 100 according to the present embodiment. Figure 1 As shown, the measurement system 100 includes a camera 1 and a processing system 2. The measurement system 100 acquires biological information of the subject P from data acquired by, for example, capturing an image of the subject P while sleeping.

[0023] The camera 1 is configured to include the measured person P in the shooting field of view. The camera 1 includes an optical system such as a lens and an imaging element. The imaging element is implemented, for example, by a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. The imaging element generates imaging data by converting light incident from the measured person P through the optical system into an electrical signal. In addition, the camera 1 can be a fixed-setting camera, a camera that can adjust the camera angle and zoom by remote operation, or a depth camera that can obtain three-dimensional information.

[0024] The processing system 2 includes a main device 21 and a terminal 22. The processing system 2 analyzes the image data acquired by the camera 1 to acquire the biological information of the person to be measured P. In addition, the acquired biological information is displayed to the person to be measured.

[0025] In this embodiment, the main device 21 is connected to the camera 1 via wired or wireless communication so that it can communicate. The main device 21 is, for example, a general-purpose computer such as a desktop computer. In addition, the terminal 22 is connected to the main device 21 via the Internet so that it can communicate with each other. The terminal 22 is, for example, a smart phone or a tablet terminal. The hardware structure of the main device 21 and the terminal 22 will be described later.

[0026] In addition, the main device 21 and the terminal 22 may be physically integrated like a personal computer or a tablet computer, or may further include a server system between the main device 21 and the terminal 22. In addition, the processing system 2 may be integrated with the camera 1. The camera 1 and the processing system 2 are not limited to this physical form as long as they perform the same functions.

[0027] Figure 2 2 is a diagram showing the hardware configuration of the processing system 2. Figure 2 To illustrate the respective hardware structures of the main device 21 and the terminal 22.

[0028] The main device 21 includes a processor 211 , an input / output port 212 , a memory 213 , an input device 214 , and an output device 215 .

[0029] The processor 211 is an example of a circuit, and controls the operation of the main device 21 by executing a given program. The program executed by the processor 211 may be stored in the memory 213 or in a storage device located outside the main device 21. The processor is, for example, a CPU (Central Processing Unit).

[0030] The memory 213 stores data. The data stored in the memory 213 includes a data processing program 2130. The memory 213 includes a volatile memory (for example, RAM (Random Access Memory)) and a nonvolatile memory (for example, ROM (Read Only Memory), a hard disk drive, and a solid state drive).

[0031] The data processing program 2130 is a program for acquiring biological information from the image data. The data processing program 2130 may be stored in the memory 213 or in an external storage device accessible to the processor 211 .

[0032] The data processing program 2130 is executed by the processor 211 to enable the processor 211 to implement at least eight functions (a setting unit 2131, a division unit 2132, an extraction unit 2133, a determination unit 2134, an accumulation unit 2135, a generation unit 2136, a derivation unit 2137, and an output unit 2138). In one implementation example, the data processing program 2130 includes a module for each function. The content of each function is described later.

[0033] The input device 214 receives input of information to the main device 21. This information is, for example, information for setting a target area. The input device 214 is composed of, for example, a mouse and a keyboard.

[0034] The output device 215 outputs information according to the instruction of the processor 211. The information is, for example, the image data acquired by the camera 1 and the biological information of the subject P. The output device 215 is composed of, for example, a display and a speaker.

[0035] In addition, the input of information to the main device 21 and the output of information from the main device 21 can also be performed by the terminal 22.

[0036] The terminal 22 includes a processor 221 , an input / output port 222 , a memory 223 , an input device 224 , and an output device 225 .

[0037] The processor 221 is an example of a circuit, and controls the operation of the terminal 22 by executing a given program. The program executed by the processor 221 may be stored in the memory 223 or in a storage device located outside the terminal 22. The processor is, for example, a CPU.

[0038] The memory 223 stores data. The data stored in the memory 223 include, for example, data captured by the camera 1 and biological information acquired by the main device 21. The memory 223 includes a volatile memory (e.g., RAM) and a nonvolatile memory (e.g., ROM, hard disk drive, and solid state drive).

[0039] The input device 224 receives input of information to the terminal 22. This information is, for example, information for setting a target area. The input device 224 is composed of, for example, a mouse, a keyboard, and a touch panel.

[0040] The output device 225 outputs information according to the instruction of the processor 221. The information is, for example, the image data acquired by the camera 1 and the biological information of the subject P. The output device 225 is composed of, for example, a display and a speaker.

[0041] [Comparative Example]

[0042] The non-contact measurement method of biological information does not cause side effects such as discomfort and skin rashes caused by wearing a measurement device, so it is preferred for the person being measured. As a non-contact measurement method of biological information, there is a method of obtaining biological information based on data obtained by shooting with a camera. For example, the person being measured is photographed while sleeping, and the breathing information of the person being measured is obtained from the obtained camera data.

[0043] In such a method, objects other than the subject may be captured in the image data. Therefore, the following method is sometimes used: the subject or an area corresponding to a part of the subject's body is set as a target area in the image data, and biological information is obtained based on the subject's movement in the target area.

[0044] In this case, there is a case where the body movement that is the basis of the bio-information is small and the measurement sensitivity of the bio-information is insufficient. In addition, there is also a method of increasing the signal based on the bio-information by performing a process for enhancing the subtle changes in the image data using a method called image magnification. However, this method sometimes causes the signal (noise) other than the signal based on the bio-information to increase, resulting in insufficient measurement accuracy of the bio-information.

[0045] [Measurement of biological information according to the embodiment]

[0046] Therefore, in the measurement of biological information involved in the present embodiment, the object area of ​​each frame image included in the frame image group captured and acquired over time is divided into a plurality of sub-areas, and only the sub-areas with similar changes are selected from the plurality of sub-areas. Then, the change is calculated based on the data in the selected sub-areas. According to this method, only the signal in the sub-area selected from the object area as described above is used for the measurement of biological information, so the measurement accuracy of the biological information can be improved. In addition, for the camera data that has been processed to enhance subtle changes, noise and signals based on biological information are also distinguished, and the change of the object area is calculated only based on the signals based on biological information, so the S / N ratio in the measurement of biological information can be improved.

[0047] [Image data used for measurement of biological information]

[0048] The image data acquired by the camera 1 and used for measuring the biological information of the person P to be measured will be described.

[0049] The image data is a frame image group consisting of a plurality of frame images acquired continuously. The time interval for acquiring the frame images is not particularly limited, but is preferably shorter than the cycle of the biological information to be the object. Specifically, the interval for acquiring the frame images is, for example, 33 milliseconds.

[0050] In one implementation example, the subject is captured in a frame image of the video data. The posture of the subject is not particularly limited, and may be supine, standing, or sitting. In the case of a supine position, the subject may be supine, sideways, or prone. In addition, the video data may be obtained by photographing the subject while sleeping, the subject at rest, the subject in motion, or the subject during a medical examination (such as a magnetic resonance imaging examination and a computer tomography examination). In addition, resting refers to a state in which the subject is awake rather than asleep, but has no continuous body movements. Therefore, when at rest, the relative position between the subject and the camera remains almost unchanged.

[0051] The biological information has periodicity and is represented as waveform information. For example, the biological information is respiratory information. The respiratory information specifies at least one of the number of breaths, respiratory time, exhalation time, inhalation time, amplitude, time from exhalation to the start of inhalation, and ventilation volume. The respiratory time refers to the time obtained by adding the exhalation time and the inhalation time.

[0052] As the image data, unprocessed data may be used, or data subjected to a predetermined process (for example, a process based on a so-called image magnification method (for example, Riesz Pyramid)) may be used.

[0053] Note that, at the time point when the process of acquiring biological information described later is started, all frame images of the imaging data to be analyzed may or may not have been acquired.

[0054] [Method for measuring biological information]

[0055] A method of measuring biological information based on the obtained imaging data will be described. Figure 3 1 is a diagram for explaining a method of acquiring biological information from image data obtained by the camera 1. Figure 3 In , frame image A represents a frame image acquired at a certain time contained in the camera data. Figure 3 In FIG. 1 , the target region B indicates a region in the frame image A used for measuring biological information.

[0056] <1. Setting of target area>

[0057] The setting of the object area performed by the setting unit 2131 is described. The object area is an area in the frame image used for measuring the biological information. The object area can be predetermined or specified by the measurer. Predetermining the object area means, for example, processing the entire frame image as the object area, and treating the part with a characteristic pattern of the clothing worn by the person being measured P as the object area. The measurer can also specify the object area via the input device 214 of the main device 21 or the input device 224 of the terminal 22. In addition, by setting a part of the frame image as the object area, the amount of calculation required for measuring the biological information can be reduced.

[0058] <2. Segmentation of target area>

[0059] The segmentation of the object area by the segmentation unit 2132 is described. The segmentation unit 2132 divides the object area into a plurality of sub-areas. The sub-areas may be in units of one pixel or in units of a plurality of pixels. When the sub-areas are in units of one pixel, the accuracy of measuring the biological information is improved, but there is a concern that the processing time for measuring the biological information will be prolonged due to the increase in the amount of calculation. On the other hand, when the sub-areas are in units of a plurality of pixels, it can be expected that the processing time for measuring the biological information will be shortened due to the reduction in the amount of calculation, but there is a concern that the accuracy of measuring the biological information will be reduced. Therefore, the measurer appropriately selects the unit for segmenting the object area according to the relationship between the accuracy of measuring the biological information desired by the user and the processing time required for the measurement. When the object area is the same object area, the position and range of the sub-areas in each frame image are common between the plurality of frame images.

[0060] exist Figure 3In the example, the target area B includes sub-areas C, D, and E, and is divided into 36 sub-areas of 6×6. In addition, the number of sub-areas is not limited to Figure 3 The numbers shown can be appropriately set according to the conditions for implementing the measurement system.

[0061] <3. Extraction of variation>

[0062] The extraction of the variation amount of the sub-region by the extraction unit 2133 will be described. The extraction unit 2133 extracts the variation amount of each of the plurality of sub-regions generated by dividing the target region.

[0063] The variation is, for example, the variation of the grayscale value of the sub-region. Due to the physical change in the position of a part of the subject's body, the grayscale value of the sub-region corresponding to the part may change. Based on this, the variation extracted from the sub-region of the camera data means the physical variation.

[0064] The variation is calculated, for example, as the difference between the grayscale value in the sub-region in the reference frame image and the grayscale value in the sub-region in the target frame image. Alternatively, the variation may be calculated as the difference between the average value or median of the grayscale values ​​in the sub-region in a plurality of predetermined frame images and the grayscale value in the sub-region in the target frame image.

[0065] Furthermore, the variation amount may be a variation amount of the phase based on the phase signal of each sub-region extracted by frequency analysis.

[0066] exist Figure 3 In the example of , the extraction unit 2133 extracts the variation of each sub-region. Figure 3 , the variations extracted from the sub-regions C, D, and E are represented as variations F, G, and H, respectively.

[0067] <4. Determination of Sub-region Groups>

[0068] The determination of the sub-region group by the determination unit 2134 is described. The determination unit 2134 selects sub-regions with similar variations as the sub-region group. In one implementation example, the similarity of the variations of two sub-regions means that the difference between the variations of the two sub-regions is less than a specified value. The specified value may be fixed or may be changed for each camera data. In addition, the determination of whether the sub-regions are similar may also be based on whether the Euclidean distance between the phase signals of each sub-region is less than a given value.

[0069] exist Figure 3 In the example of , a sub-region group is determined from 36 sub-regions. Figure 3 The five sub-regions constituting the sub-region group are shown by hatching.

[0070] In addition, when the frame image includes an object that is stationary in an area other than the measured person, the grayscale value in the sub-area corresponding to the object is substantially constant between frame images. Although the multiple sub-areas corresponding to such an object have similar amounts of change, it is preferable to avoid selecting them as sub-areas constituting a sub-area group. Therefore, for example, a sub-area whose amount of change between frame images is less than a prescribed value may also be excluded in the process of determining a sub-area group. The prescribed value may be fixed or may be changed for each camera data.

[0071] In addition, for the same purpose, the determination unit 2134 may normalize the phase signal obtained from each sub-region, and when the variance of the normalized phase signal is smaller than a predetermined value, the sub-region that is the source of the phase signal is excluded from the determination process of the sub-region group. The predetermined value may be fixed or may be changed for each imaging data.

[0072] When the number of subregions constituting the subregion group is less than the prescribed number, the determination unit 2134 sometimes returns the process to the setting of the target region. This is because, for example, the subject P may have moved and left the range of the set target region when the number of subregions constituting the subregion group is less than the prescribed number. Therefore, it is necessary to newly set the target region at a position different from the set target region. That is, when the number of subregions constituting the subregion group is less than the prescribed number, the setting unit 2131 newly sets the target region. In addition, the prescribed value may be fixed or may be changed for each imaging data.

[0073] <5. Accumulation of changes>

[0074] The accumulation of the variation by the accumulation unit 2135 is described. The accumulation unit 2135 accumulates the variation of each sub-region constituting the sub-region group to calculate the variation of the frame image. In one implementation example, the average value or median of the variation of the plurality of sub-regions constituting the sub-region group is calculated as the variation of the frame image.

[0075] exist Figure 3 , the variation I calculated based on the variation of only the five sub-regions constituting the sub-region group is represented as the variation of the frame image A.

[0076] <6. Generation of Waveform Data>

[0077] The generation of waveform data by the generator 2136 will be described. The above-mentioned processes 1. to 5. are performed on a plurality of frame images to calculate the variation of each frame image. The generator 2136 generates waveform data based on the variation of each frame image of the image data.

[0078] <7. Export of biological information>

[0079] The following describes the derivation of biological information by the derivation unit 2137. The derivation unit 2137 derives the measurement result of the biological information based on the generated waveform data. For example, the measurement result of the biological information is derived based on the cycle of the waveform data.

[0080] <8. Output of biological information>

[0081] The output of the biological information by the output unit 2138 will be described. The measured biological information is displayed on the output device 215 and / or 225. The waveform data may be displayed together with the measured biological information.

[0082] [Processing flow]

[0083] Figure 4 FIG. 2 is a flowchart of a process performed in the processing system 2 to measure biological information. In one implementation example, Figure 4 The processing is called and executed from the main program when the processor 211 executes a given program. The camera data is a frame image group consisting of L frame images captured over time. The processor 211 measures the biometric information based on the Mth to Nth frame images (M and N are integers satisfying 1≤M<N≤L) among the L frame images. The values ​​of M and N can be predetermined or input by the user when the processor 211 executes a given program. In addition, Figure 4 At the time point when the processing starts, the camera data may not be completely acquired. That is, the processor 211 may also execute the processing for the frame images acquired sequentially by the camera 1. Figure 4 processing.

[0084] In step S10, the processing system 2 will Figure 4 The value of the variable K used in the processing is set to M.

[0085] In step S12 , the processing system 2 sets the target region in the K-th frame image in the imaging data received from the camera 1 as described above, for example, as a function of the setting unit 2131 .

[0086] In step S14 , the processing system 2 divides the target region set in step S12 into a plurality of sub-regions as described above, for example, as a function of the dividing unit 2132 .

[0087] In step S16 , the processing system 2 extracts the variation amount of each of the plurality of sub-regions generated in step S14 as described above, for example, as a function of the extraction unit 2133 .

[0088] In step S18 , the processing system 2 determines the subregion group as described above, that is, selects subregions whose amounts of variation are similar to each other, for example, as a function of the determination unit 2134 .

[0089] In step S20, the processing system 2 determines whether the number of sub-regions constituting the sub-region group determined in step S18 is greater than or equal to a prescribed number. The prescribed number is at least 2. If the number of sub-regions determined is greater than or equal to the prescribed number ("Yes" in step S20), the processing system 2 causes the control to proceed to step S22. Otherwise ("No" in step S20), the control returns to step S12.

[0090] In addition, the determination in step S20 may also be based on the ratio of the number of selected sub-regions to the total number of sub-regions. In this case, if the ratio is greater than a predetermined value, the processing system 2 causes the control to proceed to step S22, otherwise, the control returns to step S12.

[0091] In step S22 , the processing system 2 accumulates the variation amounts of similar sub-regions as described above, for example, as a function of the accumulation unit 2135 , and calculates the variation value of the K-th frame image.

[0092] In step S24, the processing system 2 determines whether the value of the variable K has reached the above-mentioned N. If the value of the variable K has reached the above-mentioned N ("Yes" in step S24), the processing system 2 causes the control to proceed to step S28, otherwise ("No" in step S24), the processing system 2 causes the control to proceed to step S26.

[0093] In step S26, processing system 2 adds 1 to the value of variable K and returns control to step S16.

[0094] In step S28 , the processing system 2 calculates waveform data based on the amount of change from the Mth to the Nth frame images as described above, for example, as a function of the generation unit 2136 .

[0095] In step S30 , the processing system 2 derives the measurement result of the biological information desired by the measurer based on the waveform data obtained in step S28 as described above, for example, as a function of the deriving unit 2137 .

[0096] In step S32, the processing system 2 outputs the biological information derived in step S30 to the output device 215 and / or the output device 225 as described above, for example, as a function of the output unit 2138. Thereafter, the processing system 2 ends the biological information measurement subroutine and returns the processing to the main routine.

[0097] In the above-mentioned biological information measurement method, only the region including the variation derived from the biological information among the sub-regions obtained by dividing the target region is selectively used for the biological information measurement, thereby improving the measurement accuracy of the biological information measurement.

[0098] In addition, by setting the target region, the amount of calculation can be reduced, and by excluding a region that does not contain a biological information signal from the target region, noise can be prevented from being measured, thereby improving the S / N ratio in the measurement of biological information.

[0099] In addition, when a frame image is newly acquired after the biological information is acquired, the biological information can also be updated based on the amount of change contained in the newly acquired frame image. For example, imagine a case where biological information is measured based on camera data consisting of L frame images captured over time. When the processing system 2 acquires the L+1th frame image after the Lth acquired frame image, it performs the processing of steps S12 to S22 on the L+1th frame image to calculate the amount of change in the L+1th frame image. The processing system 2 generates new waveform data based on the calculated amount of change in the L+1th frame image and the previously generated waveform data. The processing system 2 updates the biological information based on the newly generated waveform data. The updated biological information is biological information based on the camera data consisting of L+1 frame images captured over time.

[0100] For example, when the biological information measurement result is derived in parallel with the acquisition of the image data, the biological information is updated based on the frame image added to the image data, thereby providing the measurer with the latest biological information of the subject.

[0101] [Other embodiments related to setting of target area]

[0102] A specific example of another embodiment related to the setting of the target area will be described.

[0103] <Example 1>

[0104] Figure 5 is a diagram for explaining a method for setting a target area. Figure 5 In this case, in the frame image displayed on the output device 215 and / or the output device 225, the target area is set as a rectangular area set by the measurer to include the entirety of the subject P. This setting is implemented via the input device 214 and / or 224.

[0105] In addition, the object area may be set automatically by the processing system 2 instead of according to the input by the measurer. For example, the processing system 2 determines the area where the measured person P is photographed through image recognition based on machine learning such as semantic segmentation, and automatically sets a suitable area of ​​a certain shape such as a rectangle containing the area where the measured person P is photographed as the object area. The size and position of the object area relative to the overall frame image may also change in each frame image according to the activities of the measured person, etc. It is preferred to make the ratio of the size of the object area to the size of the photographed measured person P equal. In addition, the object area is not limited to a rectangle, etc., and may also be a shape of a pixel unit.

[0106] <Example 2>

[0107] A method of setting a target region based on a landmark in a frame image will be described. The processing system 2 detects a landmark in a frame image and sets a target region based on the landmark.

[0108] For example, the landmarks may be the face or neck of the person being measured. The positions of the landmarks may be determined based on the input of the person being measured or based on a learned model (eg, Yolov5-face).

[0109] exist Figure 6 A specific example is shown in (a) to (d). Here, the processing system 2 rotates the frame image (shown in (a) of the figure) by a certain angle within the plane (shown in (b) of the figure), and estimates the position of the landmark at multiple angles through machine learning (shown in (c) of the figure), and adopts the detection result of the landmark position at the angle with the highest confidence (shown in (d) of the figure).

[0110] If the landmark is a face and the biological information is respiratory information, the processing system 2 determines the area of ​​the chest or abdomen based on the position of the facial area including the face as the landmark. The processing system 2 extracts the respiratory information as the biological information based on the estimated movement of the area of ​​the chest or abdomen. However, in the case where the face is tilted relative to the axis of the body, the actual position of the area of ​​the chest or abdomen may be offset from the determined position. Therefore, the processing system 2 may also determine the area around the determined position of the area of ​​the chest or abdomen as the target area, and extract the respiratory information as the biological information for the target area.

[0111] Alternatively, the target area may be set in a circular shape within a certain angle range with the determined landmark position as the rotation center. Figure 7 The processing system 2 calculates the face area ( Figure 7 (a)), and calculate its center point ( Figure 7Then, the processing system 2 draws double concentric circles according to the center point, and sets a fan-shaped shape of a certain angle according to the annular area surrounded by the double concentric circles ( Figure 7 (c)). The processing system 2 applies the shape to the angle image with high confidence, and sets the range of the fan-shaped shape to the range where the chest or abdomen may exist ( Figure 7 (d)).

[0112] In addition, the position and size of the chest and abdomen of the subject P can also be estimated based on the feature quantity obtained from the image of the face of the subject P, that is, the facial feature quantity. The facial feature quantity is, for example, the distance between two facial feature points, and the brightness average, brightness histogram, and area in a triangular area formed by connecting three facial feature points. Facial feature points are multiple points set on the face. Facial feature points are, for example, set to the inner corner of the eye, the outer corner of the eye, the pupil, the eyebrow, the eyebrow peak, the eyebrow tail, the nose tip, the nose wing, the root of the nose, and / or the corner of the mouth. Alternatively, the position and size of the chest and abdomen of the subject P can also be estimated based on the bone feature quantity obtained from the image of the whole body. The bone feature quantity is, for example, the brightness average, brightness histogram, and area of ​​the area surrounded by the feature points of the shoulder, waist, and neck estimated based on the image of the whole body.

[0113] By limiting the target area in this way, the amount of calculation related to the measurement of biological information can be reduced.

[0114] [Way]

[0115] Those skilled in the art will appreciate that the above-described multiple exemplary embodiments are specific examples of the following aspects.

[0116] (Item 1) A measurement method in one embodiment may also be a measurement method for measuring biological information based on a group of frame images obtained by photographing a subject over time, comprising the following steps: dividing an object area in each frame image constituting the group of frame images into a plurality of sub-areas; extracting a change amount of each sub-area in the plurality of sub-areas from the group of frame images; determining sub-areas in the plurality of sub-areas having similar changes; and acquiring the biological information of the subject based on a cumulative value of the changes of each sub-area in the similar sub-areas.

[0117] According to the measurement method described in the first aspect, there is provided a technique for improving the measurement accuracy of biological information in the measurement of biological information based on data obtained by photographing a person to be measured.

[0118] (Item 2) In the measurement method described in Item 1, the frame image may include the subject who is sleeping.

[0119] According to the measurement method described in the second aspect, there is provided a technique for improving the measurement accuracy of biological information in the measurement of biological information based on data obtained by photographing a subject while sleeping.

[0120] (Item 3) In the measurement method described in Item 1, the frame image may include the subject at rest.

[0121] According to the measurement method described in claim 3, there is provided a technique for improving the measurement accuracy of biological information in the measurement of biological information based on data obtained by photographing a subject at rest.

[0122] (Item 4) In the measurement method described in Item 1 or Item 2, the biological information may be respiratory information.

[0123] According to the measurement method described in claim 4, there is provided a technique for improving the measurement accuracy of respiratory information in the measurement of respiratory information based on data obtained by imaging a subject.

[0124] (Item 5) In the measurement method described in Item 4, the respiratory information may include at least one of the number of respirations, respiratory time, exhalation time, inhalation time, amplitude, time from exhalation to start of inhalation, and ventilation volume.

[0125] According to the measurement method described in the fifth item, a technology is provided for improving the measurement accuracy of respiratory information in the measurement of respiratory information based on data obtained by photographing a subject, wherein the respiratory information includes at least one of the number of breaths, respiratory time, exhalation time, inhalation time, amplitude, time from exhalation to the start of inhalation, and ventilation volume.

[0126] (Item 6) In the measurement method described in Items 1 to 5, the sub-region may be composed of one pixel or a plurality of pixels.

[0127] According to the measurement method described in item 6, in the measurement of biological information based on data obtained by photographing a person to be measured, the target area is divided into sub-areas consisting of one pixel or a plurality of pixels. By changing the size of the sub-areas, the measurement accuracy of the biological information and the amount of calculation related to the process of obtaining the biological information can be adjusted.

[0128] (Item 7) In the measurement method described in Items 1 to 6, the method may further include the following step: setting the object area in the frame image.

[0129] According to the measurement method described in the seventh item, in the measurement of biological information based on data obtained by photographing a person to be measured, a target area is set. By setting the target area, the amount of calculation related to the process of obtaining biological information can be reduced, and the measurement accuracy of biological information can be improved by excluding areas irrelevant to the acquisition of biological information.

[0130] (Item 8) In the measurement method described in Items 1 to 7, it may also include the following steps: determining whether the object area includes more than a specified number of sub-areas with similar variations; and setting the object area based on determining that the object area does not include more than a specified number of sub-areas with similar variations.

[0131] According to the measurement method described in the eighth item, in the measurement of biological information based on data obtained by photographing a subject, a target area is set, and the set target area is further divided into sub-areas. If the number of similar sub-areas among these sub-areas is not more than a predetermined number, the target area is set again, thereby providing a technology for measuring the biological information of the subject even when the subject moves outside the target area.

[0132] (Item 9) In the measurement method described in Item 7 or Item 8, it may also include the following steps: estimating the position and size of the chest or abdomen of the subject based on the position and orientation of the subject's head, and in the setting step, setting the area including the chest or abdomen as the object area based on the position and size of the chest or abdomen.

[0133] According to the measurement method described in the ninth item, the following technology is provided: in the measurement of biological information based on data obtained by photographing a subject, an area including the chest or abdomen of the subject is set as a target area and the biological information is measured. By setting the area including the chest or abdomen of the subject as the target area, noise can be reduced, thereby improving the measurement accuracy of the biological information.

[0134] (Item 10) In the measurement method described in Item 9, in the estimating step, the position and size of the chest or abdomen of the subject may be estimated based on the feature value of the face of the subject.

[0135] According to the measurement method described in the tenth item, the following technology is provided: in the measurement of biological information based on data obtained by photographing a subject, the position and size of the chest are estimated based on the feature amount of the subject's face, and the area including the chest is set as the target area to measure the biological information. By setting the area including the chest or abdomen of the subject as the target area, noise can be reduced, thereby improving the measurement accuracy of the biological information.

[0136] (Item 11) The program in one aspect may be a program executed by a processor mounted on a computer, the program causing the computer to execute the measurement method described in any one of Items 1 to 10.

[0137] According to the program described in the eleventh item, there is provided a technique for improving the measurement accuracy of biological information in the measurement of biological information based on data obtained by photographing a subject.

[0138] (Item 12) A measurement system in one embodiment may include: a camera that photographs a subject over time to generate a frame image group; and a processing device that performs processing to measure the biological information of the subject based on the frame image group generated by the camera, wherein the processing device performs the following processing: dividing the object area in each frame image constituting the frame image group into a plurality of sub-areas; extracting the amount of change of each sub-area in the plurality of sub-areas from the frame image group; determining sub-areas in the plurality of sub-areas having similar amounts of change; and acquiring the biological information of the subject based on the accumulated value of the amount of change of each sub-area in the similar sub-areas.

[0139] According to the measurement system described in the twelfth item, there is provided a technology for improving the measurement accuracy of biological information in the measurement of biological information based on data obtained by photographing a person to be measured.

[0140] Although the embodiments of the present invention have been described, the embodiments disclosed this time should be considered to be illustrative in all aspects and not restrictive. The scope of the present invention is indicated by the claims, and it is intended to include all modifications within the meaning and scope equivalent to the claims.

Claims

1. A method for measuring biological information based on a group of frame images acquired by photographing a person to be measured over time, the method comprising the following steps: dividing the object area in each frame image constituting the frame image group into a plurality of sub-areas; Extracting the variation of each sub-region in the plurality of sub-regions from the frame image group; determining a sub-region among the plurality of sub-regions having a similar amount of change; as well as The biological information of the subject is acquired based on the accumulated value of the variation amount in each of the similar sub-regions.

2. The assay method according to claim 1, wherein The frame image includes the subject who is sleeping.

3. The measuring method according to claim 1, wherein The frame image includes the subject at rest.

4. The assay method according to claim 1, wherein The biological information is breathing information.

5. The assay method according to claim 4, wherein The respiratory information includes at least one of the number of respirations, respiratory time, exhalation time, inhalation time, amplitude, time from exhalation to start of inhalation, and ventilation volume.

6. The assay method according to claim 1, wherein The sub-region is composed of one pixel or a plurality of pixels.

7. The measuring method according to claim 1, wherein The following steps are also included: The object area is set in the frame image.

8. The assay method according to claim 7, further comprising the following steps: determining whether the target region includes more than a predetermined number of sub-regions having similar amounts of change; as well as The target region is set based on a determination that the target region does not include a predetermined number or more of sub-regions having similar amounts of variation.

9. The assay method according to claim 7, wherein The following steps are also included: estimating the position and size of the chest or abdomen of the subject based on the position and orientation of the subject's head, In the setting step, a region including the chest or the abdomen is set as the target region based on the position and size of the chest or the abdomen.

10. The measuring method according to claim 9, wherein In the estimating step, the position and size of the chest or abdomen of the subject are estimated based on the feature quantity of the face of the subject. 11 . A computer program product comprising a program, wherein a processor mounted on a computer executes the program to cause the computer to execute the measurement method according to claim 1 .

12. A measurement system comprising: a camera that captures the subject over time to generate a frame image group; and a processing device for executing processing for measuring biological information of the subject based on the frame image group generated by the camera, The processing device performs the following processing: dividing the object area in each frame image constituting the frame image group into a plurality of sub-areas; Extracting the variation of each sub-region in the plurality of sub-regions from the frame image group; determining a sub-region among the plurality of sub-regions having a similar amount of change; as well as The biological information of the subject is acquired based on the accumulated value of the variation amount in each of the similar sub-regions. 13 . A recording medium storing a program in a non-transitory state, wherein a processor mounted on a computer executes the program to cause the computer to execute the measurement method according to claim 1 .

Images