Measuring method, program, and measuring system

Abstract

To provide a technology to improve measuring accuracy of biological information in the measurement of biological information based on the data that images a person to be measured.SOLUTION: A biological information measuring method is a measuring method for measuring biological information from a frame image group acquired by imaging a person to be measured temporally. The biological information measuring method includes: a step S14 for dividing an object region in each frame image constituting the frame image group into a plurality of sub regions; a step S16 for extracting each variation amount of the plurality of sub regions from the frame image group; a step S18 for determining sub regions with similar variation amounts of the plurality of sub regions; and a step S30 for acquiring biological information on the person to be measured on the basis of integrated values of each variation amount in the similar sub regions.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a measurement method, a program, and a measurement system, and more particularly to a technique for improving the measurement accuracy of biological information. [Background technology]

[0002] Biometric information is used as an indicator of a subject's current health condition, to diagnose illnesses, and to evaluate the effectiveness of treatment. There is an increasing need to measure biological information at home in order to detect changes in physical condition early and treat illnesses early.

[0003] In daily measurements of biological information, non-contact measurement methods that do not cause side effects such as discomfort and skin rashes due to wearing a measuring device are preferable for the subject. In this regard, JP2016-522027A (Patent Document 1) discloses a technology for detecting vital signs from a group of frame images of a subject.

[0004] In addition, when acquiring biometric information based on a video of a person being measured, the movement resulting from the biometric information may be slight, and the measurement sensitivity of the biometric information of the person being measured may be insufficient. In this regard, Shoichiro Takeda, Megumi Isogai, Shinya Shimizu, Hideaki Kimata, "Local Riesz Pyramid for Faster Phase-Based Video Magnification", June 2020, IEICE Transactions on Information and Systems (Non-Patent Document 1) discloses a technology that emphasizes minute changes in a video. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special Publication No. 2016-522027 [Non-patent literature]

[0006] [Non-Patent Document 1] Shoichiro Takeda, Megumi Isogai, Shinya Shimizu, Hideaki Kimata, “Local Riesz Pyramid for Faster Phase-Based Video Magnification”, June 2020, IEICE Transactions on Information and Systems Summary of the Invention [Problem to be solved by the invention]

[0007] By using the technology disclosed in Patent Document 1, it is possible to obtain biometric information from a group of frame images of the subject, but there are cases where the movements resulting from the biometric information are so minute that the biometric information cannot be measured with sufficient measurement sensitivity.

[0008] In addition, by using the technology disclosed in Non-Patent Document 1, it is possible to emphasize minute movements in the video, but in some cases, not only the movements originating from the target biometric information but also movements unrelated to the target biometric information are emphasized. In such cases, noise becomes large and the biometric information may not be obtained with sufficient measurement accuracy.

[0009] The present invention has been made in view of the above circumstances, and has an object to provide a technique for improving the measurement accuracy of biological information. [Means for solving the problem]

[0010] A measurement method according to a first aspect of the present disclosure is a method for measuring biometric information from a group of frame images acquired by photographing a subject over time, and comprises the steps of dividing a target area in each frame image constituting the group of frame images into a plurality of sub-areas, extracting an amount of variation for each of the plurality of sub-areas from the group of frame images, determining sub-areas among the plurality of sub-areas having similar amounts of variation, and acquiring the biometric information of the subject based on an integrated value of the amounts of variation for each of the similar sub-areas.

[0011] A program according to a second aspect of the present disclosure is a program executed by a processor installed in a computer, causing the computer to execute the calculation method according to the first aspect.

[0012] A measurement system according to a third aspect of the present disclosure comprises a camera that photographs a person being measured over time to generate a group of frame images, and a processing device that executes a process of measuring biometric information of the person being measured from the group of frame images generated by the camera, wherein the processing device executes the steps of dividing a target area in each frame image that constitutes the group of frame images into a plurality of sub-areas, extracting the amount of variation of each of the plurality of sub-areas from the group of frame images, determining sub-areas among the plurality of sub-areas that have similar amounts of variation, and acquiring the biometric information of the person being measured based on an integrated value of the amount of variation of each of the similar sub-areas. Effect of the Invention

[0013] According to the present disclosure, it is possible to improve the measurement accuracy of biological information in measurement based on data obtained by photographing a subject. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a schematic diagram of a measurement system. [Diagram 2] FIG. 2 is a diagram illustrating a hardware configuration of the processing system. [Diagram 3]11A and 11B are diagrams for explaining a method of identifying sub-areas with similar amounts of variation and acquiring the amount of variation of a target area. [Figure 4] 4 is a flowchart showing a process for measuring biological information. [Diagram 5] FIG. 13 is a diagram for explaining a method for setting a target region. [Figure 6] FIG. 13 is a diagram for explaining a method of inputting a frame image and outputting the positions of landmarks. [Figure 7] 13 is a diagram for explaining a method for setting a donut-shaped target area within a certain angle range with the position of a landmark as the center of rotation. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings, in which the same or corresponding parts in the drawings are designated by the same reference characters and will not be described repeatedly.

[0016] [Overall configuration of the measurement system] Fig. 1 is a diagram showing a schematic overall configuration of a measurement system 100 according to this embodiment. As shown in Fig. 1, the measurement system 100 includes a camera 1 and a processing system 2. The measurement system 100 acquires biological information of the subject P from data acquired by, for example, photographing the subject P while sleeping.

[0017] The camera 1 is disposed so as to include the subject P in the field of view. The camera 1 includes an optical system such as a lens and an imaging element. The imaging element is realized, for example, by a charge coupled device (CCD) sensor and a complementary metal oxide semiconductor (CMOS) sensor. The imaging element generates imaging data by converting light incident from the subject P via the optical system into an electrical signal. The camera 1 may be a fixed installation type, may be a type in which the imaging angle and zoom can be adjusted by remote control, or may be a depth camera capable of acquiring three-dimensional information.

[0018] The processing system 2 includes a main unit 21 and a terminal 22. The processing system 2 analyzes the photographed data acquired by the camera 1, and acquires biological information of the subject P. In addition, the processing system 2 displays the acquired biological information to the person being measured.

[0019] In this embodiment, the main device 21 is communicably connected to the camera 1 by wire or wirelessly. The main device 21 is, for example, a general-purpose computer such as a desktop computer. The terminal 22 is communicably connected to the main device 21 via, for example, the Internet. The terminal 22 is, for example, a smartphone or a tablet terminal. The hardware configurations of the main device 21 and the terminal 22 will be described later.

[0020] Note that main device 21 and terminal 22 may be physically integrated, for example, as a personal computer or a tablet, or may further include a server system interposed between main device 21 and terminal 22. Processing system 2 may be integrated with camera 1. Camera 1 and processing system 2 are not limited to their physical forms as long as they exhibit equivalent functions.

[0021] Fig. 2 is a diagram showing a hardware configuration of the processing system 2. The hardware configurations of the main device 21 and the terminal 22 will be described with reference to Fig. 2.

[0022] The main unit 21 includes a processor 211 , an input / output port 212 , a memory 213 , an input device 214 , and an output device 215 .

[0023] The processor 211 is an example of an electric circuit, and controls the operation of the main device 21 by executing a given program. The program executed by the processor 211 may be stored in the memory 213, or may be stored in a storage device external to the main device 21. The processor is, for example, a CPU (Central Processing Unit).

[0024] The memory 213 stores data. The data stored in the memory 213 includes a data processing program 2130. The memory 213 includes a volatile memory (e.g., a Random Access Memory (RAM)) and a non-volatile memory (e.g., a Read Only Memory (ROM), a hard disk drive, and a solid state drive).

[0025] The data processing program 2130 is a program for acquiring biological information from the captured image data. The data processing program 2130 may be stored in the memory 213 or in an external storage device accessible by the processor 211.

[0026] The data processing program 2130, when executed by the processor 211, causes the processor 211 to realize at least eight functions (a setting unit 2131, a dividing unit 2132, an extracting unit 2133, a deciding unit 2134, an accumulating unit 2135, a generating unit 2136, a derivation unit 2137, and an output unit 2138). In one implementation example, the data processing program 2130 includes a module for each function. The contents of each function will be described later.

[0027] The input device 214 receives information input to the main device 21. The information is, for example, an area for setting a target area. The input device 214 is, for example, configured with a mouse and a keyboard.

[0028] The output device 215 outputs information according to instructions from the processor 211. The information is, for example, the photographed data acquired by the camera 1 and the biological information of the subject P. The output device 215 is, for example, configured with a display and a speaker.

[0029] The input of information to main device 21 and the output of information from main device 21 may be performed by terminal 22.

[0030] The terminal 22 includes a processor 221 , an input / output port 222 , a memory 223 , an input device 224 , and an output device 225 .

[0031] The processor 221 is an example of an electric circuit, and controls the operation of the terminal 22 by executing a given program. The program executed by the processor 221 may be stored in the memory 223, or may be stored in a storage device external to the terminal 22. The processor is, for example, a CPU.

[0032] The memory 223 stores data. The data stored in the memory 223 is, for example, data captured by the camera 1 and biometric information acquired by the main device 21. The memory 223 includes a volatile memory (e.g., RAM) and a non-volatile memory (e.g., ROM, a hard disk drive, and a solid state drive).

[0033] The input device 224 accepts information input to the terminal 22. The information is, for example, an area for setting a target area. The input device 224 is, for example, configured with a mouse, a keyboard, and a touch panel.

[0034] The output device 225 outputs information according to instructions from the processor 221. The information is, for example, the photographed data acquired by the camera 1 and the biological information of the subject P. The output device 225 is, for example, configured with a display and a speaker.

[0035] [Conventional methods for measuring biological information] Non-contact measurement methods of biological information are preferable for the subject because they do not cause side effects such as discomfort and skin rashes caused by wearing a measuring device. One method of non-contact measurement of biological information is to obtain biological information based on data obtained by photographing with a camera. For example, the state of the subject while sleeping is photographed, and the breathing information of the subject is obtained from the photographed data.

[0036] In such a method, there is a case where a person other than the subject is captured in the photographed data. Therefore, a method is sometimes used in which an area corresponding to the subject or a part of the subject's body is set as a target area in the photographed data, and biometric information is acquired based on the subject's movement in the target area.

[0037] Even in such a case, the body movements that are the source of the biological information may be so minute that the measurement sensitivity of the biological information may be insufficient. There is also a method of increasing the signal based on the biological information by performing a process that emphasizes minute changes in the captured data using a technique called video magnification. However, with this method, signals (noise) other than the signal based on the biological information may also be increased, and the measurement accuracy of the biological information may be insufficient.

[0038] [Features of measurement of biological information according to the embodiment] Therefore, in the measurement of biological information according to this embodiment, the target area of ​​each frame image included in the frame image group captured over time is divided into a plurality of sub-areas, and only sub-areas having similar amounts of variation are selected from the plurality of sub-areas. Then, the amount of variation is calculated based on the data in the selected sub-areas. According to this method, only the signals in the sub-areas selected from the target area as described above are used for measuring the biological information, so that the measurement accuracy of the biological information can be improved. In addition, even for the captured data that has been processed to emphasize minute changes, the signal based on the biological information and noise are distinguished, and the amount of variation in the target area is calculated based only on the signal based on the biological information, so that the S / N ratio in the measurement of the biological information can be improved.

[0039] [Photographic data used to measure biological information] The photographed data acquired by the camera 1 and used to measure the biological information of the subject P will be described.

[0040] The photographing data is a group of frame images consisting of a plurality of frame images acquired continuously. The time interval for acquiring the frame images is not particularly limited, but is preferably shorter than the period of the target biological information. Specifically, the interval for acquiring the frame images is, for example, 33 milliseconds.

[0041] In one implementation example, the subject is included in a frame image of the photographed data. The posture of the subject is not particularly limited, and may be lying down, standing, or sitting. In the lying down position, the subject may be lying on his / her back, side, or face down. The photographed data may be acquired by photographing the subject while sleeping, while at rest, while active, or while undergoing a medical examination (e.g., magnetic resonance imaging and computed tomography). Note that the state of rest refers to a state in which the subject is awake and not asleep, but does not have continuous body movement. Therefore, the relative position between the subject and the camera remains almost the same when the subject is at rest.

[0042] The biological information has periodicity and is expressed as waveform information. The biological information is, for example, respiratory information. The respiratory information specifies at least one of the respiratory rate, respiratory time, expiratory time, inhalation time, amplitude, time from exhalation to the start of inhalation, and ventilation volume. The respiratory time is the sum of the expiratory time and the inhalation time.

[0043] In addition, as the shooting data, unprocessed data may be used, or data that has been subjected to a given processing (for example, processing using a technique known as video magnification (for example, the Rees pyramid)) may be used.

[0044] It should be noted that at the time when the process of acquiring biometric information, which will be described later, is started, all frame images of the photographic data to be analyzed may or may not have been acquired.

[0045] [Method of measuring biological information] A method for measuring biometric information based on the acquired photographic data will now be described. Fig. 3 is a diagram for explaining a method for acquiring biometric information from photographic data acquired by camera 1. In Fig. 3, frame image A represents one frame image acquired at a certain timing included in the photographic data. In Fig. 3, target area B represents an area of ​​frame image A that is used for measuring biometric information.

[0046] <1. Setting the target area> Setting of the target area by the setting unit 2131 will be described. The target area is an area in the frame image used for measuring biological information. The target area may be determined in advance or may be specified by the person being measured. The target area being determined in advance means, for example, that the entire frame image is processed as the target area, and that a part of the clothes worn by the person being measured P that has a distinctive pattern is set as the target area. The target area may be specified by the person being measured via the input device 214 of the main device 21 or the input device 224 of the terminal 22. Note that by setting a part of the frame image as the target area, the amount of calculation required for measuring biological information can be reduced.

[0047] <2. Dividing the target area> The division of the target region by the dividing unit 2132 will be described. The dividing unit 2132 divides the target region into a plurality of sub-regions. The sub-regions may be in units of one pixel or in units of multiple pixels. When the sub-regions are in units of one pixel, the measurement accuracy of the biometric information is improved, but the processing time for measuring the biometric information may be longer due to an increase in the amount of calculation. On the other hand, when the sub-regions are in units of multiple pixels, the processing time for measuring the biometric information is expected to be shorter due to a decrease in the amount of calculation, but the measurement accuracy of the biometric information may be reduced. Therefore, the measurer appropriately selects the unit for dividing the target region based on the relationship between the measurement accuracy of the biometric information desired by the user and the processing time required for measurement. When the target region is the same, the position and range of the sub-regions in each frame image are common among a plurality of frame images.

[0048] In the example of Fig. 3, the target area B is divided into 36 sub-areas (6 x 6) including sub-areas C, D, and E. Note that the number of sub-areas is not limited to that shown in Fig. 3, and can be appropriately set depending on the situation in which the measurement system is realized.

[0049] <3. Extraction of fluctuations> The following describes extraction of the variation amount of the sub-regions by the extraction unit 2133. The extraction unit 2133 extracts the variation amount of each of the multiple sub-regions generated by dividing the target region.

[0050] The amount of variation is, for example, the amount of variation in the gradation value of the sub-region. The gradation value of the sub-region corresponding to a part of the subject's body may vary due to a physical variation in the position of the part. Therefore, the amount of variation extracted for the sub-region of the imaging data means a physical variation.

[0051] The amount of variation is calculated, for example, as the difference between the gradation value in the sub-region in the reference frame image and the gradation value in the sub-region in the target frame image. Alternatively, the amount of variation may be calculated as the difference between the average or median gradation value in the sub-region in a predetermined number of frame images and the gradation value in the sub-region in the target frame image.

[0052] The amount of variation may also be the amount of phase variation based on the phase signal of each sub-region extracted by frequency analysis.

[0053] In the example of Fig. 3, the extraction unit 2133 extracts the amount of variation of each sub-region. In Fig. 3, the amounts of variation extracted from sub-regions C, D, and E are shown as amounts of variation F, G, and H, respectively.

[0054] <4. Determination of sub-regions> The determination of the sub-region group by the determination unit 2134 will be described. The determination unit 2134 selects sub-regions having similar amounts of variation as the sub-region group. In one implementation example, the similarity of the amounts of variation of two sub-regions means that the difference between the amounts of variation of the two sub-regions is equal to or less than a predetermined value. The predetermined value may be fixed or may be changed for each piece of shooting data. In addition, the determination of whether the sub-regions are similar may be made based on whether the Euclidean distance between the phase signals of the respective sub-regions is equal to or less than a given value.

[0055] In the example of Fig. 3, a sub-region group is determined from among 36 sub-regions. In Fig. 3, five sub-regions that make up a sub-region group are shown by shading.

[0056] In addition, when a frame image includes a stationary object in an area other than the subject, the gradation value does not change substantially between frame images in the sub-area corresponding to the object. Although the sub-areas corresponding to such an object have similar variations, it is preferable to avoid selecting them as sub-areas constituting a sub-area group. Therefore, for example, a sub-area whose variation between frame images is equal to or less than a predetermined value may be excluded in the process of determining the sub-area group. The predetermined value may be fixed or may be changed for each piece of photographed data.

[0057] For the same purpose, the determination unit 2134 may normalize the phase signal obtained from each sub-region, and if the variance of the normalized phase signal is smaller than a predetermined value, the sub-region from which the phase signal originated may be excluded in the sub-region group determination process. The predetermined value may be fixed or may be changed for each piece of imaging data.

[0058] When the number of sub-areas constituting the sub-area group is less than a predetermined number, the determination unit 2134 may return the process to the setting of the target area. When the number of sub-areas constituting the sub-area group is less than a predetermined number, for example, the subject P may have moved and moved out of the range of the set target area. Therefore, it becomes necessary to set a new target area at a position different from the set target area. That is, when the number of sub-areas constituting the sub-area group is less than a predetermined number, the setting unit 2131 sets a new target area. The predetermined value may be fixed or may be changed for each piece of shooting data.

[0059] <5. Accumulation of fluctuations> The accumulation of the amount of variation by the accumulation unit 2135 will be described. The accumulation unit 2135 accumulates the amount of variation of each of the sub-regions constituting the sub-region group to calculate the amount of variation of the frame image. In one implementation example, the average or median of the amount of variation of the multiple sub-regions constituting the sub-region group is calculated as the amount of variation of the frame image.

[0060] In FIG. 3, the amount of variation I calculated based only on the amounts of variation of the five sub-regions that make up the sub-region group is shown as the amount of variation of frame image A.

[0061] <6. Waveform data generation> The generation of waveform data by the generation unit 2136 will be described. The above-mentioned processes 1 to 5 are performed on multiple frame images, thereby calculating the amount of variation of each frame image. The generation unit 2136 generates waveform data based on the amount of variation of each frame image of the shooting data.

[0062] <7. Extraction of Biological Information> The following describes derivation of the biological information by the derivation unit 2137. The derivation unit 2137 derives the measurement result of the biological information based on the generated waveform data. For example, the measurement result of the biological information is derived based on the period of the waveform data.

[0063] <8. Output of Biometric Information> The output of biometric information by the output unit 2138 will be described. The measured biometric information is displayed on the output devices 215 and / or 225. Waveform data may be displayed together with the measured biometric information.

[0064] [Flow of processing] FIG. 4 is a flowchart of the processing performed for measuring biometric information in the processing system 2. In one implementation example, the processing in FIG. 4 is called and executed from the main routine when the processor 211 executes a given program. The imaging data is a group of frame images composed of L frame images taken over time. The processor 211 measures biometric information based on the M-th to N-th frame images (where M and N are integers satisfying 1 ≦ M < N ≦ L) among the L frame images. The values of M and N may be determined in advance, or may be input by the user when the processor 211 executes a given program. Also, the imaging data does not necessarily have to be all acquired at the time when the processing in FIG. 4 starts. That is, the processor 211 may execute the processing in FIG. 4 for the frame images sequentially acquired by the camera 1.

[0065] In step S10, the processing system 2 sets the value of the variable K used in the processing in FIG. 4 to M.

[0066] In step S12, the processing system 2 sets the target region in the K-th frame image of the imaging data received from the camera 1, for example, as described above as the function of the setting unit 2131.

[0067] In step S14, the processing system 2 divides the target region set in step S12 into a plurality of sub-regions, for example, as described above as the function of the division unit 2132.

[0068] In step S16, the processing system 2 extracts the amount of variation of each of the plurality of sub-regions generated in step S14, for example, as described above as the function of the extraction unit 2133.

[0069] In step S18, the processing system 2 determines a sub-region group, for example as the function of the determination unit 2134 described above, that is, selects sub-regions having similar amounts of variation.

[0070] In step S20, the processing system 2 determines whether the number of sub-areas constituting the sub-area group determined in step S18 is equal to or greater than a predetermined number. The predetermined number is at least 2. If the number of identified sub-areas is equal to or greater than the predetermined number (YES in step S20), the processing system 2 proceeds to control step S22, and if not (NO in step S20), the processing system 2 returns control to step S12.

[0071] The determination in step S20 may be based on the ratio of the number of selected sub-areas to the total number of sub-areas. In this case, if the ratio is equal to or greater than a predetermined value, the processing system 2 advances control to step S22, and if not, returns control to step S12.

[0072] In step S22, the processing system 2, for example as the function of the accumulating unit 2135 described above, accumulates the amounts of variation in similar sub-regions to calculate the variation value of the Kth frame image.

[0073] In step S24, the processing system 2 determines whether or not the value of the variable K has reached the above-mentioned L. If the value of the variable K has reached the above-mentioned L (YES in step S24), the processing system 2 advances the control to step S28, and if not (NO in step S24), the processing system 2 advances the control to step S26.

[0074] In step S26, the processing system 2 counts up the value of the variable K by 1, and returns the control to step S16.

[0075] In step S28, the processing system 2 calculates waveform data based on the amount of variation of the Mth to Nth frame images, for example as described above as the function of the generation unit 2136.

[0076] In step S30, the processing system 2 derives the measurement result of the biological information desired by the subject based on the waveform data obtained in step S28, for example as described above as the function of the derivation unit 2137.

[0077] In step S32, the processing system 2 outputs the biological information derived in step S30 to the output device 215 and / or the output device 225, for example as described above as a function of the output unit 2138. Thereafter, the processing system 2 ends the biological information measurement subroutine and returns the process to the main routine.

[0078] In the above-mentioned method for measuring biological information, only the sub-regions obtained by dividing the target region, which include the variation derived from the biological information, are selectively used for measuring the biological information, thereby improving the measurement accuracy of the biological information measurement.

[0079] In addition, by setting the target region, it is possible to reduce the amount of calculations. Furthermore, by excluding the region that does not contain the signal of the biological information from the target region, it is possible to prevent noise from being measured and improve the S / N ratio in the measurement of the biological information.

[0080] In addition, when a new frame image is acquired after the biometric information is acquired, the biometric information may be updated based on the amount of variation contained in the newly acquired frame image. For example, assume a case where biometric information is measured based on imaging data consisting of L frame images captured over time. When the processing system 2 acquires the L+1th frame image after the Lth acquired frame image, the processing system 2 executes the processes of steps S12 to S22 for the L+1th frame image and calculates the amount of variation in the L+1th frame image. Based on the calculated amount of variation in the L+1th frame image and the previously generated waveform data, the processing system 2 generates new waveform data. Based on the newly generated waveform data, the processing system 2 updates the biometric information. The updated biometric information is biometric information based on imaging data consisting of L+1 frame images captured over time.

[0081] For example, when the measurement results of biometric information are derived in parallel with the acquisition of the photographic data, the biometric information is updated based on the frame image added to the photographic data, so that the latest biometric information of the person being measured can be provided to the person being measured.

[0082] [Other embodiments for setting the target area] Specific examples of other embodiments relating to setting of the target region will be described below.

[0083] <Example 1> Fig. 5 is a diagram for explaining a method for setting the target region. In Fig. 5, the entire body of the subject P is included in the frame image. In such a case, the subject sets the target region as a rectangular region that the subject sets and inputs so as to include the entire subject P in the frame image displayed on the output device 215 and / or the output device 225. This setting is performed via the input device 214 and / or 224.

[0084] The processing system 2 may automatically set the target area without relying on the setting input by the person measuring. For example, the processing system 2 identifies the area in which the subject P is reflected by image recognition using machine learning such as semantic segmentation, and automatically sets an appropriate fixed-shape area such as a rectangle containing the reflected area as the target area. The size and position of the target area relative to the entire frame image may vary for each frame image depending on the movement of the subject. It is preferable that the size of the target area is set so that the ratio to the size of the subject P reflected in the frame is equal. Furthermore, this target area is not limited to a rectangle, and may be a shape measured in pixels.

[0085] <Example 2> A method for setting a target area based on landmarks in a frame image will now be described. The processing system 2 detects landmarks in a frame image and sets a target area based on the landmarks.

[0086] For example, the landmarks may be the face or neck of the subject. The positions of the landmarks may be determined based on the subject's input or based on a trained model (e.g., Yolo-v5face).

[0087] Specific examples are shown in Figures 6(a) to 6(d). Here, the processing system 2 estimates the landmark positions at multiple angles by machine learning (as shown in Figure 6(c)) while rotating the frame image (as shown in Figure 6(a)) at a certain angle within the plane (as shown in Figure 6(b)), and adopts the detection result of the landmark position at the angle with the highest confidence (as shown in Figure 6(d)).

[0088] If the landmark is a face and the biometric information is respiratory information, the processing system 2 determines the chest or abdominal region from the position of the face region including the landmark face. The processing system 2 may extract the respiratory information as biometric information from the movement of the estimated chest or abdominal region. However, if the face is tilted with respect to the axis of the body, the actual position of the chest or abdominal region may deviate from the determined position. Therefore, the processing system 2 may determine the periphery of the determined position of the chest or abdominal region as a target region, and extract the respiratory information as biometric information from this target region.

[0089] The target area may be set in a doughnut shape within a certain angle range with the determined landmark position as the center of rotation. A specific example will be described with reference to FIG. 7. The processing system 2 calculates the face area from the face detection result in the angle image with high confidence (FIG. 7(a)), and calculates its center point (FIG. 7(b)). The processing system 2 then draws double concentric circles from the center point, and sets a fan shape of a certain angle from the doughnut area surrounded by the double concentric circles (FIG. 7(c)). The processing system 2 applies this shape to the angle image with high confidence, and sets the range of the fan shape to a range where the chest or abdomen may be present (FIG. 7(d)).

[0090] The position and size of the chest and abdomen of the subject P may be estimated based on facial feature values, which are feature values ​​obtained from an image of the face of the subject P. The facial feature values ​​are, for example, the distance between two facial feature points, and the average brightness, brightness histogram, and area of ​​a triangular area formed by connecting three facial feature points. The facial feature points are a plurality of points set on the face. The facial feature points are, for example, set on the inner corners of the eyes, the outer corners of the eyes, the pupils, the inner corners of the eyebrows, the brow ridge, the inner corners of the eyebrows, the tip of the nose, the wings of the nose, the root of the nose, and / or the corners of the mouth. Alternatively, the position and size of the chest and abdomen of the subject P may be estimated based on skeletal feature values ​​obtained from an image of the whole body. The skeletal feature values ​​are, for example, the average brightness, brightness histogram, and area of ​​an area surrounded by feature points of the shoulders, waist, and neck estimated from an image of the whole body.

[0091] By limiting the target region in this manner, it is possible to reduce the amount of calculation required for measuring biological information.

[0092] [Aspects] It will be appreciated by those skilled in the art that the exemplary embodiments described above are examples of the following aspects.

[0093] (Item 1) A measurement method in one aspect is a method for measuring biometric information from a group of frame images acquired by photographing a person being measured over time, and may include the steps of dividing a target area in each frame image constituting the group of frame images into a plurality of sub-areas, extracting an amount of variation in each of the plurality of sub-areas from the group of frame images, determining sub-areas among the plurality of sub-areas which have similar amounts of variation, and acquiring the biometric information of the person being measured based on an integrated value of the amount of variation in each of the similar sub-areas.

[0094] According to the measurement method described in paragraph 1, a technique for improving the measurement accuracy of biological information in measurement based on data obtained by photographing a subject is provided.

[0095] (Item 2) In the measurement method described in item 1, the frame image may include the subject while sleeping.

[0096] According to the measurement method described in paragraph 2, a technique is provided for improving the measurement accuracy of biological information in measurement based on data obtained by photographing a subject while sleeping.

[0097] (Item 3) In the measurement method described in item 1, the frame image may include the subject at rest.

[0098] According to the measurement method described in paragraph 3, a technique is provided for improving the measurement accuracy of biological information in measurement based on data obtained by photographing a subject at rest.

[0099] (Item 4) In the measurement method according to item 1 or 2, the biological information may be respiratory information.

[0100] According to the measurement method described in paragraph 4, a technique for improving the measurement accuracy of respiratory information based on image data of a subject is provided.

[0101] (Clause 5) In the measurement method described in clause 4, the respiratory information may include at least one of respiratory rate, respiratory time, expiratory time, inhalation time, amplitude, time from exhalation to start of inhalation, and ventilation volume.

[0102] According to the measurement method described in paragraph 5, a technique is provided for improving the measurement accuracy of respiratory information including at least one of respiratory rate, respiratory time, expiratory time, inhalation time, amplitude, time from exhalation to the start of inhalation, and ventilation volume based on photographed data of a subject.

[0103] (Item 6) In the measurement method according to any one of items 1 to 5, the sub-region may be made up of one pixel or a plurality of pixels.

[0104] According to the measurement method described in paragraph 6, in measuring biometric information based on data obtained by photographing a subject, a target area is divided into sub-areas each consisting of one pixel or multiple pixels. By changing the size of the sub-area, it is possible to adjust the measurement accuracy of the biometric information and the amount of calculation required for the process of acquiring the biometric information.

[0105] (Item 7) The measuring method according to items 1 to 6 may further include a step of setting a region of the target in the frame image.

[0106] According to the measurement method described in paragraph 7, a target area is set in the measurement of biometric information based on data obtained by photographing a subject. Setting the target area reduces the amount of calculation required for processing to obtain the biometric information, and improves the measurement accuracy of the biometric information by excluding areas that are not related to obtaining the biometric information.

[0107] (Clause 8) The measurement method described in clauses 1 to 7 may further include a step of determining whether the target area includes a predetermined number or more of sub-areas in which the amount of variation is similar, and a step of setting the target area in response to determining that the target area does not include the predetermined number or more of sub-areas in which the amount of variation is similar.

[0108] According to the measurement method described in paragraph 8, in measuring biological information based on data obtained by photographing a person to be measured, a target area is set, and the set target area is further divided into sub-areas. If the number of similar sub-areas among the sub-areas is less than a predetermined number, the target area is set again, thereby providing a technology for measuring the biological information of the person to be measured even if the person to be measured moves outside the target area.

[0109] (Clause 9) In the measurement method described in Clause 7 or 8, the method may further include a step of estimating the position and size of the chest or abdomen of the person being measured based on the position and orientation of the head of the person being measured, and the setting step may set an area including the chest or abdomen as the target area based on the position and size of the chest or abdomen.

[0110] According to the measurement method described in paragraph 9, a technique is provided for measuring biological information based on data obtained by photographing a subject, with a region including the subject's chest or abdomen as the target region. By using a region including the subject's chest or abdomen as the target region, noise can be reduced and the measurement accuracy of biological information can be improved.

[0111] (Clause 10) In the measurement method described in Clause 9, the estimating step may estimate the position and size of the chest or abdomen of the person being measured based on facial features of the person being measured.

[0112] According to the measurement method described in paragraph 10, in measuring biometric information based on data obtained by photographing a person to be measured, the position and size of the chest are estimated based on the features of the person's face, and a technique is provided for measuring biometric information by setting an area including the chest as a target area. By setting an area including the chest or abdomen of the person to be measured as a target area, noise can be reduced and the measurement accuracy of biometric information can be improved.

[0113] (Item 11) The program according to one aspect may be a program executed by a processor installed in a computer, causing the computer to execute the calculation method according to items 1 to 10.

[0114] According to the program described in paragraph 11, there is provided a technique for improving the measurement accuracy of biological information in measurement based on data obtained by photographing a subject.

[0115] (Item 12) A measurement system in one embodiment includes a camera that photographs a person being measured over time to generate a group of frame images, and a processing device that executes a process of measuring biometric information of the person being measured from the group of frame images generated by the camera, and the processing device executes the steps of dividing a target area in each frame image that constitutes the group of frame images into a plurality of sub-areas, extracting an amount of variation of each of the plurality of sub-areas from the group of frame images, determining sub-areas among the plurality of sub-areas that have similar amounts of variation, and acquiring the biometric information of the person being measured based on an integrated value of the amount of variation of each of the similar sub-areas.

[0116] According to the measurement system described in paragraph 12, a technique for improving the measurement accuracy of biological information in measurement based on data obtained by photographing a subject is provided.

[0117] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present disclosure is indicated by the claims, not by the description of the above-mentioned embodiments, and is intended to include all modifications within the meaning and scope of the claims. In addition, it is intended that each technology in the embodiments can be implemented alone or, if necessary, in combination with other technologies in the embodiments as far as possible. [Explanation of symbols]

[0118] 1 camera, 2 processing system, 21 main body device, 22 terminal, 100 measurement system, 211, 221 processor, 212, 222 input / output port, 213, 223 memory, 214, 224 input device, 215, 225 output device, 2130 data processing program, 2131 setting unit, 2132 division unit, 2133 extraction unit, 2134 determination unit, 2135 accumulation unit, 2136 generation unit, 2137 derivation unit, 2138 output unit.

Claims

1. A measurement method for measuring biological information from a group of frame images obtained by photographing a subject over time, comprising: dividing a target region in each frame image constituting the group of frame images into a plurality of sub-regions; extracting the amount of variation of each of the plurality of sub-regions from the group of frame images; determining sub-regions having similar amounts of variation among the plurality of sub-regions; acquiring the biological information of the subject based on the integrated value of the amount of variation of each of the similar sub-regions.

2. The measurement method according to claim 1, wherein the frame images include the subject during sleep.

3. The measurement method according to claim 1, wherein the frame images include the subject at rest.

4. The measurement method according to claim 1 or claim 2, wherein the biological information is respiratory information.

5. The measurement method according to claim 4, wherein the respiratory information includes at least one of respiratory rate, respiratory time, exhalation time, inhalation time, amplitude, time from the start of exhalation to the start of inhalation, and ventilation volume.

6. The measurement method according to claim 1 or claim 2, wherein the sub-region consists of one pixel or a plurality of pixels.

7. The measurement method according to claim 1 or claim 2, further comprising the step of setting the target region in the frame image.

8. determining whether the target region includes a predetermined number or more of sub-regions having similar amounts of variation; and setting the target region in response to determining that the target region does not include a predetermined number or more of sub-regions having similar amounts of variation. The measurement method according to claim 7.

9. further comprising the step of estimating the position and size of the chest or abdomen of the subject based on the position and orientation of the head of the subject, wherein the setting step sets, as the target region, a region including the chest or abdomen based on the position and size of the chest or abdomen. The measurement method according to claim 7.

10. The measurement method according to claim 9, wherein the estimating step estimates the position and size of the chest or abdomen of the subject based on the feature amount of the face of the subject.

11. A program executed by a processor mounted on a computer, the program causing the computer to execute the measurement method according to claim 1 or claim 2.

12. A camera that captures an object to be measured over time to generate a group of frame images, and A processing device that executes a process of measuring biometric information of the object to be measured from the group of frame images generated by the camera, the measurement system comprising: The processing device includes: Dividing a target region in each frame image constituting the group of frame images into a plurality of sub-regions; Extracting a variation amount of each of the plurality of sub-regions from the group of frame images; Determining sub-regions having similar variation amounts among the plurality of sub-regions; and Obtaining biometric information of the object to be measured based on an integrated value of the variation amounts of each of the similar sub-regions.

Images