Respiratory information estimation device and respiratory information estimation method
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- MITSUBISHI ELECTRIC CORP
- Filing Date
- 2020-06-02
- Publication Date
- 2026-07-10
Smart Images
Figure CN115666382B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to a respiratory information estimation device and a respiratory information estimation method for measuring respiratory information of a subject in a non-contact manner. Background Technology
[0002] In recent years, individuals have been able to easily monitor their health status simply by wearing wearable devices such as smartwatches, even without going to the hospital. However, this requires individuals to wear these wearable devices constantly in their daily lives, which places a significant burden on them.
[0003] Among biological signals, respiration differs from pulse and other signals because it is a signal that can be consciously controlled. For example, it is known that breathing techniques, such as in yoga, can induce parasympathetic dominance and promote relaxation of the mind and body; therefore, the use of respiration is anticipated as an application for controlling health. One method for measuring respiration involves wrapping a band sensor around the chest or abdomen. The expansion and contraction of the band sensor, caused by the movement of the chest or abdomen during respiration, allows for the measurement of respiration based on changes in its inductance. However, as mentioned above, wearing such a sensor for extended periods can be burdensome for the user.
[0004] As a method that does not burden the subject, it is considered to measure respiration in a non-contact manner. As a method for measuring respiration in a non-contact manner, techniques for estimating the subject's respiratory information based on photographic images of the subject are known. The following technique is disclosed: photographing the upper body of the subject in a resting state, detecting the movement of the upper body caused by breathing, and estimating respiratory information (for example, see Patent Document 1).
[0005] In the above technique, the measurement is based on the premise that the subject is in a quiet state. Based on photographic images over a certain period of time, the breathing direction axis, which represents the specific direction of upper body movement caused by breathing, is determined. Based on this breathing direction axis, the body motion component is removed, thereby correcting the measurement data and calculating the breathing waveform.
[0006] Existing technical documents
[0007] Patent documents
[0008] Patent Document 1: US Patent No. 9861302 Summary of the Invention
[0009] The problem the invention aims to solve
[0010] However, the existing technology has the following problems: if the body is at rest, the correct breathing direction axis can be calculated, but if there is body movement that is not caused by breathing during a certain period of time, the correct breathing direction axis cannot be calculated, and the accuracy decreases.
[0011] For example, when conducting measurements while riding in a vehicle, it is easy to anticipate bodily movements that are not caused by breathing, such as the driver's upper body moving due to vehicle vibrations. Therefore, it is necessary to estimate breathing with high accuracy even when the subject's upper body is moving during the measurement.
[0012] Therefore, the purpose of this disclosure is to be able to estimate respiratory information with high accuracy even when there is no physical movement caused by breathing during a certain period of time.
[0013] means for solving problems
[0014] The respiratory information estimation device disclosed herein includes: a specific region setting unit that sets a specific region and a motion vector calculation point based on an image including the upper body of a subject, the motion vector calculation point being a reference point for the motion vector; a respiratory reference axis calculation unit that calculates a respiratory reference axis based on a respiratory center point and the motion vector calculation point, the respiratory center point being the starting point of the motion caused by breathing; a motion vector calculation unit that calculates a motion vector based on the amount of movement of the motion vector calculation point in each of the continuously captured images; a respiratory signal calculation unit that calculates a respiratory signal based on the component of the motion vector in the direction of the respiratory reference axis; and a respiratory information calculation unit that calculates a respiratory information estimation result based on the respiratory signal, the respiratory information estimation result being a result of estimating the body motion caused by breathing.
[0015] The effects of the invention
[0016] This disclosure uses a respiratory reference axis calculated based on a specific region of the subject to serve as the baseline for breathing. Thus, even when there is body movement that is not caused by breathing during a certain period of time, the subject's respiratory information can be estimated with high accuracy. Attached Figure Description
[0017] Figure 1 This is a block diagram that outlines the structure of the respiratory information estimation device in Embodiment 1.
[0018] Figure 2 This is an explanatory diagram showing the results of the facial detection.
[0019] Figure 3 It is an explanatory diagram showing a specific area.
[0020] Figure 4 This is an explanatory diagram showing the points where motion vectors are calculated.
[0021] Figure 5 This is a schematic diagram illustrating an example of chest movement caused by breathing.
[0022] Figure 6 This is a schematic diagram illustrating an example of a method for calculating the respiratory center point.
[0023] Figure 7 This is an explanatory diagram showing the breathing reference axis.
[0024] Figure 8 This is a schematic diagram illustrating a method for calculating the projected motion vector based on a breathing reference axis and the motion vector.
[0025] Figure 9 This is a schematic diagram illustrating an example of a respiratory signal output at predetermined intervals.
[0026] Figure 10 This is a flowchart illustrating the operation of the respiratory information estimation device in Embodiment 1.
[0027] Figure 11 This is a block diagram that outlines the structure of the respiratory information estimation device in Embodiment 2.
[0028] Figure 12 This is an explanatory diagram showing the motion vectors.
[0029] Figure 13 This is an explanatory diagram showing the corrected motion vector.
[0030] Figure 14 This is a schematic diagram illustrating the method for calculating the angular difference between the breathing reference axis and the breathing direction axis in Embodiment 2, and for correcting the breathing direction axis using a predetermined threshold.
[0031] Figure 15 This is a flowchart illustrating the operation of the respiratory information estimation device in Embodiment 2.
[0032] Figure 16 This is a block diagram that outlines the structure of the respiratory information estimation device in Embodiment 3.
[0033] Figure 17 This is a block diagram that outlines the structure of the motion vector calculation unit in Embodiment 3.
[0034] Figure 18 This is a schematic diagram illustrating the method for removing non-respiratory components from motion vectors in Embodiment 3.
[0035] Figure 19 This is a diagram illustrating an example of the hardware structure of the respiratory information estimation device in Embodiment 3.
[0036] Figure 20 This is a diagram illustrating another example of the hardware structure of the respiratory information estimation device in Embodiment 3. Detailed Implementation
[0037] Implementation method 1.
[0038] Figure 1 This is a block diagram that schematically illustrates the structure of the respiratory information estimation device 100a according to Embodiment 1. The respiratory information estimation device 100a is an apparatus that performs the respiratory information estimation method of Embodiment 1. Figure 1 As shown, the respiratory information estimation device 100a is connected to the image acquisition unit 200 and includes a specific region setting unit 110, a motion vector calculation unit 120, a respiratory reference axis calculation unit 130, a respiratory signal calculation unit 140, and a respiratory information calculation unit 150.
[0039] First, an overview of the respiratory information estimation device 100a will be described. The respiratory information estimation device 100a accepts image data of a total of N frames captured by the image acquisition unit 200 at a predetermined frame rate r as input. Here, N is an integer of 2 or higher. When the frame number assigned to each frame is set to i (i = 1, 2, ..., N), the frame provided at the next timing after frame (i) is frame (i+1). Furthermore, at a predetermined time t1, the respiratory information estimation device 100a acquires image data from the N frames of image data and outputs a respiratory information estimation result B(k) obtained by estimating body motion caused by breathing. Here, parameter k is the output number updated after the predetermined time t1. For example, the respiratory information estimation result estimated and output by the respiratory information estimation device 100a at the next timing after the respiratory information estimation result B(k) is respiratory information estimation result B(k+1).
[0040] Here, i, representing the frame number, and k, representing the output number, are integers greater than or equal to 1. The frame rate r can be set to 30 frames per second, taking into account factors such as the processing speed of the microcomputer or the image size. Furthermore, the number of frames N used to output the respiratory information estimation result B(k) is any value. Here, a predetermined time t1 is set such that t1 × r equals exactly N, thus enabling the processing of all image data available within time t1. For example, the number of frames N used to output the respiratory information estimation result B(k) is equivalent to the number of frames in 10 seconds, which, according to the example above, becomes 300 frames. Typically, there are approximately 3 breaths in 10 seconds; therefore, considering the time from data accumulation to prompting the user with the respiratory information estimation result B(k), t1 is set to 10 seconds, and N is set to 300. In this case, for example, parameter k is updated every second after the predetermined time t1 is reached. The respiratory information estimation result B(k+1) uses data from 300 frames, from frame 31 to frame 330. Furthermore, the number of people (objects) contained in the image data can be one or more. For simplicity, the following explanation focuses on the case where the number of people in the image data is one.
[0041] In the image data, only the upper body of the subject needs to be displayed. The image can be a grayscale image, an RGB (Red, Green, Blue) image, an IR (infrared) image, or a distance image containing distance information to the subject. For simplicity, the grayscale image will be described here. Furthermore, the image data can be two-dimensional or three-dimensional. For simplicity, the two-dimensional data will be described below. In this case, the image data is configured, for example, with Nx pixels horizontally and Ny pixels vertically. Nx and Ny are integers greater than or equal to 1. The image acquisition unit 200 can be, for example, a CCD (Charge Coupled Device) and positioned to acquire the upper body of the subject. For example, it can be configured to capture the image from a direction directly facing the subject's body when the subject is facing forward. Alternatively, it can be configured to capture the image from the subject's side when the subject is facing forward.
[0042] The components constituting the respiratory information estimation device 100a will now be described. The specific region setting unit 110 sets a specific region Si containing the upper body of the subject based on the frame (i) contained in the image data input from the image acquisition unit 200.
[0043] In Implementation 1, the specific region Si is the region corresponding to the subject's chest. However, the specific region Si can also be a region other than the subject's chest. For example, it could be a region that detects movement caused by breathing, such as the shoulder or abdomen. Furthermore, the specific region Si can be multiple regions. Additionally, the specific region Si can consist of one or more points.
[0044] Furthermore, the method for setting the specific region Si can be based on facial detection results, or it can be based on skeletal detection results containing skeletal information such as the human shoulder or elbow. Alternatively, the method for setting the specific region Si can be by extracting the AAM (Active Appearance Models) of the specific region Si through fitting a model to the subject. The case of setting the specific region Si based on facial detection results will be explained later. Furthermore, the specific region Si is represented by the coordinate system of frame (i), with the upper left point of frame (i) set as the origin, the rightward direction in frame (i) set as the positive x-axis, and the downward direction in frame (i) set as the positive y-axis.
[0045] Figure 2 This is an explanatory diagram showing the results of the facial detection. Figure 3 This is an explanatory diagram showing a specific area. It was acquired in the image acquisition section. Figure 2In the image of such a frame (i), facial regions are detected based on the upper body of the subject. In facial region detection, methods such as Haar-Like features are used. The specific region setting unit 110 detects a quadrilateral region as a facial region. For example, the quadrilateral region is represented by its upper left point Pu and its lower right point Pd as the facial region detection result. Next, the specific region setting unit 110 calculates the height Hf and width Wf of the subject's face based on the upper left point Pu and the lower right point Pd of the quadrilateral region. Furthermore, the specific region setting unit 110 calculates the height Hs, width Ws, and position of a specific region Si based on the calculated face height Hf and face width Wf. For example, if the face height Hf and face width Wf are large, the specific region Si is large; if the face height Hf and face width Wf are small, the specific region Si is small. For example, as... Figure 3 Therefore, starting from the lower right point Pd of the quadrilateral region, considering the length of the neck, the position below one-quarter of the height Hf of the face is set as the position of the specific region Si. For example, the height Hs of the specific region Si is the same as the height Hf of the subject's face, and the width Ws of the specific region Si is twice the width Wf of the subject's face. Based on the height Hs and the width Ws of the specific region Si, the specific region setting unit 110 sets four vertices (P1, P2, P3, P4) surrounding the specific region Si.
[0046] If, during the period from the image in frame (i) to the image in frame (i+1), the vertex of a specific region Si moves by a distance exceeding a set threshold, or if the vertex of a specific region Si moves outside the frame, the specific region setting unit 110 may also reset the specific region Si. If a feature point of an object captured at coordinates (x, y) (x = 1, 2, ..., Nx; y = 1, 2, ..., Ny) in frame (i) is captured at other coordinate positions in frame (i+1) that are separated by a threshold Td, the specific region setting unit 110 resets the specific region Si. Furthermore, the pixel value at coordinates (x, y) in frame (i) may change in the next frame (i+1). In this case, if the difference in pixel values is within the threshold Tv, it can be considered that the pixel was captured at the same coordinate position, and the specific region Si can be reset accordingly.
[0047] Figure 4 This is an explanatory diagram showing the points where motion vectors are calculated. Figure 5This is an explanatory diagram showing the breathing reference axis. The specific region setting unit 110 sets M motion vector calculation points Pf(m) within the specific region Si. Here, m represents a parameter taking an integer value from 1 to M. A motion vector calculation point is a point that serves as the reference for the motion vector. The motion vector calculation points Pf(m) corresponding to the parameter m exist as Pf(1), Pf(2), ..., Pf(M). For example, as... Figure 4 The motion vector calculation points Pf(m) configured in that way, with each row and column of 3, are equally spaced within a specific region Si. Figure 4 In the example, the value of M is 12. At the motion vector calculation point Pf(m), the motion vector fi(m) in frame (i) is calculated. Then, the projected motion vector ft(m) is calculated based on the motion vector fi(m). The explanation of calculating the projected motion vector ft(m) based on the motion vector fi(m) will be described below. The motion vector fi(m) corresponding to the frame number i (i = 1, 2, ..., N) exists as f1(m), f2(m), ..., fN(m). In addition, the motion vector fi(m) corresponding to the value m exists as fi(1), fi(2), ..., fi(M). Hereinafter, the explanation will be based on the value fi(m). Figure 4 The interval g is used to set the motion vector calculation point Pf(m). Furthermore, the motion vector calculation point Pf(m) can be set randomly, not at equal intervals. The number of motion vector calculation points can be set to multiple or a single point. Alternatively, the motion vector calculation unit 120 can set the motion vector calculation point Pf(m).
[0048] The position of the motion vector calculation point Pf(m) is set based on the four vertices (P1, P2, P3, P4) surrounding the specific region Si, the number of motion vector calculation points M, and the pre-set interval g between each motion vector calculation point Pf(m). A larger number of motion vector calculation points M allows for more precise motion measurement, but it also easily includes noise such as body movements not caused by breathing. A narrower interval g allows for more precise motion measurement, but it also easily includes noise such as body movements not caused by breathing.
[0049] The motion vector calculation unit 120 receives frame (i), a specific region Si corresponding to frame (i), and each motion vector calculation point Pf(m) contained in the specific region Si, calculates the motion vector fi(m) at the motion vector calculation point Pf(m), and provides it to the respiratory signal calculation unit 140. Hereinafter, optical flow will be used for explanation.
[0050] The motion vector calculation unit 120 calculates the motion vector fi(m) of the motion vector calculation point Pf(m) in frame (i) based on the amount of movement from the coordinate position of the motion vector calculation point Pf(m) in the previous frame (i-1) to the coordinate position of the motion vector calculation point Pf(m) in frame (i). For all M motion vector calculation points Pf(m), N frame-level motion vectors fi(m) are calculated. Here, the method for calculating the motion vector fi(m) is not limited to optical flow. For example, HOG features (Histograms of Oriented Gradients) can also be used.
[0051] Figure 5 This is a schematic diagram illustrating an example of chest movement caused by respiration. Generally, the motion vector fi(m) is calculated during inhalation in the direction of widening from the center of the chest, and during exhalation in the direction of contraction towards the center of the chest. Figure 5 (a) shows the movement of the chest during inhalation. Figure 5 (b) shows the movement of the chest during exhalation.
[0052] For example, when a specific region Si including the chest and abdomen is set, the specific region setting unit 110 compares the motion vectors corresponding to the positions of the chest and abdomen, thereby determining whether it is thoracic breathing or abdominal breathing. At this time, at least two motion vector calculation points Pf(m) are set at different positions where the motion vectors of the chest and abdomen appear.
[0053] The breathing reference axis calculation unit 130 calculates the breathing center point Pc, which serves as the starting point of the motion caused by breathing, based on frame (i). Furthermore, the breathing reference axis calculation unit 130 sets the direction toward the motion vector calculation point Pf(m) to positive, calculates the breathing reference axis Rr(m) as a straight line from the breathing center point Pc toward the motion vector calculation point Pf(m), and provides it to the breathing signal calculation unit 140. The breathing reference axis Rr(m) refers to the axis set at each motion vector calculation point Pf(m) in the direction of motion of the motion vector calculation point Pf(m) generated by breathing. For example, when shooting from a direction directly opposite the subject's body while the subject is facing forward, it is desirable to shoot in a manner where the breathing reference axis Rr(m) is calculated symmetrically from the center line of the body.
[0054] Figure 6This is a schematic diagram illustrating an example of a method for calculating the breathing center point. The breathing reference axis calculation unit 130 calculates the breathing center point Pc as follows: A curve C is drawn that approximates the outline of the subject's shoulder. Normals N1 and N2 are obtained at the points where curve C overlaps with the ends of the shoulders. Normal N1 is the normal at the point where it overlaps with the ends of the left shoulder, and normal N2 is the normal at the point where it overlaps with the ends of the right shoulder. The intersection of normals N1 and N2 is set as the breathing center point Pc. Furthermore, consider cases where the shoulder outline is difficult to visually confirm, such as when it is hidden behind a seatbelt. In this case, the intersection of the center line L and the normal N1 or normal N2 obtained from the shoulder on that side, where it can be visually confirmed, can also be set as the breathing center point Pc. This center line L is a line drawn vertically from the center of the upper body in the horizontal direction. If the feature points of an object captured at coordinates (x, y) (x = 1, 2, ..., Nx; y = 1, 2, ..., Ny) in frame (i) are captured at other coordinate positions above the separation threshold Td in frame (i+1), the specific region setting unit 110 resets the specific region Si. This is because the curve C, which overlaps with the shoulder contour line, is also affected by body movements not caused by breathing. Furthermore, the Hough transform can be used to detect a straight line in the direction of the center of movement toward the lungs, and the breathing center point Pc can be calculated based on this intersection.
[0055] Figure 7 The method for calculating the respiratory reference axis Rr(m) is illustrated in summary. The respiratory reference axis calculation unit 130 sets the direction from the respiratory center point Pc toward the calculated motion vector calculation point Pf(m) to positive, and calculates the straight line connecting the respiratory center point Pc and each motion vector calculation point Pf(m) as the respiratory reference axis Rr(m). The respiratory center point Pc is a reference point located near the center of the upper body. Furthermore, it is shown as a dashed line for easier observation. Figure 7 The straight line connecting the breathing center point Pc to the calculation point Pf(m) of each motion vector is the breathing reference axis Rr(m).
[0056] Figure 8 This is a schematic diagram illustrating a method for calculating the projected motion vector based on a breathing reference axis and the motion vector. Figure 8The result is shown as m=12. The motion vector calculation unit 120 calculates the completed projection motion vector ft(m) based on the calculated motion vector fi(m) and the breathing reference axis Rr(m) calculated by the breathing reference axis calculation unit 130. The breathing signal calculation unit 140 receives the completed projection motion vector ft(m) from the motion vector calculation unit 120 and receives the breathing reference axis Rr(m) from the breathing reference axis calculation unit 130. The breathing signal calculation unit 140 calculates the breathing signal Ct(k) based on the completed projection motion vector ft(m) and the breathing reference axis Rr(m). In other words, the breathing signal Ct(k) is calculated based on the component of the motion vector in the direction of the breathing reference axis. By projecting the motion vector fi(m) at each motion vector calculation point Pf(m) onto the corresponding breathing reference axis Rr(m), the sum of the calculated completed projection motion vectors ft(m) is calculated as in Equation 1, thereby obtaining the breathing signal component F(i) in frame (i).
[0057] [Formula 1]
[0058]
[0059] (Equation 1)
[0060] The breathing signal calculation unit 140 calculates the frame breathing signal Ct(i) representing the breathing of frame (i) as Ct(i) = Ct(i-1) + F(i). Additionally, Ct(0) is set to 0. Furthermore, the breathing signal calculation unit 140 provides the breathing signal Ct(k) representing the migration of N frames from the frame breathing signal Ct(i) to the breathing information calculation unit 150. Alternatively, the breathing signal calculation unit 140 can also calculate the projection completion motion vector ft(m).
[0061] Figure 9 This is a schematic diagram illustrating an example of a respiratory signal Ct(k) output at predetermined intervals. Figure 9 The horizontal axis in the figure represents time t [s]. Figure 9 The vertical axis represents the amount of motion [au] (au: arbitrary unit). Time t [s] is a time within a predetermined time range t1. For example, the N-frame respiratory signal Ct(k) at time t becomes Figure 9 Such waveform signals.
[0062] The respiratory information calculation unit 150 applies a bandpass filter (e.g., 0.1 Hz to 0.5 Hz) that passes through the frequency band corresponding to breathing to the respiratory signal Ct(k) provided by the respiratory signal calculation unit 140. The respiratory information estimation result B(k) is calculated based on the filtered signal and output.
[0063] The respiratory information estimation result B(k) is, for example, a respiratory rate (bpm: breaths per minute) signal calculated based on the average of the peak intervals of the respiratory signal Ct(k). Figure 9 When the average peak interval of the respiratory signal Ct(k) is set to 3 seconds, the respiratory rate becomes 20 bpm. Respiratory information is not limited to the respiratory rate; it can also include respiratory time, expiratory time, inspiratory time, amplitude, the time from expiration to the start of inhalation (pause time), or ventilation volume. Furthermore, the frequency band corresponding to respiration varies among individuals based on age or gender; therefore, a bandpass filter corresponding to these individual differences is used.
[0064] The following describes the actions performed by the respiratory information estimation device 100a. Figure 10 This is a flowchart illustrating the operation of the respiratory information estimation device 100a according to Embodiment 1. Figure 10 The flowchart shown includes an image acquisition step S10, a specific region setting step S20, a motion vector calculation step S30, a breathing reference axis calculation step S40, a breathing signal calculation step S50, and a breathing information calculation step S60.
[0065] In the image acquisition step S10, image data containing the upper body of the subject is acquired.
[0066] In the specific region setting step S20, the specific region Si of the object and the motion vector calculation point Pf(m) are set according to the image shown by the obtained image data.
[0067] In the motion vector calculation step S30, the motion vector fi(m) is calculated, which represents the motion at M motion vector calculation points Pf(m) set within a specific region Si. In the breathing reference axis calculation step S40, the breathing reference axis is calculated based on the starting point of the motion caused by breathing, i.e., the breathing center point, and the motion vector calculation points.
[0068] In the breathing signal calculation step S50, the breathing signal Ct(k) representing the migration of the N-frame amount of the frame breathing signal Ct(i) is calculated. The frame breathing signal Ct(i) is obtained by taking the sum of the projected motion vectors ft(m) obtained by projecting M motion vectors fi(m) onto the breathing reference axis Rr(m).
[0069] In the respiratory information calculation step S60, a bandpass filter that transmits through the frequency band corresponding to respiration is applied to the respiratory signal Ct(k). In the respiratory information calculation step S60, the respiratory information estimation result B(k) is calculated based on the filtered signal.
[0070] As described above, according to Embodiment 1, even in the presence of body movements not caused by breathing, the estimation result of the subject's breathing information can be calculated with high accuracy.
[0071] In Implementation 1, by setting a breathing reference axis Rr(m), when the motion vector components are projected onto the breathing reference axis, the motion vectors generated by body movement that are not along the breathing direction axis become smaller values during projection. Therefore, the influence of body movement not caused by breathing can be reduced, and the breathing information estimation results can be calculated with high accuracy.
[0072] Implementation method 2.
[0073] Figure 11 This is a block diagram that schematically shows the structure of the respiratory information estimation device 100b in Embodiment 2. The respiratory information estimation device 100b is connected to the image acquisition unit 200 and includes a specific region setting unit 110, a motion vector calculation unit 220, a respiratory reference axis calculation unit 130, a respiratory signal calculation unit 140, a respiratory information calculation unit 150, a respiratory direction axis calculation unit 310, a respiratory direction axis correction unit 320, and a corrected motion vector calculation unit 230. The specific region setting unit 110, the respiratory reference axis calculation unit 130, the respiratory signal calculation unit 140, and the respiratory information calculation unit 150 of the respiratory information estimation device 100b in Embodiment 2 are the same as those of the respiratory information estimation device 100a in Embodiment 1.
[0074] The motion vector calculation unit 220 provides the motion vector fi(m) to the respiratory signal calculation unit 140 and the respiratory direction axis calculation unit 310. The difference between the motion vector calculation unit 220 and the motion vector calculation unit 120 is that the motion vector calculation unit 120 in Embodiment 1 only provides the motion vector fi(m) to the respiratory signal calculation unit 140. Furthermore, the respiratory information estimation device 100b executes the respiratory information estimation method in Embodiment 2.
[0075] The breathing direction axis calculation unit 310 receives N frames of motion vectors fi(m) from the motion vector calculation unit 220. At each motion vector calculation point Pf(m), for example, using 300 frames of motion vectors fi(m), the breathing direction axis Rd(m) is calculated using principal component analysis. Specifically, the variance of the 300 frames of motion vectors fi(m) is calculated for each motion vector calculation point Pf(m), and the vector with the largest variance is calculated as the breathing direction axis Rd(m) at that motion vector calculation point Pf(m). The breathing direction axis calculation unit 310 provides the breathing direction axis Rd(m) to the breathing direction axis correction unit 320. For example, when the time t1 is set to 10 seconds and the camera frame rate r is set to 30 frames per second, 300 frames of motion vectors fi(m) are used to calculate the breathing direction axis Rd(m).
[0076] Figure 12 This is an explanatory diagram showing the motion vector fi(m). Figure 13 This is an explanatory diagram showing the corrected motion vector ff(m). Figure 12 and Figure 13 In the example, the value of M is 12.
[0077] The breathing direction axis correction unit 320 corrects for deviations in the breathing direction axis caused by individual differences based on the angular difference between the breathing reference axis Rr(m) provided by the breathing reference axis calculation unit 130 and the breathing direction axis Rd(m) provided by the breathing direction axis calculation unit 310, and calculates the corrected breathing direction axis. Furthermore, the breathing direction axis correction unit 320 calculates the corrected breathing direction axis Rf(m) and provides it to the corrected motion vector calculation unit 230. The corrected motion vector calculation unit 230 calculates the corrected motion vector ff(m) along the direction of the corrected breathing direction axis Rf(m) and provides it to the breathing signal calculation unit 140.
[0078] Figure 14 (a) to Figure 14 (c) is a schematic diagram illustrating the following method: calculating the angular difference between the respiratory reference axis Rr(m) and the respiratory direction axis Rd(m), and using a predetermined threshold to correct the respiratory direction axis Rd(m). Figure 14 As shown in (a), if the angle difference θdiff between the breathing reference axis Rr(m) and the breathing direction axis Rd(m) at a certain motion vector calculation point Pf(m) is within the threshold θth, then the breathing direction axis Rd(m) is directly used as the corrected breathing direction axis Rf(m) for processing.
[0079] like Figure 14As shown in (b), when the angle difference θdiff is greater than the threshold θth, and the angle obtained by subtracting the angle difference θdiff from 180° is greater than the threshold θth, the breathing direction axis Rd(m) is corrected to the breathing reference axis Rr(m), and processed as the corrected breathing direction axis Rf(m).
[0080] like Figure 14 As shown in (c), when the angle difference θdiff is larger than the threshold θth, and the angle obtained by subtracting the angle difference θdiff from 180° is smaller than the threshold θth, the direction of the breathing direction axis Rd(m) is reversed by 180° and processed as the corrected breathing direction axis Rf(m).
[0081] exist Figure 14 In case (b), it is explained that the breathing direction axis Rd(m) is corrected to the breathing reference axis Rr(m), but it is also possible to calculate the point Pf(m) without using the motion vector obtained by subtracting the angle difference θdiff from 180°, which is larger than the threshold θth, and whose angle is larger than the threshold θth. Furthermore, in Figure 14 In (c), if the motion vector calculation point Pf(m) is not a single point (i.e., M = 1), then the motion vector calculation point Pf(m) obtained by subtracting the angle difference θdiff from 180° and having an angle smaller than the threshold θth can be omitted.
[0082] As a method for calculating the angle difference θdiff, cosine similarity can be used. Cosine similarity can represent how close the angles of vectors are to each other. Therefore, when the cosine similarity is close to 1, the two vectors are close to the same direction, and when the cosine similarity is close to -1, the two vectors are close to opposite directions. The threshold θth is expected to be within 45°, for example.
[0083] Figure 15 This is a flowchart illustrating the operation of the respiratory information estimation device 100b according to Embodiment 2. Figure 15 The flowchart shown includes the following steps: image acquisition step S10, specific region setting step S20, motion vector calculation step S30, breathing reference axis calculation step S40, breathing direction axis calculation step S41, breathing direction axis correction step S42, correction completed motion vector calculation step S43, breathing signal calculation step S50, and breathing information calculation step S60.
[0084] The image acquisition step S10, specific region setting step S20, motion vector calculation step S30, respiratory reference axis calculation step S40, respiratory signal calculation step S50, and respiratory information calculation step S60 of the respiratory information estimation device 100b in Embodiment 2 are the same as those of the respiratory information estimation device 100a in Embodiment 1.
[0085] In the breathing direction axis calculation step S41, the breathing direction axis Rd(m) is calculated based on the motion vector fi(m) of multiple consecutive frames.
[0086] In the breathing direction axis correction step S42, the breathing direction axis Rd(m) is corrected according to the relationship between the breathing reference axis Rr(m) and the breathing direction axis Rd(m), and the corrected breathing direction axis Rf(m) is calculated.
[0087] In step S43, the corrected motion vector ff(m) is calculated along the corrected breathing direction axis Rf(m).
[0088] As described above, the respiratory information estimation device 100b according to Embodiment 2 can reduce the decrease in accuracy caused by the deviation of the respiratory reference axis Rr(m) due to individual differences by calculating the respiratory direction axis Rd(m), and can reduce the influence of body movement not caused by breathing, and can calculate the respiratory information estimation result with higher accuracy.
[0089] Furthermore, in the above explanation, if the angle difference θdiff between the breathing direction axis Rd(m) and the breathing reference axis Rr(m) at the motion vector calculation point Pf(m) is within the threshold θth, then the breathing direction axis Rd(m) is directly treated as the corrected breathing direction axis Rf(m). If the angle difference θdiff is above the threshold θth, then the corrected breathing reference axis Rr(m) is used as the corrected breathing direction axis Rf(m). However, it is also possible to use only the projected motion vector of the motion vector calculation point Pf(m) where the angle difference θdiff is within the threshold θth to calculate the respiratory signal component F(i).
[0090] Implementation method 3.
[0091] Figure 16This is a block diagram that schematically illustrates the structure of the respiratory information estimation device 100c according to Embodiment 3. The respiratory information estimation device 100c is identical to the specific region setting unit 110, respiratory reference axis calculation unit 130, respiratory signal calculation unit 140, and respiratory information calculation unit 150 of the respiratory information estimation device 100a according to Embodiment 1. However, the motion vector calculation unit 420 has a different structure than the motion vector calculation unit 120 of Embodiment 1. Furthermore, the respiratory information estimation device 100c executes the respiratory information estimation method as the information processing method of Embodiment 3.
[0092] Figure 17 This is a block diagram that schematically shows the structure of the motion vector calculation unit 420 in Embodiment 3. The motion vector calculation unit 420 includes a motion component calculation unit 420a, a non-respiratory component calculation unit 420b, and a respiratory component calculation unit 420c.
[0093] The motion component calculation unit 420a, like the motion vector calculation unit 120 in Embodiment 1, receives the frame (i), the specific region Si, and the motion vector calculation point Pf(m), calculates the motion vector fi(m) at the motion vector calculation point Pf(m), and provides it to the respiratory component calculation unit 420c.
[0094] The non-breathing component calculation unit 420b calculates the non-breathing component di(q) based on the region outside the specific region Si. q is an integer greater than or equal to 1 representing the point from which the non-breathing component is calculated, and is arbitrarily set. The total number of points from which the non-breathing component is calculated is u. The non-breathing component calculation unit 420b provides the average value d_avei (=Σdi(q) / u) of the non-breathing component di(q) calculated from u points to the breathing component calculation unit 420c. Furthermore, in the above description, the average value d_avei of the non-breathing component is calculated based on multiple points di(q), but it can also be calculated based on a single point.
[0095] The respiratory component calculation unit 420c calculates the respiratory component fs(m) (=fi(m)-d_avei) based on the motion vector fi(m) provided by the motion component calculation unit 420a and the average value d_avei of the non-respiratory component provided by the non-respiratory component calculation unit 420b.
[0096] Figure 18 This is a schematic diagram illustrating the method for removing the average value d_avei of non-respiratory components from the motion vector fi(m). Regarding... Figure 18 (a) shows the desired respiratory component fs(m), calculate Figure 18 (b) the average value of non-respiratory components such as body movement, d_avei, and as Figure 18(c) The operation is performed on the motion vector fi(m) to remove the average value d_avei of the non-respiratory component. For example, the non-respiratory component di(q) is calculated in the region enclosed by the dashed box where no motion caused by breathing occurs, and the average value d_avei of the non-respiratory component is calculated. Thus, the respiratory component fs(m) is calculated by obtaining the difference between the average value d_avei of the non-respiratory component and the motion vector fi(m) of the specific region Si, and is provided to the respiratory signal calculation unit 140.
[0097] As described above, the respiratory information estimation device 100c according to Embodiment 3 can calculate the average value d_avei of the non-respiratory components and obtain the difference from the motion vector fi(m) to calculate the respiratory components, thus enabling robust respiratory information estimation for motion. Therefore, Embodiment 3 can calculate the respiratory information estimation results with higher accuracy.
[0098] In a respiratory information estimation device configured in this way, even when it is difficult to identify the subject's upper body or other parts in a photographic image, the subject can be identified, and the respiratory information estimation result can be calculated using the respiratory center point or other methods when the subject was previously identified.
[0099] Furthermore, in the above description, the image data used to output the respiratory information estimation result B(k) is described as image data consisting of a total of N frames captured at a predetermined frame rate r. However, it is not limited to this. For example, N1 consecutive images out of the N frames can also be used as image data for outputting the respiratory information estimation result B(k).
[0100] Figure 19 This is a diagram illustrating an example of the hardware structure of a respiratory information estimation device 100 (100a, 100b, 100c). Figure 20 This is another example of the hardware structure of the respiratory information estimation device 100.
[0101] The respiratory information estimation device 100 comprises, for example, at least one processor 101a and a memory 101b. The processor 101a is, for example, a Central Processing Unit (CPU) that executes the program stored in the memory 101b. In this case, the function of the respiratory information estimation device 100 is implemented through software, firmware, or a combination of both. The software and firmware are stored in the memory 101b as programs. Thus, the program for implementing the function of the respiratory information estimation device 100 (e.g., the respiratory information estimation method described in this embodiment) is executed by a computer.
[0102] Memory 101b is a computer-readable recording medium, such as volatile memory (RAM) and non-volatile memory (ROM), or a combination of volatile and non-volatile memory.
[0103] The respiratory information estimation device 100 can also be configured as a dedicated hardware processing circuit 101c, such as a single circuit or a composite circuit. In this case, the function of the respiratory information estimation device 100 is implemented by the processing circuit 101c.
[0104] The embodiments of this disclosure have been described above, but this disclosure is not limited to these embodiments.
[0105] Explanation of reference numerals in the attached figures
[0106] 200 Image acquisition unit, 100a, 100b, 100c Respiratory information estimation devices, 110 Specific area setting unit, 120, 320 Motion vector calculation unit, 130 Respiratory reference axis calculation unit, 140 Respiratory signal calculation unit, 150 Respiratory information calculation unit, 420a Motion component calculation unit, 420b Non-respiratory component calculation unit, 420c Respiratory component calculation unit.
Claims
1. A respiratory information estimation device, characterized in that, The respiratory information estimation device includes: The specific area setting unit sets a specific area and motion vector calculation point based on an image containing the upper body of the subject. The motion vector calculation point is the reference point for the motion vector. The breathing reference axis calculation unit detects two straight lines in the direction of the center of motion toward the lungs, calculates the breathing center point based on the intersection of the two straight lines, and calculates the breathing reference axis as the straight line from the breathing center point toward the motion vector calculation point. The breathing center point is the starting point of the motion caused by breathing. The motion vector calculation unit calculates motion vectors based on the amount of movement of the motion vector calculation point in each of the continuously captured images. The respiratory signal calculation unit calculates the respiratory signal based on the component of the motion vector in the direction of the respiratory reference axis; as well as The respiratory information calculation unit calculates a respiratory information estimation result based on the respiratory signal, which is a result of estimating the body movement caused by breathing.
2. The respiratory information estimation device according to claim 1, characterized in that, The breathing reference axis calculation unit calculates the following intersection point as the breathing center point and calculates the vector from the breathing center point toward the motion vector calculation point as the breathing reference axis. The intersection point is the intersection of the following two lines: the normal of a curve that approximates the outline of the subject's shoulder at the part overlapping with the end of the right shoulder, and the normal of the curve at the part overlapping with the end of the left shoulder.
3. The respiratory information estimation device according to claim 1, characterized in that, The breathing reference axis calculation unit calculates the following intersection point as the breathing center point and calculates the vector from the breathing center point toward the motion vector calculation point as the breathing reference axis. The intersection point is the intersection of the following two lines: the normal line of a curve that approximates the outline of the subject's shoulder at the part that overlaps with the end of the right or left shoulder, and the center line drawn vertically at the center of the subject's upper body in the horizontal direction.
4. The respiratory information estimation device according to any one of claims 1 to 3, characterized in that, The specific region setting unit sets multiple motion vector calculation points in the specific region. The breathing reference axis calculation unit calculates the breathing reference axis at each of the motion vector calculation points. The motion vector calculation unit calculates the projected motion vector, which is the component along the direction of the breathing reference axis, based on the motion vectors calculated at each of the motion vector calculation points. The breathing signal calculation unit calculates the sum of the motion vectors after projection as a frame breathing signal, and calculates the breathing signal representing the migration of the frame breathing signal of each image in multiple images.
5. The respiratory information estimation device according to any one of claims 1 to 3, characterized in that, The respiratory information calculation unit applies a bandpass filter corresponding to the frequency of breathing to the respiratory signal and calculates the respiratory information estimation result.
6. The respiratory information estimation device according to claim 4, characterized in that, The respiratory information calculation unit applies a bandpass filter corresponding to the frequency of breathing to the respiratory signal and calculates the respiratory information estimation result.
7. The respiratory information estimation device according to any one of claims 1 to 3 and 6, characterized in that, The motion vector calculation unit calculates the motion components in regions where no motion is generated by breathing as non-breathing components, and calculates the breathing components by taking the difference between the average value of the motion components in the specific region and the average value of the non-breathing components.
8. The respiratory information estimation device according to claim 4, characterized in that, The motion vector calculation unit calculates the motion components in regions where no motion is generated by breathing as non-breathing components, and calculates the breathing components by taking the difference between the average value of the motion components in the specific region and the average value of the non-breathing components.
9. The respiratory information estimation device according to claim 5, characterized in that, The motion vector calculation unit calculates the motion components in regions where no motion is generated by breathing as non-breathing components, and calculates the breathing components by taking the difference between the average value of the motion components in the specific region and the average value of the non-breathing components.
10. A respiratory information estimation device, characterized in that, The respiratory information estimation device includes: The specific area setting unit sets a specific area based on an image containing the upper body of the subject; The motion vector calculation unit calculates motion vectors based on the amount of movement of a motion vector calculation point in each of the continuously captured images. This motion vector calculation point is a reference point for the motion vector. The breathing direction axis calculation unit uses the motion vector and calculates the breathing direction axis through principal component analysis; The breathing reference axis calculation unit detects two straight lines in the direction of the center of motion toward the lungs, calculates the breathing center point based on the intersection of the two straight lines, and calculates the breathing reference axis as the straight line from the breathing center point toward the motion vector calculation point. The breathing center point is the starting point of the motion caused by breathing. The breathing direction axis correction unit corrects the deviation of the breathing direction axis caused by individual differences based on the angle difference between the breathing reference axis and the breathing direction axis, and calculates the corrected breathing direction axis. The corrected motion vector calculation unit calculates the corrected motion vector in the direction along the corrected breathing direction axis; The respiratory signal calculation unit calculates the respiratory signal based on the component of the corrected motion vector in the direction of the respiratory reference axis. as well as The respiratory information calculation unit calculates a respiratory information estimation result based on the respiratory signal, which is a result of estimating the body movement caused by breathing.
11. The respiratory information estimation device according to claim 10, characterized in that, If the angle difference between the breathing reference axis and the breathing direction axis at the motion vector calculation point is within a predetermined threshold, then the corrected breathing direction axis is set as the breathing direction axis. If the angle difference is greater than the predetermined threshold and the angle obtained by subtracting the angle difference from 180° is greater than the threshold, then the corrected breathing direction axis is set as the breathing reference axis. If the angle difference is greater than the predetermined threshold and the angle obtained by subtracting the angle difference from 180° is smaller than the threshold, then the corrected breathing direction axis is set as the axis after reversing the direction of the breathing direction axis by 180°.
12. The respiratory information estimation device according to claim 10 or 11, characterized in that, The motion vector calculation unit calculates the motion components in regions where no motion is generated by breathing as non-breathing components, and calculates the breathing components by taking the difference between the average value of the motion components in the specific region and the average value of the non-breathing components.
13. A method for estimating respiratory information, characterized in that, Based on an image containing the upper body of the subject, a specific region and motion vector calculation points are defined. These motion vector calculation points serve as the reference points for the motion vectors. Two straight lines pointing towards the center of motion toward the lungs are detected, and the respiratory center point is calculated based on the intersection of these two lines. The straight line from the respiratory center point toward the motion vector calculation point is calculated as the respiratory reference axis. This respiratory center point is the starting point of the motion caused by breathing. In each of the continuously captured images, the motion vector is calculated based on the movement of the points. The respiratory signal is calculated based on the component of the motion vector along the respiratory reference axis. The respiratory information estimation result is calculated based on the respiratory signal, and the respiratory information estimation result is the result of estimating the body movement caused by breathing.