Foldable electronic devices

Abstract

A foldable electronic device is provided. The foldable electronic device may include: a main body including a first body and a second body, the first body and the second body being configured to fold toward or unfold toward each other along a folding line where the first body and the second body meet; an image sensor including a first image sensor and a second image sensor disposed at the first body; and a processor configured to, when an object contacts the first image sensor and the main body is folded along the folding line, obtain a contact image of the object from the first image sensor disposed at the first body and obtain an image of a marker displayed on the second body from the second image sensor disposed at the first body, and estimate biometric information based on the contact image of the object and the image of the marker.

Description

[0001] This application claims priority from Korean Patent Application No. 10-2019-0115510 filed on September 19, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] Apparatuses and methods consistent with example embodiments relate to a foldable electronic device and a method of estimating bio-information by using the foldable electronic device. Background Art

[0003] Generally, methods of non-invasively measuring blood pressure include a method of measuring blood pressure using an upper arm cuff and a method of estimating blood pressure without a cuff (eg, based on a light signal emitted to and reflected from a subject).

[0004] The Korotkoff sound method is one type of cuff-based blood pressure measurement method. In this method, the pressure in a cuff wrapped around the upper arm is increased, and while the pressure is reduced, blood pressure is measured by listening to the sounds produced in the blood vessels through a stethoscope. Another cuff-based blood pressure measurement method is the oscillometric method using an automated machine. In this oscillometric method, a cuff is wrapped around the upper arm, the pressure in the cuff is increased, and while the pressure in the cuff is gradually reduced, the pressure in the cuff is continuously measured. Blood pressure is measured based on the point where the pressure signal changes the most.

[0005] Cuffless blood pressure measurement methods generally include a method of estimating blood pressure by calculating pulse transit time (PTT) and a pulse wave analysis (PWA) method of estimating blood pressure by analyzing the pulse waveform. Summary of the Invention

[0006] According to aspects of an example embodiment, a foldable electronic device is provided, comprising: a main body portion, comprising a first body and a second body, the first body and the second body being configured to fold toward each other or unfold toward each other along a folding line where the first body and the second body meet; an image sensor portion, comprising a first image sensor and a second image sensor disposed at the first body; and a processor configured to: when an object contacts the first image sensor and the main body portion is folded along the folding line, obtain a contact image of the object from the first image sensor disposed at the first body, and obtain an image of a mark displayed on the second body from the second image sensor disposed at the first body, and estimate biometric information based on the contact image of the object and the image of the mark.

[0007] The image sensor portion may be provided at an inner side of the first body that is not exposed to the outside of the foldable electronic device when the body portion is folded.

[0008] The first image sensor may be configured to obtain a contact image when the object gradually changes a contact pressure applied to the first image sensor while the object is in contact with the first image sensor.

[0009] The foldable electronic device may further include a display portion, the display portion including a first display and a second display respectively arranged on the inner side of the first body and the inner side of the second body, and the inner side of the first body and the inner side of the second body are not exposed to the outside of the foldable electronic device when the main body portion is folded.

[0010] The first display and the second display may be integrally formed to be foldable.

[0011] The processor may be further configured to output the image of the mark to a second display of the second body.

[0012] The second image sensor may be further configured to obtain an image of the mark output to the second display while the second body is rotated to press the object in contact with the first image sensor.

[0013] The processor may also be configured to obtain the contact pressure applied by the object to the first image sensor based on a change in the size of the mark while the second body rotates to press the object, or to obtain the contact pressure applied by the object to the first image sensor based on the size of the mark at a random time.

[0014] The processor may be further configured to output the processing result to a display unit.

[0015] The processor may be further configured to output the bio-information estimation result to the first display, and output information used for estimating the bio-information to the second display.

[0016] The processor may be further configured to: output the biometric information estimation history to the second display; and in response to a user input selecting the estimation history for a specific time, the processor may be further configured to output the biometric information estimation result for the specific time to the first display.

[0017] The processor may be further configured to obtain a pulse wave signal based on the contact image, and obtain a contact pressure between the object and the first image sensor based on the marker image.

[0018] The processor may be further configured to obtain an oscillometric envelope representing the amplitude and contact pressure of the pulse wave signal, and estimate the bio-information based on the oscillometric envelope.

[0019] The biological information may include one or more of blood pressure, vascular age, arterial stiffness, aortic pressure waveform, vascular compliance, pressure index, and fatigue level.

[0020] According to aspects of an example embodiment, there is provided a foldable electronic device comprising: a main body portion, comprising a first body and a second body, the first body and the second body being configured to fold toward each other or unfold toward each other along a folding line where the first body and the second body meet; an image sensor disposed at the first body and configured to obtain a contact image of an object; a display disposed at the main body portion and configured to obtain touch data while the object is in contact with the image sensor and the second body is rotated to press the object against the image sensor; and a processor configured to estimate biometric information based on the contact image and the touch data.

[0021] The processor may be further configured to obtain a pulse wave signal based on the contact image of the object, obtain a contact pressure between the object and the image sensor based on the touch data, and estimate bio-information based on the pulse wave signal and the contact pressure.

[0022] The processor may be further configured to obtain an oscillometric envelope representing the amplitude and contact pressure of the pulse wave signal, and estimate the bio-information based on the oscillometric envelope.

[0023] The processor may be further configured to convert a change in a statistical value of pixel intensity obtained in a predetermined area of ​​the display during a predetermined period of time or a statistical value of pixel intensity at a random time into contact pressure by using a predetermined contact pressure conversion model.

[0024] According to aspects of an example embodiment, a foldable electronic device is provided, comprising: a main body portion, comprising a first body and a second body, the first body and the second body being configured to fold toward each other or unfold toward each other along a folding line where the first body and the second body meet; a first image sensor, disposed at the first body and configured to obtain a first contact image from a first object; a second image sensor, disposed at the second body and configured to obtain a second contact image from a second object; and a processor, configured to estimate biometric information based on the first contact image and the second contact image.

[0025] The first image sensor and the second image sensor may be disposed on an outer side of the first body and an outer side of the second body, respectively, and the outer side of the first body and the outer side of the second body may be exposed to the outside of the foldable electronic device when the body portion is folded.

[0026] The first object and the second object may be different parts of a palm. When the main body is unfolded and placed on the palm, the first image sensor and the second image sensor may be configured to obtain a first contact image and a second contact image from the first object and the second object, respectively.

[0027] The first object and the second object are fingers of different hands of the user. When the main body is unfolded and placed on the palm, when the first object and the second object contact each other, the first image sensor and the second image sensor can be configured to obtain a first contact image and a second contact image, respectively.

[0028] The processor may be further configured to obtain a first pulse wave signal and a second pulse wave signal based on the first contact image and the second contact image, respectively.

[0029] The processor may be further configured to obtain characteristic points corresponding to each other from each of the first pulse wave signal and the second pulse wave signal, calculate pulse transit time (PTT) based on a time difference between the obtained characteristic points, and estimate biological information based on the calculated PTT.

[0030] The processor may be further configured to obtain information of an angle formed between the first body and the second body while obtaining the contact image of the object, and estimate biometric information based on the contact image, the touch data, and the information of the angle.

[0031] According to aspects of an example embodiment, there is provided a method for estimating biometric information by using a foldable electronic device, the foldable electronic device including a main body portion, the main body portion including a first body and a second body, the first body and the second body being configured to fold toward each other or unfold toward each other along a folding line where the first body and the second body meet, the method comprising: obtaining a contact image of the object by using a first image sensor disposed at the first body when an object contacts a first image sensor and the main body portion is folded along the folding line; obtaining an image of a marker displayed on the second body by using a second image sensor disposed at the first body when the object contacts the first image sensor and the main body portion is folded along the folding line; and estimating biometric information based on the contact image and the marker image.

[0032] The method may further include outputting the image of the marker to a display provided on a second body of the main body portion.

[0033] The obtaining of the marker image may include obtaining the image of the marker output to the display by using a second image sensor while the object is in contact with the first image sensor and the second body is rotated to press the object against the first image sensor.

[0034] The step of estimating the bio-information may include: obtaining a pulse wave signal based on a contact image of the object; obtaining a contact pressure based on a change in the size of the mark while the second body rotates to press the object or based on the size of the mark at a random time; and estimating the bio-information based on the pulse wave signal and the contact pressure.

[0035] According to aspects of an example embodiment, there is provided a method for estimating biometric information by using a foldable electronic device, the foldable electronic device including a main body portion, the main body portion including a first body and a second body, the first body and the second body being configured to fold toward each other or unfold toward each other along a folding line where the first body and the second body meet, the method comprising: obtaining a contact image of an object by using an image sensor disposed at the first body; obtaining touch data when the object is in contact with the image sensor and the second body is rotated to press the object against the image sensor by using a display disposed on the first body; and estimating biometric information based on the contact image and the touch data.

[0036] The estimating of the bio-information may include obtaining a pulse wave signal based on the contact image of the object; obtaining a contact pressure between the object and the image sensor based on the touch data; and estimating the bio-information based on the pulse wave signal and the contact pressure. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and / or other aspects will become more apparent by describing certain example embodiments with reference to the accompanying drawings, in which:

[0038] Figures 1A to 1C is a schematic diagram illustrating a structure of a foldable electronic device according to an example embodiment;

[0039] Figure 2 is a block diagram illustrating a foldable electronic device according to example embodiments;

[0040] Figures 3A to 3E is a diagram explaining an example of estimating biological information;

[0041] Figure 4A and Figure 4B is a diagram explaining another example of estimating biological information;

[0042] Figure 5 is a block diagram illustrating a foldable electronic device according to another example embodiment;

[0043] Figures 6A to 6C is a diagram explaining another example of estimating biological information;

[0044] Figure 7 is a block diagram illustrating a foldable electronic device according to another example embodiment;

[0045] Figure 8 is a flowchart illustrating a method of estimating bio-information according to an example embodiment;

[0046] Figure 9 is a flowchart illustrating a method of estimating bio-information according to another example embodiment; and

[0047] Figure 10 is a flowchart illustrating a method of estimating bio-information according to another example embodiment. DETAILED DESCRIPTION

[0048] Example embodiments are described in more detail below with reference to the accompanying drawings.

[0049] In the following description, the same reference numerals are used for the same elements even in different drawings. Matters defined in the description, such as detailed configurations and elements, are provided to assist in a comprehensive understanding of the exemplary embodiments. However, it is apparent that the exemplary embodiments can be practiced without those specifically defined matters. In addition, since well-known functions or structures would obscure the description with unnecessary detail, they are not described in detail.

[0050] It will be understood that although the terms first, second, etc. may be used to describe various elements herein, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. In addition, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. It will also be understood that, unless explicitly described to the contrary, when an element is referred to as "comprising" another element, the element is not intended to exclude one or more other elements, but rather to include one or more other elements. In the following description, terms such as "unit" and "module" indicate a unit for processing at least one function or operation, and they can be implemented using hardware, software, or a combination thereof.

[0051] Expressions such as "at least one of," when following a list of elements, modify the entire list, not the individual elements within the list. For example, the expression "at least one of a, b, and c" should be understood to include only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0052] Hereinafter, example embodiments of a foldable electronic device and a method of estimating bio-information using the foldable electronic device will be described in detail with reference to the accompanying drawings.

[0053] Figures 1A to 1C is a schematic diagram illustrating a structure of a foldable electronic device according to example embodiments.

[0054] Reference Figure 1A The foldable electronic device 100 includes a main body 110 and a display 130 mounted on the main body 110 .

[0055] The main body 110 includes a first body 111 and a second body 112, which are connected to each other by a hinge located at a rotation axis 113 (e.g., also referred to as a folding axis, folding line, hinge axis, or hinge line). In addition, when rotated about the rotation axis, the first body 111 and the second body 112 can be converted from a folded state to an unfolded state, and vice versa. In this case, when the first body 111 and the second body 112 rotate, the surface on which the display unit 130 is provided (hereinafter referred to as the "display surface" or "front surface") can be folded inward. However, the display surface is not limited thereto and can be folded in the opposite direction.

[0056] The display unit 130 disposed on the display surface of the main body 110 may include a first display 131 disposed on the display surface of the first body 111 and a second display 132 disposed on the display surface of the second body 112. As shown here, the first display 131 and the second display 132 may be integrally formed to be foldable. However, the first display 131 and the second display 132 are not limited thereto and may be separated from each other. In addition, the first display 131 and the second display 132 may include touch screens for receiving user touch input. Furthermore, the first display 131 and the second display 132 may include fingerprint sensors for obtaining a fingerprint image when a user touches the fingerprint sensor with a body part.

[0057] In addition, refer to Figure 1C , the display unit 130 may further include a third display 133, which is provided on a surface of the main body 110 that is exposed to the outside when the main body 110 is folded (hereinafter referred to as a "cover surface" or "rear surface") (that is, the third display 133 is provided on the cover surface of the second body 112). The third display 133 may also include a touch screen for receiving a user's touch input. In addition, the third display 133 may also include a fingerprint sensor for obtaining a fingerprint image when the user touches the fingerprint sensor with a body part. However, the third display 133 is not limited to this and may be omitted if necessary, or may be manufactured in a compact size without a touch screen so that only a minimum amount of information can be displayed on the third display 133.

[0058] When the main body portion 110 is folded, the user can input commands (such as a request to estimate bio-information, a command for displaying a history of bio-information estimation, a command for outputting health monitoring results, etc.) to the third display 133. In addition, when the user unfolds the main body portion 110 during an operation for estimating bio-information through the third display 133, information displayed on the third display 133 can be enlarged on the first display 131 and the second display 132.

[0059] In addition, the foldable electronic device 100 may include one or more image sensor units mounted on the main body 110. For example, referring to Figure 1A and Figure 1B , the first image sensor portion 120a and the second image sensor portion 120b may be respectively provided on the display surface and the cover surface of the first body 111. Figure 1C As shown in FIG, the third image sensor portion 120 c may be provided on the cover surface of the second body 112. However, the image sensor portion is not limited thereto, some of the image sensor portions may be omitted, and another image sensor portion may be further provided on the display surface of the second body 112.

[0060] Each of the first image sensor section 120a, the second image sensor section 120b, and the third image sensor section 120c may include one or more image sensors. For example, as shown here, the first image sensor section 120a, provided on the display surface of the first body 111, may include two image sensors 121 and 122; the second image sensor section 120b, provided on the cover surface of the first body 111, may include three image sensors 123, 124, and 125. Furthermore, the third image sensor section 120c, provided on the cover surface of the second body 112, may include a single image sensor 126. However, these are merely examples of image sensors, and the number of image sensors included in each image sensor section is not limited to the illustrated embodiment.

[0061] The foldable electronic device 100 may include a processor installed in the main body 110 for processing its various functions, as well as various other hardware modules (e.g., a communication module, a storage module, etc.). For example, in response to a request to estimate biometric information, the processor may obtain various information using at least one of the first image sensor unit 120a, the second image sensor unit 120b, and the third image sensor unit 120c and / or the display unit 130, and may estimate the biometric information using the obtained information. In this case, the biometric information may include various cardiovascular information such as blood pressure, vascular age, arterial stiffness, aortic pressure waveform, vascular compliance, stress index, and fatigue level.

[0062] Figure 2 is a block diagram illustrating a foldable electronic device according to an embodiment. Figures 3A to 3E : are diagrams explaining an example of estimating biological information.

[0063] Reference Figure 2 , the foldable electronic device 200 according to an embodiment includes an image sensor portion 210 , a display portion 220 , and a processor 230 .

[0064] Upon receiving a request to estimate biometric information, the processor 230 may output a marker M having a predetermined size on the second display 132 provided on the display surface of the second body 202. In addition, the processor 230 may control the image sensor portion 210 to obtain contact image data of the object and an image of the marker M.

[0065] While the object contacts the image sensor unit 210 and gradually changes the contact pressure, the image sensor unit 210 can obtain contact image data of the object. In addition, while obtaining the contact image data, the image sensor unit 210 can obtain an image of the marker M output on the display unit 220.

[0066] For example, refer to Figure 3A The image sensor portion 210 may be provided on the display surface of the first body 201 of the foldable electronic device 200 and may include at least two image sensors 1 and 2. When an object OBJ (e.g., a user's finger) touches the first image sensor 1, the first image sensor 1 may obtain contact image data of the finger.

[0067] For example, the first image sensor 1 may detect light scattered or reflected from the object OBJ and may generate contact image data of the object OBJ. While the object is in contact with the first image sensor 1, the first image sensor 1 may generate continuous image data of the object OBJ.

[0068] The first image sensor 1 may be a digital camera sensor or an optical image sensor (such as a CMOS image sensor (CIS)), but is not particularly limited thereto. The first image sensor 1 may include a light source that can emit light onto the object OBJ when the object OBJ touches the first image sensor 1. The light source may include a light emitting diode (LED), a laser diode, etc., and may be formed as one light source or an array of multiple light sources. However, the light source is not limited thereto, and the first image sensor 1 may use, for example, light emitted from a first display provided on the display surface of the first body 201 as the light source for the object OBJ.

[0069] Furthermore, the first image sensor 1 may include a pixel array, and each pixel of the pixel array may include a detector (such as a photodiode, a phototransistor, etc.). The detector of each pixel may detect light scattered or reflected from the object OBJ, may convert the detected light into an electrical signal, and may output pixel data representing a contact image of the object OBJ.

[0070] When a user touches the first image sensor 1 with an object OBJ, the user can change the contact pressure between the object OBJ and the first image sensor 1. For example, as shown here, when the object OBJ is in contact with the first image sensor 1, the user can gradually apply pressure to the object OBJ by rotating the second body 202 of the foldable electronic device 200 toward the first body 201, so that the object OBJ is placed between the first body 201 and the second body 202, and the second body 202 presses the object OBJ against the first body 201. The first body 201 and the second body 202 can be folded to press the object OBJ in a pliers-like manner. However, the change in contact pressure is not limited to this, and the user can change the contact pressure by pressing the first image sensor 1 with object OBJ with gradually increasing force, or by gradually reducing the pressing force when a force greater than or equal to a predetermined threshold is applied to the first image sensor 1.

[0071] When the second body 202 rotates to press the object OBJ and gradually approaches the second image sensor 2, the second image sensor 2 may obtain an image of the marker M that is output to the second display 132 provided on the display surface of the second body 202. In this case, the second image sensor 2 may obtain continuous marker images starting from an initial state before the second body 202 contacts the object OBJ and applies a force to the object OBJ to a time when a predetermined force is applied to press the object OBJ.

[0072] Reference Figure 3B , upon receiving pixel data representing a contact image (1) of an object OBJ from the first image sensor 1, the processor 230 may obtain a PPG signal based on the received pixel data at each time as shown in a PPG signal graph (2). In this case, the pixel data at each time may indicate a pixel intensity of each pixel.

[0073] For example, the processor 230 may convert the pixel intensity at each time into the pulse wave amplitude at each time by using an amplitude conversion model that represents the relationship between pixel intensity and amplitude to obtain the pulse wave signal. For example, the amplitude conversion model may be an equation for calculating the average value of pixel intensity, but is not limited thereto. In addition, the processor 230 may convert the pixel intensity at each time into the pulse wave amplitude at each time by using an amplitude conversion model that represents the relationship between pixel intensity and amplitude. Figure 3B The region of interest is set based on the contact image (1) of FIG. 1 , and the amplitude can be obtained based on the intensity of the pixels in the set region of interest. In this case, the processor 230 can obtain a predetermined region as the region of interest from the contact image based on a characteristic point (e.g., the center of the fingerprint).

[0074] Reference Figure 3C, upon receiving the marker image (1) from the second image sensor 2, the processor 230 may obtain the contact pressure between the object OBJ and the first image sensor 1 based on the marker image (1) as shown in the contact pressure curve diagram (2).

[0075] For example, based on the marker images received during the predetermined time period, the processor 230 may obtain a size change of the marker at each time (e.g., see FIG. 1 ) from the initial time when the object contacts the first image sensor 1 to the final time. Figure 3C ). In other words, as the force applied to the object by the second body 202 gradually increases, the marker M output on the second display 132 gradually approaches the second image sensor 2, and thus the size of the marker image obtained by the second image sensor 2 also gradually increases. In addition, by converting the size change of the marker at each time, the processor 230 can obtain the contact pressure at each time. For example, the processor 230 can obtain the contact pressure by using a contact pressure conversion model that represents the relationship between the size change of the marker and the contact pressure. In this case, the contact pressure conversion model can be a universal linear estimation equation or a nonlinear estimation equation obtained from multiple users. Alternatively, the contact pressure conversion model can be an estimation equation personalized for a specific user through calibration.

[0076] In another example, instead of converting the change in size of the mark at each time into contact pressure, the processor 230 may obtain the contact pressure based on the mark size itself at a random time after the predetermined force is applied (e.g., the final time when the maximum force is applied), or based on the change in size of the mark at the final time compared to the initial time.

[0077] The processor 230 may estimate biological information based on the obtained pulse wave signal and contact pressure. For example, the processor 230 may estimate blood pressure using an oscillometric method based on the pulse wave signal and contact pressure.

[0078] Figure 3D and Figure 3E is a diagram illustrating an example of estimating blood pressure using the oscillometric method.

[0079] Figure 3D : is a diagram showing an example of the pulse wave signal obtained as described above. Figure 3D As shown in , when the user touches the first image sensor 1 with an object and gradually increases the force by pressing the object with the second body 202, the amplitude of the pulse wave signal (for example, the intensity of the detected light) also shows a gradually increasing trend during a predetermined period of time. Figure 3DAs shown in , the processor 230 can extract the peak-to-peak point of the pulse wave signal waveform by subtracting the negative (-) amplitude value in3 from the positive (+) amplitude value in2 of the waveform envelope in1 at each measurement time, and can obtain the oscillometric envelope OW by plotting the peak-to-peak amplitude at the same time point against the contact pressure value at each measurement time.

[0080] Reference Figure 3E Processor 230 may obtain features for estimating blood pressure from the obtained oscillometric envelope OW. Processor 230 may obtain the following values ​​as features from the oscillometric envelope OW: the amplitude value MA of the maximum peak point, the contact pressure value MP of the maximum peak point, contact pressure values ​​SP and DP at points to the right and left of the contact pressure value MP of the maximum peak point, and corresponding to amplitude values ​​with a preset peak ratio (e.g., 0.5 to 0.7) of the amplitude value MA of the maximum peak point. However, the features are not limited thereto, and processor 230 may obtain additional features (such as the maximum amplitude value, the time value corresponding to the maximum amplitude value, the time and amplitude values ​​of points related to the propagating wave and the reflected wave, a combination of the obtained values, etc.).

[0081] When the features are extracted, the processor 230 can estimate the blood pressure by applying a predefined blood pressure estimation model. The blood pressure estimation model can be defined as various linear or nonlinear combination functions (such as addition, subtraction, division, multiplication, logarithm, and regression equations), without specific limitations. For example, the following equation 1 represents a simple linear function.

[0082] [Equation 1]

[0083] y=ax+b

[0084] Here, y represents the estimated blood pressure value to be obtained; x represents the extracted feature value; a and b are values ​​pre-obtained through preprocessing and can be personalized for each user. For example, by using Equation 1 above, which is defined for each of the mean arterial pressure (MAP), diastolic blood pressure (DBP), and systolic blood pressure (SBP), processor 120 can independently estimate each blood pressure. For example, by inputting the extracted feature values ​​MP, DP, and SP into a function defined for each feature value, processor 230 can independently obtain MAP, DBP, and SBP.

[0085] Figure 4A and Figure 4B is a diagram explaining another example of estimating biological information.

[0086] Reference Figure 2 、 Figure 4A and Figure 4BWhen a user touches a touch area TA of the first display set on the first body 201 with an object OBJ and rotates the second body 202 to press the object with gradually increasing force, the first image sensor 1 and / or the second image sensor 2 set in the touch area can obtain contact image data of the object.

[0087] In addition, the touch screen of the first display provided on the first body 201 may generate touch data in response to contact by an object. In this case, the touch screen includes an optical panel, a capacitive panel, a resistive panel, an infrared (IR) panel, a surface acoustic wave (SAW) panel, an electromagnetic (EM) panel, an electromagnetic resonance (EMR) panel, etc.

[0088] For example, Figure 4B As shown in , touch data may be a distribution of pixel intensities. In the case where the touch screen panel is a capacitive panel, touch data may be a distribution of capacitance accumulated in each pixel. In the case where the touch screen panel is an optical panel, the touch screen may include a light source and a detector, wherein the touch data may be a distribution of the intensity of light detected by each pixel of the detector when light emitted by the light source is scattered or reflected from the object OBJ. Generally, when pressure is applied to the touch screen, the contact time and contact area change as the pressure changes, and the pixel intensity also changes accordingly.

[0089] As described above, the processor 230 can obtain a pulse wave signal based on the contact image data of the object obtained from the first image sensor 1 and / or the second image sensor 2. Furthermore, upon receiving touch data at each time, the processor 230 can obtain the contact pressure at each time based on the touch data at each time. For example, the processor 230 can convert a statistical value (such as a sum, average, median, etc.) of the pixel intensity in the touch area TA at each time into a contact pressure value at each time. In this case, a contact pressure conversion model representing the relationship between pixel intensity and contact pressure can be predefined as a linear function or a nonlinear function.

[0090] When obtaining the pulse wave signal and the contact pressure, the processor 230 may use the above reference Figure 3D and Figure 3E The oscillometric method described above estimates biometric information. For example, when obtaining statistical values ​​of pixel intensity at each time, processor 230 may obtain contact pressure based on a change in the statistical value of pixel intensity at each time compared to the pixel intensity at a random time (e.g., the initial measurement time). However, processor 230 is not limited thereto and may obtain contact pressure based on statistical values ​​at a random time (e.g., the final measurement time).

[0091] Reference above Figures 3A to 3CAn example of obtaining contact pressure by using a marker image or touch data has been described. However, the example of obtaining contact pressure is not limited thereto; when the second body 202 is folded to press an object, the processor 230 can obtain contact pressure by measuring the angle formed between the first body 201 and the second body 202 and using the angle change between the initial time and each time.

[0092] Return to reference Figure 2 , the processor 230 can output the processing results through the display unit 220. For example, the processor 230 can output the biometric information estimation results to the first display 131 and can output information used to estimate the biometric information (e.g., pulse wave signal, contact pressure, oscillometric envelope, etc.) to the second display 132. Optionally, the processor 230 can output the biometric information estimation history to the second display 132; when the user selects the biometric information estimation history for a specific time, the processor 230 can output the biometric information estimation results for the selected time to the first display 131. In addition, the processor 230 can briefly display the estimated biometric information values ​​and the like on the third display 133. When the user requests detailed information and expands the main unit 110, the processor 230 can output the detailed information to the first display 131 and the second display 132, but the present invention is not limited thereto.

[0093] Figure 5 is a block diagram illustrating a foldable electronic device according to another embodiment. Figures 6A to 6C is a diagram explaining another example of estimating biological information.

[0094] Reference Figure 5 , a foldable electronic device 500 according to an embodiment includes a first image sensor portion 511 , a second image sensor portion 512 , a display portion 520 , and a processor 530 .

[0095] Reference Figure 6A and Figure 6B , the first image sensor unit 511 may be provided on the cover surface of the first body 501, and the second image sensor unit 512 may be provided on the cover surface of the second body 502. As shown here, the first image sensor unit 511 may include three image sensors 1, 2, and 3. Also, as shown here, the second image sensor unit 512 may include one image sensor 1. However, this is merely an example, and the number of image sensors in each of the image sensor units 511 and 512 is not particularly limited thereto.

[0096] Reference Figure 6AWhen the first body 501 and the second body 420 are unfolded and placed on the palm of a hand, the first image sensor unit 511 and the second image sensor unit 512 can respectively obtain first contact image data and second contact image data from the first object OBJ and the second object OBJ that are in contact with the first image sensor unit 511 and the second image sensor unit 512. In this case, the first object OBJ1 can be one end of the palm (e.g., a finger), and the second object OBJ2 can be the other end of the palm (e.g., a portion near the wrist). However, the objects are not limited thereto.

[0097] The processor 530 may obtain the first pulse wave signal and the second pulse wave signal based on the first contact image data and the second contact image data received from the first image sensor portion 511 and the second image sensor portion 512 , respectively.

[0098] If multiple image sensors 1, 2, and 3 are included, as in the case of first image sensor unit 511, processor 530 may operate only one of the sensors and obtain a first pulse wave signal based on the first contact image data obtained by the operating image sensor. Alternatively, processor 530 may operate two or more of the multiple image sensors 1, 2, and 3 and obtain a first pulse wave signal by using multiple first contact image data obtained from the two or more image sensors. For example, processor 530 may identify an image sensor at a location determined to be in a relatively good contact state based on multiple first contact image data, and obtain a first pulse wave signal by using the first contact image data obtained from the identified image sensor. However, processor 530 is not limited to this and may combine multiple first contact image data and use the combined result.

[0099] Reference Figure 6B , when the first body 501 and the second body 502 are unfolded and the user holds the first body 501 with the thumb and index finger of the right hand, the user can touch the first image sensor unit 511 with the index finger OBJ1 of the right hand. When the user holds the second body 502 with the thumb and index finger of the left hand, the user can touch the second image sensor unit 512 with the index finger OBJ2 of the left hand. Figure 6B As shown in FIG, the first image sensor unit 511, the second image sensor unit 512, and the third image sensor unit 513 can be in contact with the right index finger, the left index finger, and the right thumb, respectively. The opposite situation is also possible, in which the user holds the first body 501 and the second body 502 with the left hand and the right hand, respectively. The first image sensor unit 511 and the second image sensor unit 512 can obtain first contact image data and second contact image data from the first object OBJ1 and the second object OBJ2 that are in contact with the first image sensor unit 511 and the second image sensor unit 512, respectively.

[0100] Figure 6A and Figure 6B An example of obtaining contact image data from two or more objects by using two or more image sensor units provided for the main body unit is shown. However, the example of obtaining contact image data is not limited thereto, and contact image data from two or more objects can be obtained by a combination of the first image sensor unit 511 and the third image sensor unit 513 or a combination of the second image sensor unit 512 and the third image sensor unit 513.

[0101] The processor 530 may obtain a first pulse wave signal and a second pulse wave signal based on the first contact image data and the second contact image data received from the first image sensor portion 511 and the second image sensor portion 512 , respectively.

[0102] The processor 530 may calculate pulse transit time (PTT) based on the first pulse wave signal and the second pulse wave signal obtained from different parts of the human body, and may estimate bio-information based on the calculated PTT.

[0103] For example, Figure 6C As shown in , the processor 530 can obtain the times T1 and T2 of the characteristic points corresponding to each other from each of the first pulse wave signal 61 and the second pulse wave signal 62, and calculate the PTT by using the delay time between the obtained characteristic points. Specifically, the characteristic points may include the maximum point of the pulse wave, the maximum point of the first-order differential signal, the local minimum point and / or the local maximum point of the second-order differential signal, etc., but are not limited thereto.

[0104] When obtaining PTT, the processor 530 may estimate the biometric information using a predefined biometric information estimation model. For example, the biometric information estimation model may be a linear function or a nonlinear function that uses PTT as a factor to obtain an estimated biometric information value. The biometric information estimation model may be defined to also include factors such as characteristic information including the user's height, weight, gender, body mass index (BMI), and measurement environment information including measured temperature and humidity.

[0105] In addition to biometric information (such as blood pressure), processor 530 can also use PTT to estimate health status based on the pulse wave delay between the first and second pulse wave signals. For example, if the calculated PTT is greater than or equal to a predetermined threshold, processor 530 can estimate the presence of an abnormality in the blood vessels and generate health status information. In this case, a threshold can be set for each stage in multiple intervals, and a healthy abnormality level can be set for each stage.

[0106] The processor 530 may output the processing result to the display unit 520. For example, the processor 230 may provide the user with health-related information (such as biometric information estimation results, biometric information estimation history, estimated health condition information, etc.) by appropriately using the first display, the second display, and the third display.

[0107] Figure 7 is a block diagram illustrating a foldable electronic device according to another embodiment.

[0108] Reference Figure 7 The foldable electronic device 700 includes an image sensor unit 710, a display 720, a processor 730, a storage device 740, and a communication interface 750. Figure 2 and Figure 5 The image sensor part 710 , the display 720 , and the processor 730 are described in detail, and thus redundant descriptions will be omitted.

[0109] The processor 730 may estimate biometric information using information obtained through the image sensor unit 710 and / or the display 720. In this case, the information may include contact image data of the object obtained through the image sensor unit 710, touch data of the object obtained through the display 720, a marker image obtained through the image sensor unit 710, and the like. In addition, when the main body unit is folded to press the object while the object is in contact with the image sensor unit 710, the information may include an angle at which the main body unit is folded (for example, an angle formed when the first and second bodies are folded).

[0110] For example, the processor 730 may estimate the bio-information based on the oscillometric method by using a pulse wave signal obtained based on contact image data of an object and contact pressure obtained based on touch data, a marker image, or angle data. Alternatively, the processor 730 may calculate PTT based on a plurality of pulse wave signals obtained by using a plurality of contact image data obtained from two or more objects, and may estimate the bio-information by using PTT.

[0111] The storage device 740 may store various information related to biometric information, such as contact image data, touch data, the angle formed between the first and second bodies, pulse wave signals, contact pressure, PTT, estimated biometric information values, health status monitoring information, etc. Alternatively, the storage device 740 may store various reference information required for estimating biometric information. For example, the reference information may include user characteristic information (such as the user's age, gender, height, weight, health status, etc.), a biometric information estimation model, an amplitude conversion model, etc., but is not limited thereto.

[0112] In this case, the storage device 740 may include at least one storage medium of the following items: flash memory, hard disk memory, multimedia card micro memory, card type memory (for example, SD memory, XD memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc., but not limited to these.

[0113] The communication interface 750 can communicate with external devices using wired or wireless communication technologies under the control of the processor 730, and can transmit and receive various data to and from the external devices. For example, the communication interface 750 can transmit the biometric information estimation results to the external device and can receive various reference information required for estimating the biometric information from the external device. In this case, the external device may include a cuff-type blood pressure measurement device and an information processing device (such as a smartphone, tablet PC, desktop computer, laptop computer, etc.).

[0114] In this case, examples of communication technology may include Bluetooth communication, Bluetooth Low Energy (BLE) communication, near field communication (NFC), WLAN communication, Zigbee communication, Infrared Data Association (IrDA) communication, Wi-Fi Direct (WFD) communication, Ultra-Wideband (UWB) communication, Ant+ communication, WIFI communication, Radio Frequency Identification (RFID) communication, 3G communication, 4G communication, 5G communication, etc. However, this is merely exemplary and is not intended to be limiting.

[0115] Figure 8 is a flowchart illustrating a method of estimating bio-information according to an embodiment. Figure 8 The method is through Figure 2 An example of a method of estimating biometric information performed by the foldable electronic device 200 is described in detail above and thus will be briefly described below.

[0116] In operation 810, the foldable electronic device 200 may obtain contact image data of an object using a first image sensor provided at the first body of the main body. In this case, the first image sensor may be provided on the display surface of the first body. In this case, the user may change the contact pressure between the object and the first image sensor by pressing the object while rotating the second body and folding the main body. The contact image data may be pixel intensity (i.e., light intensity) detected by each pixel of the detector of the first image sensor.

[0117] Furthermore, in operation 820, while the first image sensor obtains contact image data from the object, the foldable electronic device 200 may obtain a marker image by using a second image sensor provided at the first body.

[0118] In addition, the foldable electronic device 200 may output a mark on a display provided on the display surface of the second body. For example, upon receiving a request to estimate biometric information, the foldable electronic device 200 may output a mark before operation 810. Alternatively, when a user touches the first image sensor with an object, the foldable electronic device 200 may determine whether the object is in contact with the first image sensor; if it is determined that the object and the first image sensor are in contact normally, the foldable electronic device 200 may output a mark on the display.

[0119] Then, in operation 830, the foldable electronic device 200 may obtain a pulse wave signal based on the contact image data obtained in operation 810. While obtaining continuous contact image data, the foldable electronic device 200 may obtain the amplitude of the pulse wave signal at each time based on the pixel intensity at each time.

[0120] Subsequently, in operation 840, the foldable electronic device 200 may obtain contact pressure based on the marker image obtained in operation 820. The foldable electronic device 200 may obtain the contact pressure based on a size change of the marker at each time compared to an initial time or based on a marker size at a final time, etc.

[0121] Next, the foldable electronic device 200 may estimate bio-information based on the obtained pulse wave signal and contact pressure in operation 850. For example, the foldable electronic device 200 may obtain an oscillometric envelope based on the pulse wave signal and contact pressure and estimate blood pressure using the obtained oscillometric envelope.

[0122] Then, in operation 860, the foldable electronic device 200 may output the biometric information estimation result. The foldable electronic device 200 may output the biometric information estimation result in various ways by appropriately using multiple displays provided on the display surface of the first body, the display surface of the second body, the cover surface of the second body, etc. Furthermore, the foldable electronic device 200 may provide the user with the biometric information estimation result, health status information, etc. through voice, vibration, tactile sensation, etc., by appropriately using a speaker, a haptic module, etc.

[0123] Figure 9 is a flowchart illustrating a method of estimating bio-information according to another embodiment. Figure 9 The method is through Figure 2 An example of a method of estimating biometric information performed by the foldable electronic device 200 is described in detail above and thus will be briefly described below.

[0124] In operation 910, the foldable electronic device 200 may obtain contact image data of an object using a first image sensor disposed at the first body of the main body. In this case, the first image sensor may be disposed on the display surface of the first body. In this case, the user may change the contact pressure between the object and the first image sensor by pressing the object while rotating the second body and folding the main body.

[0125] In response to the contact of the object, the foldable electronic device 200 may obtain touch data through the display provided on the first body in operation 920. In this case, the touch data may be data generated according to the type of touch screen (such as the distribution of capacitance of pixels, the intensity of light signals, etc.).

[0126] Then, in operation 930 , the foldable electronic device 200 may obtain a pulse wave signal based on the contact image data obtained in operation 910 .

[0127] Subsequently, in operation 940, the foldable electronic device 200 may obtain a contact pressure based on the touch data obtained in operation 920. By using a contact pressure conversion model, the foldable electronic device 200 may convert a change in a statistical value of pixel intensity obtained in a touch area of ​​the display during a predetermined period of time or a statistical value of pixel intensity at random times into a contact pressure.

[0128] Next, the foldable electronic device 200 may estimate bio-information based on the obtained pulse wave signal and contact pressure in operation 950. For example, the foldable electronic device 200 may obtain an oscillometric envelope based on the pulse wave signal and contact pressure, and estimate blood pressure using the obtained oscillometric envelope.

[0129] Then, in operation 960, the foldable electronic device 200 may output the biometric information estimation result. The foldable electronic device 200 may visually output the biometric information estimation result by appropriately using multiple displays, speakers, haptic modules, etc. provided on the display surface of the first body, the display surface of the second body, the cover surface of the second body, etc.; and / or the foldable electronic device 200 may provide the user with the biometric information estimation result, health status, etc. by using non-visual methods such as voice, vibration, and tactile sensation.

[0130] Figure 10 is a flowchart illustrating a method of estimating bio-information according to another embodiment. Figure 10 The method is through Figure 5 An example of a method of estimating biometric information performed by the foldable electronic device 500 is described in detail above and thus will be briefly described below.

[0131] In operation 1010, the foldable electronic device 500 may obtain first contact image data of an object by using a first image sensor provided on a first body of the main body portion. In this case, the first image sensor may be provided on a cover surface of the first body.

[0132] Furthermore, in operation 1020, the foldable electronic device 500 may obtain second contact image data of the object by using a second image sensor provided on the second body of the main body portion.

[0133] Then, in operation 1030 , the foldable electronic device 500 may obtain a first pulse wave signal based on the first contact image data obtained in operation 1010 .

[0134] Subsequently, in operation 1040 , the foldable electronic device 500 may obtain a second pulse wave signal based on the second contact image data obtained in operation 1020 .

[0135] Next, in operation 1050, the foldable electronic device 500 may obtain a pulse transit time (PTT) based on the obtained first pulse wave signal and the second pulse signal. For example, the foldable electronic device 500 may obtain points corresponding to the maximum point of the pulse wave, the maximum point of the first-order differential signal, the local minimum point and / or the local maximum point of the second-order differential signal, etc. as characteristic points, and may obtain the PTT based on the time difference between the obtained characteristic points.

[0136] Then, in operation 1060, the foldable electronic device 500 may estimate biometric information based on the obtained PTT. In this case, the foldable electronic device 500 may estimate biometric information using a biometric estimation model that defines the correlation between PTT and estimated biometric information values. In addition to PTT, the biometric estimation model may also use user characteristics, measurement environment information, and other additional factors.

[0137] Subsequently, in operation 1070, the foldable electronic device 500 may output the biometric information estimation result. The foldable electronic device 200 may visually output the biometric information estimation result by appropriately using multiple displays, speakers, haptic modules, etc. provided on the display surface of the first body, the display surface of the second body, the cover surface of the second body, etc.; and / or the foldable electronic device 500 may provide the user with the biometric information estimation result, health status, etc. through non-visual methods such as voice, vibration, and tactile sensation.

[0138] Although not limited thereto, the example embodiments may be implemented as computer-readable code on a computer-readable recording medium. A computer-readable recording medium is any data storage device that can store data that can be read by a computer system later. Examples of computer-readable recording media include: read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device. The computer-readable recording medium may also be distributed on a networked computer system so that the computer-readable code is stored and executed in a distributed manner. In addition, the example embodiments may be written as a computer program transmitted via a computer-readable transmission medium (such as a carrier wave) and received and implemented in a general-purpose digital computer or a special-purpose digital computer that executes the program. In addition, it is understood that, in the example embodiments, one or more units of the above-mentioned devices and apparatus may include circuits, processors, microprocessors, etc., and may execute a computer program stored in a computer-readable medium.

[0139] The foregoing exemplary embodiments are merely exemplary and are not to be construed as limiting. The present teachings can be readily applied to other types of devices. Furthermore, the description of the exemplary embodiments is intended to be illustrative rather than limiting the scope of the claims, and many alternatives, modifications, and variations will be apparent to those skilled in the art.

Claims

1. A foldable electronic device, comprising: a main body portion including a first main body and a second main body, the first main body and the second main body being configured to be foldable toward each other or unfoldable toward each other; an image sensor portion including a first image sensor and a second image sensor provided at the first body; and The processor is configured to: When the object contacts the first image sensor and the main body is folded, a contact image of the object is obtained from the first image sensor provided at the first body, and an image of a mark displayed on the second body is obtained from the second image sensor provided at the first body, and estimating biometric information based on the contact image of the object and the image of the marker, The processor is further configured to: Obtain pulse wave signals based on contact images, obtaining a contact pressure between the object and the first image sensor based on a size change of the mark, and Bio-information is estimated based on the pulse wave signal and contact pressure.

2. The foldable electronic device according to claim 1, wherein: The image sensor portion is provided on an inner side of the first body that is not exposed to the outside of the foldable electronic device when the body portion is folded.

3. The foldable electronic device according to claim 1, wherein: The first image sensor is configured to obtain a contact image when the object gradually changes a contact pressure applied to the first image sensor while the object is in contact with the first image sensor.

4. The foldable electronic device according to claim 1, further comprising: The display portion includes a first display and a second display respectively disposed on an inner side of the first body and an inner side of the second body, wherein the first display and the second display are not exposed to the outside of the foldable electronic device when the body portion is folded.

5. The foldable electronic device according to claim 4, wherein: The first display and the second display are integrally formed to be foldable.

6. The foldable electronic device according to claim 5, wherein: The processor is further configured to output the image of the mark to a second display of the second body.

7. The foldable electronic device according to claim 6, wherein: The second image sensor is further configured to obtain an image of the mark output to the second display while the second body is rotated to press the object in contact with the first image sensor.

8. The foldable electronic device according to claim 7, wherein: The processor is further configured to obtain the contact pressure applied by the object to the first image sensor based on a change in the size of the mark while the second body rotates to press the object, or to obtain the contact pressure applied by the object to the first image sensor based on the size of the mark at a random time.

9. The foldable electronic device according to claim 4, wherein: The processor is further configured to output the processing result to the display unit.

10. The foldable electronic device according to claim 9, wherein: The processor is also configured to: outputting the biometric information estimation result to the first display, and Information for estimating the biometric information is output to the second display.

11. The foldable electronic device according to claim 9, wherein: The processor is also configured to: outputting the biometric information estimation history to a second display; and In response to a user input for selecting a biometric information estimation history at a specific time, a biometric information estimation result at the specific time is output to the first display.

12. The foldable electronic device according to claim 1, wherein: The processor is also configured to: Obtaining an oscillometric envelope representing the amplitude of the pulse wave signal and the contact pressure, and Estimating biological information based on oscillometric envelope.

13. The foldable electronic device according to claim 1, wherein: The biological information includes at least one of blood pressure, vascular age, arterial stiffness, aortic pressure waveform, vascular compliance, pressure index, and fatigue level.

14. A computer-readable storage medium storing a program, wherein: When executed by a processor, the program causes the processor to execute a method for estimating biometric information by using a foldable electronic device, the foldable electronic device including a main body portion, the main body portion including a first body and a second body, the first body and the second body being configured to be foldable toward each other or unfoldable toward each other, the method comprising: obtaining a contact image of the object by using the first image sensor provided at the first body when the object is in contact with the first image sensor and the main body portion is folded; when the object comes into contact with the first image sensor and the main body portion is folded, obtaining an image of a mark displayed on the second body by using a second image sensor provided at the first body; and estimating biometric information based on the contact image and the image of the marker, The step of estimating biological information includes: obtaining a pulse wave signal based on the contact image; obtaining a contact pressure between the object and the first image sensor based on a size change of the mark; and Estimation of biometric information based on pulse wave signals and contact pressure.

15. The computer-readable storage medium of claim 14, the method further comprising: The image of the mark is output to a display provided on the second body of the main body.

16. The computer-readable storage medium of claim 15, wherein: The obtaining of the image of the marker includes obtaining the image of the marker output to the display by using a second image sensor while an object is in contact with a first image sensor and the second body is rotated to press the object against the first image sensor.

Images