An electronic device, a server, a system, and an operating method thereof for providing a highly accurate biological signal based on information obtained in a non-contact manner

Abstract

According to various embodiments, an electronic device may be provided that includes a first communication circuit and at least one first processor, the at least one first processor being configured to: acquire, through the communication circuit, a plurality of images including a user's face acquired using a camera of a first external electronic device; acquire, through the communication circuit, while simultaneously acquiring the plurality of images, first data acquired based on a first sensor of the first external electronic device contacted with a first part of the user's body and second data acquired based on a second external electronic device contacted with a second part of the user's body; and acquire a first biosignal of a particular type based on the plurality of images, a second biosignal of the particular type based on the first data, and a third biosignal based on the second data.

Description

Technical Field

[0001] Various embodiments of the present invention relate to an electronic device, a server, a system, and an operating method thereof for providing a highly accurate biological signal based on information acquired by a non-contact method.

Background Art

[0002] The most common technique for measuring photoplethysmography (PPG) using light is to analyze the amount of transmitted light with respect to the light irradiated on the human body, and is explained by the Beer-Lambert law that the absorbance is proportional to the concentration of the absorbing substance and the thickness of the absorption layer. According to this law, since the change in transmitted light results in a signal proportional to the change in the volume of the substance through which it passes, the state of the heart can be grasped using PPG even when the absorbance of the substance is unknown.

[0003] Recently, a technology using rPPG (remote Photoplethysmography) has emerged, which is one step further from the technology using PPG. As the most popular technology for grasping signals related to heartbeats using PPG, there is a technology that directly irradiates light by bringing a device with a camera and illumination attached at a short distance, such as a smartphone, into direct contact with the human body and immediately measures the transmitted light to obtain PPG. Recently, technologies related to rPPG (remote Photoplethysmography), which grasp the change in the volume of blood vessels from the signals obtained from the images taken by a camera, are being continuously researched and developed.

[0004] The technology using rPPG can be variously applied in devices and places equipped with cameras, such as immigration control offices at airports and telemedicine, because contact between the object and the measurement equipment is not required.

[0005] However, in the technology related to rPPG, since the influence of noise generated by ambient light and the movement of the subject on the signal is significant during the process of photographing the subject with a camera, the technology of extracting only the signal related to the volume change of the measurement object from the captured video can be said to be the core technology among the technologies for measuring biological signals using rPPG.

SUMMARY OF THE INVENTION

PROBLEMS TO BE SOLVED BY THE INVENTION

[0006] rPPG (remote Photoplethysmography) will have low accuracy compared to PPG obtained using a contact-type sensor. According to various embodiments, an electronic device, a server, a system, and an operation method thereof that provide information on a biological signal having accuracy corresponding to the biological signal obtained by a contact-type method based on information obtained by a non-contact type method may be provided.

[0007] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those having ordinary knowledge in the technical field to which the present invention pertains from the following description.

MEANS FOR SOLVING THE PROBLEMS

[0008] According to various embodiments, an electronic device including a first communication circuit and at least one first processor; the at least one first processor is configured to: acquire a plurality of images including a user's face acquired using a camera of a first external electronic device through the communication circuit, and while acquiring the plurality of images through the communication circuit, acquire first data acquired based on a first sensor of the first external electronic device that is in contact with a first part of the user's body and second data acquired based on a second external electronic device that is in contact with a second part of the user's body, and an electronic device configured to acquire a first biological signal of a specific type based on the plurality of images, the second biological signal of the specific type based on the first data, and a third biological signal based on the second data may be provided.

[0009] According to various embodiments, a method of operating an electronic device includes: obtaining, through the communication circuit, a plurality of images including a user's face acquired using a camera of a first external electronic device; while obtaining the plurality of images through the communication circuit, obtaining, at the same time, first data acquired based on a first sensor of the first external electronic device that is in contact with a first part of the user's body and second data acquired based on a second external electronic device that is in contact with a second part of the user's body; and obtaining a first type of biometric signal based on the plurality of images, a second type of biometric signal of the specific type based on the first data, and a third biometric signal based on the second data. A method of operation may be provided.

Advantages of the Invention

[0010] According to various embodiments, an electronic device, a server, a system, and a method of operating the same can provide information on a biometric signal having accuracy corresponding to a biometric signal acquired in a contact method based on information acquired in a non-contact method.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18a

Figure 18b

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

[0012] The electronic devices according to various embodiments disclosed in this document can be devices in various forms. The electronic device can include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. The electronic devices according to the embodiments of this document are not limited to the devices described above.

[0013] The various embodiments in this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or alternatives of the corresponding embodiments. In connection with the description of the drawings, similar or related reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item can include one or more of the said items unless clearly indicated otherwise in the relevant context. In this document, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase of the text, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" can be used simply to distinguish the corresponding component from other corresponding components, and do not limit the corresponding component in other aspects (e.g., importance or order). When a certain (e.g., first) component is referred to as "coupled" or "connected" to another (e.g., second) component, with or without the terms "functionally" or "communicatively", it means that the certain component can be directly (e.g., wired), wirelessly, or through a third component connected to the other component.

[0014] The term "module" used in various embodiments of this document can include a unit embodied in hardware, software, or firmware, and can be used interchangeably with terms such as, for example, logic, logic block, component, or circuit. A module can be an integrated component or the smallest unit or a part of the component that performs one or more functions. For example, according to one embodiment, a module can be embodied in the form of an ASIC (application-specific integrated circuit).

[0015] Various embodiments of this document can be embodied in software (e.g., a program) that includes one or more instruction words stored in a storage medium (e.g., an internal memory or an external memory) readable by a machine (e.g., an electronic device). For example, a processor (e.g., a processor) of a machine (e.g., an electronic device) can call and execute at least one of the one or more instruction words stored from the storage medium. This enables the machine to be operated to perform at least one function by the at least one called instruction word. The one or more instruction words can include code generated by a compiler or code executable by an interpreter. The storage medium readable by the machine can be provided in the form of a non-transitory storage medium. Here, "non-transitory" only means that the storage medium is a tangible device and does not include a signal (e.g., an electromagnetic wave), and this term does not distinguish between cases where data is stored semi-permanently and temporarily in the storage medium.

[0016] According to one embodiment, the methods according to the various embodiments disclosed in this document may be provided included in a computer program product. The computer program product may be traded as a commodity between a seller and a purchaser. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™), or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a part of the computer program product may be at least temporarily stored or temporarily generated in a machine-readable storage medium such as the memory of the manufacturer's server, the application store's server, or a relay server.

[0017] According to various embodiments, each of the components (e.g., modules or programs) of the components described above may include a single or multiple entities, and some of the multiple entities may be separately arranged from other components. According to various embodiments, one or more of the corresponding components or operations described above may be omitted, or one or more other components or operations may be added. Roughly or additionally, multiple components (e.g., modules or programs) may be integrated into one component. In such a case, the integrated component can perform one or more functions of each of the multiple components in the same or similar manner as the corresponding component among the multiple components performed the functions before the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed or omitted in a different order, or one or more other operations may be added.

[0018] According to various embodiments, an electronic device includes a first communication circuit and at least one first processor. The at least one first processor is configured to: acquire a plurality of images including a user's face acquired using a camera of a first external electronic device through the communication circuit; while acquiring the plurality of images through the communication circuit, acquire first data acquired based on a first sensor of the first external electronic device that is in contact with a first part of the user's body and second data acquired based on a second external electronic device that is in contact with a second part of the user's body; and acquire a first type of biometric signal based on the plurality of images, a second type of biometric signal of the first type based on the first data, and a third biometric signal based on the second data. An electronic device may be provided.

[0019] According to various embodiments, the at least one processor may be configured to further acquire information related to photographing of the first external electronic device acquired by a second sensor of the first external electronic device while acquiring the plurality of images through the communication circuit. An electronic device may be provided.

[0020] According to various embodiments, photographing information related to photographing of the electronic device may include information related to a state related to photographing of the first external electronic device and information related to an external environment of the first external electronic device. An electronic device may be provided.

[0021] According to various embodiments, the at least one processor may be configured to further acquire personal information related to personal characteristics of the user of the first external electronic device through the communication circuit. An electronic device may be provided.

[0022] According to various embodiments, the at least one processor is configured to generate at least one artificial intelligence model by performing learning based on at least a part of the first biological signal, the second biological signal, the third biological signal, the photographing information, or the personal information, and the at least one artificial intelligence model is embodied to provide a value for the specific type of biological signal sensed in a contactless manner, and an electronic device may be provided.

[0023] According to various embodiments, the at least one artificial intelligence model is embodied to output the third biological signal based on receiving at least a part of the input of the first biological signal, the second biological signal, the photographing information, or the personal information, and an electronic device may be provided.

[0024] According to various embodiments, the at least one processor is configured to perform time synchronization of the first biological signal, the second biological signal, and the third biological signal, and an electronic device may be provided.

[0025] According to various embodiments, the at least one processor is configured to select, among the first biological signal, the second biological signal, and the third biological signal, the first biological signal related to the face closest to the heart as a reference, and synchronize the remaining second biological signal and the third biological signal with the first biological signal based on the selected first biological signal, and an electronic device may be provided.

[0026] According to various embodiments, the at least one processor is configured to generate the at least one artificial intelligence model in a state where time synchronization for the first biological signal, the second biological signal, and the third biological signal is not performed, and an electronic device may be provided.

[0027] According to various embodiments, a method of operating an electronic device includes: obtaining, through the communication circuit, a plurality of images including a user's face acquired using a camera of a first external electronic device; obtaining, through the communication circuit, while obtaining the plurality of images, first data acquired based on a first sensor of the first external electronic device that is simultaneously in contact with a first part of the user's body and second data acquired based on a second external electronic device that is in contact with a second part of the user's body; and obtaining a first biometric signal of a specific type based on the plurality of images, a second biometric signal of the specific type based on the first data, and a third biometric signal based on the second data. A method of operation may be provided that includes these operations.

[0028] According to various embodiments, a method of operation may be provided that further includes: obtaining, through the communication circuit, information related to photographing of the first external electronic device acquired by a second sensor of the first external electronic device while obtaining the plurality of images.

[0029] According to various embodiments, photographing information related to photographing of the electronic device includes information related to a state related to photographing of the first external electronic device and information related to an external environment of the first external electronic device. A method of operation may be provided that includes these.

[0030] According to various embodiments, a method of operation may be provided that further includes: obtaining, through the communication circuit, personal information related to personal characteristics of the user of the first external electronic device.

[0031] According to various embodiments, a method of operation may be provided that further includes: generating at least one artificial intelligence model by performing learning based on at least a part of the first biometric signal, the second biometric signal, the third biometric signal, the photographing information, or the personal information, wherein the at least one artificial intelligence model is implemented to provide values for the specific type of biometric signal sensed in a contact type manner.

[0032] According to various embodiments, an operation method may be provided in which the at least one artificial intelligence model is implemented to output the third biological signal based on receiving at least a part of the first biological signal, the second biological signal, the imaging information, or the personal information.

[0033] Hereinafter, the biological signal measurement system 1 according to various embodiments will be described.

[0034] According to various embodiments, the biological signal measurement system 1 may be a system implemented to provide a biological signal obtained based on an analysis of a user and / or a specimen (e.g., a part of the user's body such as a face) in a non-contact manner. The biological signal may include a photoplethysmography (PPG), a blood oxygen saturation (SPO2), a heart rate variability (HRV), an electrocardiogram (ECG), an electroencephalogram (EEG), an electromyogram (EMG), a galvanic skin response (GSR), a skin temperature (SKT), but is not limited to the described examples and may further include various types of biological signals. The biological signal measurement system 1 may collect a plurality of specific types of biological signals simultaneously in a non-contact manner and in a contact manner with higher accuracy than the non-contact manner to improve the accuracy of the biological signal obtained in the non-contact manner, and may use an artificial intelligence (AI) model learned based on the plurality of collected biological signals. Specific embodiments will be described below.

[0035] FIG. 1 is a drawing for explaining an example of the configuration of the biological signal measurement system 1 according to various embodiments. Hereinafter, FIG. 1 will be further described with reference to FIG. 2.

[0036] FIG. 2 is a drawing for explaining an example of the electronic device 10 according to various embodiments.

[0037] According to various embodiments, referring to FIG. 1, the biosignal measurement system 1 may include an electronic device 10 and a server 20. However, without being limited to the illustrated and / or described examples, the biosignal measurement system 1 may be embodied to include more devices.

[0038] According to various embodiments, the electronic device 10 may be an electronic device of a user who desires to measure a biosignal (e.g., remote Photoplethysmography, rPPG) in a non-contact manner using the biosignal measurement system 1. For example, as illustrated in FIG. 2(a), the electronic device 200 may include a user terminal such as a smartphone, a wearable device, an HMD (head mounted display) device, etc., and as illustrated in FIG. 2(b), a user device used in a form installed and / or arranged such as a kiosk, a smart mirror, etc. The electronic device 10 may be embodied to detect a specimen S in a non-contact manner and provide a biosignal based on the detection result. For example, as illustrated in FIG. 2, the electronic device 10 may acquire a plurality of images (or videos, or a single image) based on photographing a specimen S (e.g., the face of the user U), receive a biosignal (e.g., PPG) corresponding to the plurality of images from the server 20, and provide the received biosignal in a form recognizable by the user (e.g., display on a display and / or output in a sound form through a speaker).

[0039] According to various embodiments, the specimen S may be the face when measuring PPG, and may be the chest when measuring the respiratory rate, but is not limited to the described examples, and various body parts of the user U may be measured in a non-contact manner.

[0040] According to various embodiments, the server 20 may acquire a biological signal based on a specimen S detected in a non-contact manner and provide information about the acquired biological signal to the electronic device 10. For example, the server 20 may include a learning server 20a and a utilization server 20b. However, without being limited to the illustrated and / or described examples, the server 20 may be implemented as a single server that performs all the functions of the learning server 20a and the utilization server 20b. The learning server 20a may build at least one artificial intelligence model that is learned to provide a biological signal. For example, the learning server 20a may output a biological signal with an accuracy similar to the detection accuracy of the contact type as a response to receiving an input of at least one piece of information different from the non-contact biological signal acquired based on the specimen S detected in the non-contact manner, and build an artificial intelligence model implemented to do so. The artificial intelligence model learned by the learning server 20a may be provided to the utilization server 20b. The utilization server 20b may set up a communication connection with the electronic device 10 and receive information about the specimen S acquired by the electronic device 10 in a non-contact manner from the electronic device 10. The utilization server 20b may input the information about the specimen S into the artificial intelligence model, acquire information about the biological signal output from the artificial intelligence model, and transmit the acquired information about the biological signal to the electronic device 10.

[0041] On the other hand, without being limited to the described examples, the electronic device 10 may be implemented in an on-device form so that the electronic device 10 can provide a biological signal without the operation of the server 20.

[0042] Hereinafter, examples of the configurations of the electronic device 10 and the server 20 according to various embodiments will be described.

[0043] FIG. 3 is a drawing for explaining an example of a configuration of an electronic device 10 and a server 20 (e.g., a learning server 20a and a usage server 20b) according to various embodiments. In some embodiments, without being limited to the example illustrated in FIG. 3, at least one of these components may be omitted from the electronic device 20 and the server 20, or one or more other components may be added to the electronic device 20 and the server 20. In some embodiments, some of these components may be implemented as one integrated circuit. Hereinafter, FIG. 3 will be further described with reference to FIG. 4.

[0044] FIG. 4 is a drawing for explaining an example of a biological signal measurement module according to various embodiments.

[0045] Hereinafter, an example of a configuration of the electronic device 10 according to various embodiments will be described.

[0046] According to various embodiments, the electronic device 10 may include a display 11, a camera 12, a first communication circuit 13, a sensor 14, a first memory 17, and a first processor 18. On the other hand, without being limited to the illustrated and / or described examples, the electronic device 10 may be implemented to further include various electronic components (e.g., a speaker) and devices provided in a user terminal, and / or may be implemented to include even fewer components. Hereinafter, examples of each configuration will be described.

[0047] According to various embodiments, the display 11 can visually provide information to the outside (e.g., a user) of the electronic device 200. The display 11 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the corresponding device. According to one embodiment, the display 11 may include a touch circuitry set to sense a touch, or a sensor circuit (e.g., a pressure sensor) set to measure the strength of the force generated by the touch.

[0048] According to various embodiments, the camera 12 may include an image sensor for photography.

[0049] According to various embodiments, the first communication circuit 13 can assist in establishing a wireless communication channel between the electronic device 10 and an external electronic device (e.g., the server 20) and in performing communication through the established communication channel. The first communication circuit 13 operates independently of the first processor 18 and can include one or more communication processors that support wireless communication.

[0050] According to various embodiments, the sensor 14 can include a measurement sensor 15 for acquiring (or sensing) a biological signal in a contact manner and an environmental sensor 16 for acquiring (or sensing) various types of information related to photography (photography information).

[0051] For example, the measurement sensor 15 can include a PPG sensor, an SPO2 sensor, an HRV sensor, an ECG sensor, an EEG sensor, an EMG sensor, a GSR sensor, and / or an SKT sensor, and is not limited to the described examples and can further include various types of sensors. As an example, the PPG sensor can be a sensor embodied to measure a PPG signal based on a change in the amount of photosensitive light received since it emits light while in contact with the skin.

[0052] For example, the shooting information acquired by the environment sensor 16 may include a first environment sensor (e.g., an illuminance sensor 16A, etc.) for measuring information related to the surrounding environment to be shot (e.g., light quantity, illuminance, temperature, etc.), and a second environment sensor (e.g., an inclination sensor 16b, a motion sensor (not shown), etc.) for measuring information related to the state of the electronic device 10 during shooting (e.g., inclination, movement, position, height, direction). The data acquired by the environment sensor 16 may be defined as data. At least a part of the information acquired by the environment sensor 16 may be acquired not by the environment sensor 16 but by an analysis module (not shown) for analyzing an image shot by the camera 12. For example, the analysis module (not shown) can identify the light quantity, illuminance, etc. based on the values of the pixels of the image (e.g., brightness values).

[0053] According to various embodiments, the first memory 17 can store various data used by at least one component of the electronic device 210 (e.g., the first processor 18). For example, the first memory 17 can store a predetermined application. Based on the execution of the application, the operations of the electronic device 10 described below can be performed.

[0054] According to various embodiments, the electronic device 10 can be implemented such that the application acquires additional information. For example, the additional information includes personal information such as the gender, year, age, race, BMI index, etc. of the user, camera information for camera parameters (e.g., focal length, etc.) of the camera 12, the distance from the specimen, the shooting video resolution, the number of frames per second (frame per second (FPS)), etc. shooting information indicating the shooting state, and image information indicating characteristics analyzable from the image (or video) (e.g., the direction of the light illuminating the specimen (e.g., front light, backlight)). At this time, a part of the additional information (e.g., BMI index) may be acquired based on an artificial intelligence model for calculating it.

[0055] According to various embodiments, the first processor 18 can, for example, execute software to control at least one other component (e.g., a hardware or software component) of the electronic device 200 connected to the first processor 520, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operation, the first processor 520 can load instructions or data received from other components (e.g., the second communication circuit 540 or the third communication circuit 550) into the volatile memory, process the instructions or data stored in the volatile memory, and store the resulting data in the non-volatile memory. According to one embodiment, the first processor 520 can include a main processor (e.g., a central processing unit or an application processor), and an auxiliary processor (e.g., a graphics processing unit, an image signal processor, a sensor hub processor, or a communication processor) that can operate independently of or together with the main processor. Additionally or generally, the auxiliary processor can use less power than the main processor or can be configured to be specialized for a specified function. The auxiliary processor can be implemented separately from or as part of the main processor.

[0056] Hereinafter, an example of the configuration of the server 20 according to various embodiments will be described.

[0057] According to various embodiments, the learning server 20a can include a second communication circuit 21a, a second processor 22a, and a memory 23a. Since the second communication circuit 21a is the first communication circuit 13 described above, the second processor 22a is the first processor 18 described above, and the second memory 23a can be implemented together with the first memory 17 described above, duplicate descriptions will be omitted. On the other hand, without being limited to the illustrated and / or described examples, the learning server 20a can be implemented to further include more components and / or can be implemented to include fewer components. Hereinafter, examples of each configuration will be described.

[0058] According to various embodiments, the memory 23a may include a database 24a, an artificial intelligence model generation module 25a, and a first biological signal measurement module 26a. The modules 25a and 26a may be embodied in the form of computer-readable computer code, programs, software, applications, APIs, and / or instructions, and based on the execution of the modules 25a and 26a, the second processor 22a of the learning server 20a may be induced to perform specific operations.

[0059] According to various embodiments, the database 24a may be embodied to store various types of information for generating an artificial intelligence model. For example, the database 24a may store biological signals measured by a non-contact method and corresponding biological signals measured by a contact method, environmental data measured by the environmental sensor 16, and additional information, which are acquired by the first biological signal measurement module 26a described later. As described above, the additional information may include personal information, camera information, shooting information, and / or image information.

[0060] According to various embodiments, the artificial intelligence model generation module 25a may learn an artificial intelligence model capable of providing a biological signal with an accuracy similar to (or close to) the accuracy of a biological signal measured by a contact method (e.g., PPG) based on a biological signal measured by a non-contact method (e.g., rPPG). An example of the learning of the artificial intelligence model will be specifically described later.

[0061] According to various embodiments, the first biological signal measurement module 26a may be embodied to measure a non-contact biological signal and a contact biological signal based on information received from the electronic device 10. For example, as illustrated in FIG. 4, the first biological signal measurement module 26a may include information collected by a non-contact method (e.g., a plurality of images including a specimen S photographed by the camera 12)

[0062] A non-contact acquisition module 421 for acquiring a biological signal (e.g., a non-contact biological signal) based on , and a contact acquisition module 423 for acquiring a biological signal (e.g., a contact biological signal) based on information collected by a contact method (e.g., sensing information collected from sensor 14) can be included.

[0063] On the other hand, without being limited to the described examples, at least a part of the above configuration (e.g., database 24a, artificial intelligence model generation module 25a, and first biological signal measurement module 26a) can be implemented in the electronic device 10. For example, by implementing the first biological signal measurement module 26a in the electronic device 10, the electronic device 10 can acquire a non-contact biological signal and a contact biological signal, and transmit the plurality of signals to the learning server 20a so that the learning server 20a can learn an artificial intelligence model.

[0064] According to various embodiments, the utilization server 20b can include a third communication circuit 21b, a third processor 22b, and a memory 24b. Since the third communication circuit 21b can be implemented with the above-described first communication circuit 13, the third processor 22b can be implemented with the above-described first processor 18, and the third memory 24b can be implemented with the above-described first memory 17, duplicate descriptions are omitted. On the other hand, without being limited to the illustrated and / or described examples, the utilization server 20b can be implemented to further include more components and / or can be implemented to include fewer components. Examples of each configuration will be described below.

[0065] According to various embodiments, the third memory 24b may be embodied to store at least one artificial intelligence model 23b, 23c learned by the learning server 20a (e.g., the artificial intelligence model generation module 25a), and the second biological signal measurement module 24b that is the same as the aforementioned first biological signal measurement module 26a. The utilization server 20b can acquire a specific type of biological signal based on a plurality of images of the specimen S detected in a non-contact manner received from the electronic device 10, environmental data (e.g., information sensed by the environmental sensor 14), additional information (e.g., at least a part of personal information, camera information, shooting information, or image information), and the at least one artificial intelligence model 23b, 23c, and transmit the acquired specific type of biological signal to the electronic device 10.

[0066] On the other hand, without being limited to the described examples, at least a part of the above configuration (e.g., at least one artificial intelligence model 23b, 23c, and the aforementioned second biological signal measurement module 24b) may be embodied in the electronic device 10. For example, when the second biological signal measurement module 24b is embodied in the electronic device 10, the electronic device 10 can acquire non-contact biological signals and contact biological signals and transmit the plurality of signals to the utilization server 20b.

[0067] On the other hand, without being limited to the described examples, if the utilization server 20b is not embodied, at least one learned artificial intelligence model 23b, 23c may be used in a form stored in a single server that performs all the functions of the learning server 20a and the utilization server 20b, and / or the electronic device 10.

[0068] Hereinafter, an example of an operation of acquiring (or collecting) data for artificial intelligence model learning of the learning server 20a according to various embodiments will be described.

[0069] FIG. 5 is a flowchart showing an example of an operation of acquiring (or collecting) data for artificial intelligence model learning of the learning server 20a according to various embodiments. The operations may be performed regardless of the order of the operations shown and / or described, and more operations may be performed and / or fewer operations may be performed. Hereinafter, FIG. 5 will be further described with reference to FIGS. 6 to 7.

[0070] FIG. 6 is a drawing for explaining an example of an operation of simultaneously collecting biological signals in a non-contact manner and a contact manner of the electronic device 10 according to various embodiments. FIG. 7 is a drawing for explaining an example of an operation of accumulating data for artificial intelligence model learning in the learning server 20a according to various embodiments.

[0071] According to various embodiments, the learning server 20a (e.g., the second processor 22a) can, in operation 501, acquire a first biological signal of a specific type based on photographing a specimen S using the camera 12 of the electronic device 10, acquire a second biological signal based on the first contact sensor 15, and, in operation 503, acquire a third biological signal of the specific type based on the second contact sensor. For example, referring to FIG. 6, the learning server 20a can utilize the user U, the electronic device 10 held by the user U, and an external measurement sensor 600 that contacts a part of the body of the user U to acquire all (or simultaneously, or during a specific period) of the first biological signal, the second biological signal, and the third biological signal of the specific type related to each other described above. The meaning that the biological signals are related to each other can mean that they are acquired during a specific period in which the relevance to each other is higher than a threshold value. For example, the specific type of the biological signal can be PPG. As shown in FIG. 6, the user U can use the camera 10 of the electronic device 12 to photograph the first specimen (e.g., the face S1) of the user while the biological signal measurement device 600 is provided (or worn) on the third specimen (e.g., the finger S3 of the first hand) of the user U, and contact the second specimen (e.g., the finger S2 of the second hand) with a contact sensor S2 disposed in a specific area (e.g., the rear surface) of the electronic device 12. Accordingly, referring to FIG. 7, the learning server 20a can receive a plurality of images of the first specimen S1 photographed by the camera 12 of the electronic device 10 and sensing data (e.g., mobile PPG, MPPG) measured by the measurement sensor 15, and can receive sensing data (e.g., PPG) measured by the external measurement sensor 600 from the external measurement sensor 600.

[0072] For example, the non-contact measurement module 421 of the learning server 20a can acquire a specific type of non-contact biological signal based on the analysis of a plurality of images of the first specimen S1. For example, the specific type of non-contact biological signal can be rPPG. The plurality of images can be images acquired based on at least one parameter of the camera being set to a value within a specific range. For example, the images can be acquired with the camera parameters set such that a video is captured with the FPS in the range of 20 to 30 frames per second.

[0073] For example, the first contact measurement module 423a of the learning server 20a can acquire the specific type of first contact biological signal self-measured by the electronic device 10 based on the sensing data received from the measurement sensor 15. The specific type of first contact biological signal self-measured by the electronic device 10 is PPG and can be defined as MPPG (mobile PPG). The first contact measurement module 423a can acquire the MPPG that has already been measured by the electronic device 10 and received from the electronic device 10 by the learning server 20a, but is not limited to the described example and may be implemented to acquire the sensing data received from the measurement sensor 15 and measure the MPPG based on the analysis of the acquired sensing data.

[0074] For example, the second contact measurement module 423b of the learning server 20a can acquire the specific type of second contact biological signal based on the sensing data received from the external measurement device 600. The second specific type of contact biological signal is PPG and can have a relatively high accuracy compared to the accuracy of the aforementioned MPPG (mobile PPG). The second contact measurement module 423b can acquire the PPG that has already been measured by the external measurement device 600 and received from the external measurement device 600 by the learning server 20a, but is not limited to the described example and may be implemented to acquire the sensing data received from the external measurement device 600 and measure the PPG based on the analysis of the acquired sensing data.

[0075] According to various embodiments, the learning server 20a (e.g., the data acquisition module 700) can acquire additional learning information in operation 505. The additional learning information can include at least a part of environmental data acquired based on the environmental sensor 16 and additional information including personal information, camera information, shooting information, or image information acquired based on an application (not shown). For example, the electronic device 10 can use the environmental sensor 16 to acquire information related to the surrounding environment to be photographed and information related to the state of the electronic device 10 at the time of shooting, and transmit it to the learning server 20a. Also, for example, the electronic device 10 can transmit personal information input through the execution screen to the learning server 20a based on the execution of an application. Also, for example, the electronic device 10 can transmit at least a part of camera information acquired based on the set authority and shooting information and image information acquired based on the analysis of the video and / or image captured by the camera 12 to the learning server 20a based on the execution of an application.

[0076] According to various embodiments, the data acquisition module 700 can store the above-described specific types of biological signals (e.g., non-contact biological signals, contact type 1 biological signals, and contact type 2 biological signals) and additional learning information in a related form in the database 24a. At this time, the data acquisition module 700 may be implemented to perform time synchronization of specific types of biological signals (e.g., non-contact biological signals, contact type 1 biological signals, and contact type 2 biological signals), but is not limited to the described examples and time synchronization may not be performed.

[0077] Hereinafter, an example of an operation of acquiring a non-contact biological signal will be described as at least a part of the operation 501 of the learning server 20a according to various embodiments.

[0078] FIG. 8 is a flowchart showing an example of an operation of acquiring a non-contact biological signal of the learning server 20a according to various embodiments. The operations may be performed regardless of the order of the operations shown and / or described, and more operations may be performed and / or fewer operations may be performed. Hereinafter, FIG. 8 will be further described with reference to FIGS. 9 to 11.

[0079] FIG. 9 is a drawing for explaining an example of an operation of obtaining a difference value between color channels (e.g., G value and R value, G value and B value) for noise reduction according to various embodiments. FIG. 10 is a drawing for explaining an example of an operation of obtaining a characteristic value according to various embodiments. FIG. 11 is a drawing for explaining an example of an operation of obtaining a characteristic value according to various embodiments.

[0080] According to various embodiments, the learning server 20a (e.g., the non-contact measurement module 421) can acquire a plurality of images including a specimen in operation 801 and acquire a first biological signal based on the plurality of images in operation 803. For example, the learning server 20a (e.g., the non-contact measurement module 421) can acquire a value for each color channel from a plurality of images including a specimen and acquire rPPG based on the value for each color channel. For example, the color channels may each mean the R channel, the G channel, and the B channel of the RGB color space, but are not limited to the described and / or illustrated examples and may mean channels of other color spaces (e.g., CMY, HSV, etc.).

[0081] At this time, according to various embodiments, referring to FIG. 9, an operation of obtaining rPPG may be performed based on a difference value between color channels (e.g., G value and R value, G value and B value) for noise reduction.

[0082] Fig. 9(a) is a graph showing the Red channel values extracted in the RGB color space, and Fig. 9(b) is a graph showing the Green channel values extracted in the RGB color space. Referring to Figs. 9(a) and 9(b), it can be seen that the extracted color channel values vary with time. At this time, the extracted color channel values may vary due to the heartbeat of the subject, but may also vary due to the movement of the subject or the change in the intensity of external light. More specifically, the variation of the color channel values occurs slowly and greatly because it is more affected by the movement of the subject and the change in the intensity of external light, and the variation occurs quickly and slightly because it is more affected by the heartbeat of the subject. Therefore, since the variation due to the movement of the subject and the change in the intensity of external light is greater than the variation due to the heartbeat, the relative difference between at least two color channel values can be used to reduce this.

[0083] Exemplarily, the difference value between the Green channel value and the Red channel value can be used to reduce noise. More specifically, the Green channel value and the Red channel value obtained in the same image frame can reflect the same movement and the same intensity of external light, and the difference value between the Green channel value and the Red channel value of the same frame can reduce noise due to the movement of the subject and the change in the intensity of external light, etc., but it is not limited to this, and the relative difference between at least two color channel values can be used to reduce noise.

[0084] Fig. 9(c) is a graph showing the difference value between the Green channel value and the Red channel value. As shown in Fig. 9(c), the difference value between the Green channel value and the Red channel value can reduce noise due to the movement of the subject and the change in the intensity of external light, etc.

[0085] In addition, the method for reducing the noise described above can be performed on at least one of the plurality of acquired image frames, and may be performed on each of the plurality of consecutive image frames.

[0086] Although not shown in FIG. 9(c), noise can also be reduced by using the difference value between the Green channel value and the Blue channel value, and noise can also be reduced by using the difference value between the Red channel value and the Blue channel value.

[0087] Also, as described above, at least two color channel values can be selected to obtain a difference value in order to reduce noise by using the relative difference between at least two color channel values.

[0088] At this time, the at least two color channel values can be selected in consideration of the absorbance of blood.

[0089] According to various embodiments, referring to FIGS. 10 to 11, the learning server 20a acquires time-series data (e.g., a first characteristic value and a second characteristic value) of the difference value between the Green channel value and the Red channel value and the difference value between the Green channel value and the Blue channel value respectively, and obtains a third characteristic value by merging the acquired time-series data, and rPPG can be obtained based on the third characteristic value.

[0090] FIG. 10(a) is a graph showing color channel values obtained according to an embodiment, and more specifically, a graph showing the difference value between the Green channel value and the Red channel value. However, this is only shown specifically by the difference value between the Green channel value and the Red channel value for convenience of explanation, and is not limited thereto, and can be various color channel values, difference values, processed values, etc.

[0091] Referring to FIG. 10(a), it can be seen that the difference value between the Green channel value and the Red channel value (hereinafter referred to as the "G-R value") may not have a constant magnitude of change over time.

[0092] At this time, the G-R value may not be constant depending on the movement of the person being measured. For example, when the movement of the person being measured is small, the change in the G-R value may be small, and when the movement of the person being measured is large, the change in the G-R value may be large, but it is not limited to this.

[0093] Also, the G-R value may not be constant depending on the intensity of external light. For example, when the intensity of external light is weak, the change in the G-R value may be small, and when the intensity of external light is strong, the change in the G-R value may be large, but it is not limited to this.

[0094] Therefore, characteristic values can be extracted in order to reduce noise caused by factors such as the movement of the person being measured and the intensity of external light.

[0095] Also, a window for the characteristic value can be set in order to extract the characteristic value.

[0096] At this time, the window for the characteristic value may mean a preset time interval, or may mean a preset number of frames, but is not limited thereto and may mean a window for setting at least a part of a plurality of frames in order to obtain the characteristic value.

[0097] FIG. 10(b) is a schematic diagram for explaining the window for the characteristic value. More specifically, it is a schematic diagram for explaining the window for the characteristic value set with 18 image frames obtained by dividing 180 image frames into 10 equal parts. However, this only shows the window for the characteristic value set with 18 image frames obtained by dividing 180 image frames into 10 equal parts for convenience of explanation, and is not limited thereto, and the window for the characteristic value can be set in various ways and numbers.

[0098] Referring to FIG. 10(b), the plurality of acquired image frames can be set as a group by a window for characteristic values. For example, as shown in FIG. 19(b), 180 image frames can be set as a group including 18 image frames each by a window for characteristic values. More specifically, it may be included in the first image frame group 2210 from the first image frame to the 18th image frame, and may be included in the second image frame group 2220 from the 19th image frame to the 36th image frame, but is not limited thereto.

[0099] At this time, the characteristic value can be obtained for the image frame group set by the window for the characteristic value. For example, the characteristic value can be obtained for the color channel value for the first image frame group 2210, and can be obtained for the color channel value for the second image frame group 2220.

[0100] Also, for example, when the characteristic value is an average value, the average value of the color channel values for the image frame group can be obtained. More specifically, the average value of the G-R values for the first to 18th image frames included in the first image frame group 2210 can be obtained, and the average value of the G-R values for the 19th to 36th image frames included in the second image frame group 2220 can be obtained, but is not limited thereto.

[0101] Also, for example, when the characteristic value is a standard deviation value, the standard deviation value of the color channel values for the image frame group can be obtained. More specifically, the standard deviation value of the G-R values for the first to 18th image frames included in the first image frame group 2210 can be obtained, and the standard deviation value of the G-R values for the 19th to 36th image frames included in the second image frame group 2220 can be obtained, but is not limited thereto.

[0102] However, without being limited to the above-described examples, various characteristic values can be obtained for the image frame group.

[0103] Also, the characteristic value can be obtained for at least some of the image frames included in the image frame group divided by a window for the characteristic value. For example, the characteristic value can be obtained for the color channel values of at least some of the 18 image frames included in the first image frame group 2210, and can be obtained for the color channel values of at least some of the 18 image frames included in the second image frame group 2220.

[0104] Also, for example, when the characteristic value is a deviation value, the deviation value of the color channel values for at least some of the image frames included in the image frame group can be obtained. More specifically, the deviation value of the G-R value of the first image frame included in the first image frame group 2210 with respect to the average G-R value of the first image frame group can be obtained, and the deviation value of the G-R value of the 19th image frame included in the second image frame group 2220 with respect to the average G-R value of the second image frame group can be obtained, but it is not limited thereto.

[0105] Also, for example, when the characteristic value is a deviation value, the deviation value of the color channel values for at least some of the image frames included in the image frame group can be obtained. More specifically, the deviation value of the G-R value of the first image frame included in the first image frame group 2210 with respect to the average G-R value of the first image frame group can be obtained, and the deviation value of the G-R value of the second image frame included in the first image frame group 2210 can be obtained, but it is not limited thereto.

[0106] Also, the obtained characteristic value can be normalized.

[0107] For example, when the characteristic value is a deviation value, the deviation value can be normalized by a standard deviation value. More specifically, when a deviation value of the G-R value of the first image frame included in the first image frame group 2210 with respect to the average G-R value of the first image frame group 2210 is obtained, the deviation value of the G-R value of the first image frame can be normalized by the standard deviation value of the first image frame group 2210, but it is not limited thereto and can be normalized in various ways.

[0108] Also, when normalized in this way, the magnitude of the change amount is normalized, and the change of the value due to the heartbeat can be reflected better, and the noise due to the movement of the subject and the noise due to the change in the intensity of external light can be effectively reduced.

[0109] FIG. 11(a) is a graph showing two characteristic values obtained according to an embodiment. More specifically, it is a graph showing a first characteristic value obtained based on the G-R value and a second characteristic value obtained based on the G-B value. However, this is only shown specifically for convenience of explanation and is not limited thereto, and can be characteristic values obtained based on various color channel values, difference values, and processed values.

[0110] At this time, the first characteristic value obtained based on the G-R value may be affected by the G-R value. For example, when the external light is light close to the Blue channel, the G-R value may not be able to well reflect the change in blood due to the heartbeat.

[0111] Or for example, it can be affected by the difference between the absorbance of the Green channel and the absorbance of the Red channel and reflect the change in blood due to the heartbeat.

[0112] Also, the second characteristic value obtained based on the G-B value may be affected by the G-B value. For example, when the external light is light close to the Red channel, the G-B value may not be able to well reflect the change in blood due to the heartbeat.

[0113] Alternatively, for example, it can reflect the change in blood due to heartbeat under the influence of the difference in absorbance between the Green channel and the Blue channel.

[0114] Also, referring to FIG. 11(a), the first characteristic value and the second characteristic value can have a complementary relationship with each other. For example, in a section where the first characteristic value cannot well reflect the change due to heartbeat, the second characteristic value can well reflect the change due to heartbeat, and the opposite case is also included.

[0115] Therefore, the first characteristic value and the second characteristic value can be used to reduce noise due to changes in the wavelength of external light or to better reflect the change in blood due to heartbeat.

[0116] FIG. 11(b) is a graph showing a third characteristic value obtained by using the first characteristic value and the second characteristic value. More specifically, it is a graph showing a third characteristic value obtained by combining the first characteristic value and the second characteristic value. However, this is only shown specifically for convenience of explanation and is not limited thereto.

[0117] Also, the third characteristic value can be obtained based on the calculation of the first characteristic value and the second characteristic value. For example, the third characteristic value can be obtained based on the sum calculation of the first characteristic value and the second characteristic value, but it is not limited thereto and can be obtained based on various calculations such as difference calculation, product calculation, etc.

[0118] Also, the third characteristic value can be obtained by assigning various weighting values to the first characteristic value and the second characteristic value. For example, it can be obtained based on Equation 1 below, but it is not limited thereto.

[0119] (Equation 1) Third characteristic value = a · First characteristic value + b · Second characteristic value

[0120] Also, referring to FIGS. 11(a) and 11(b), the third characteristic value can better reflect the change in blood due to heartbeat than the first characteristic value and the second characteristic value, and can reduce noise due to changes in the wavelength of external light.

[0121] Hereinafter, as at least a part of the operation 505 of the learning server 20a according to various embodiments, an example of an operation for performing time synchronization for a plurality of biological signals will be described.

[0122] FIG. 12 is a flowchart showing an example of an operation for acquiring a non-contact biological signal of the learning server 20a according to various embodiments. The operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed and / or fewer operations may be performed. Hereinafter, with reference to FIGS. 13 to 14, FIG. 12 will be further described.

[0123] FIG. 13 is a drawing for explaining an example of a biological signal according to various embodiments. FIG. 14 is a drawing for explaining an example of a time synchronization operation of a biological signal of the learning server 20a according to various embodiments.

[0124] According to various embodiments, the learning server 20a (e.g., the data acquisition module 700) can perform time synchronization between a plurality of biological signals in operation 1201 and store the plurality of time-synchronized biological signals in operation 1203. For example, referring to FIG. 13, based on the fact that the positions of the specimens S1 and S2 from which the plurality of biological signals 1301 and 1301 are measured are different, each of the plurality of biological signals 1301 and 1303 can have different magnitudes (or intensities) or patterns over time from each other. For example, the second biological signal 1303 acquired by the measurement sensor 15 based on the contact of the specimen S2 (e.g., the finger of the first hand), which is relatively farther from the heart, can be delayed by a specific time td compared to the first biological signal 1301 acquired based on photographing the specimen S1 (e.g., the face) close to the heart of the user U. Accordingly, the learning server 20a can synchronize the times of the plurality of biological signals 1301 and 1303 based on the specific time td for more sophisticated learning. For example, referring to FIG. 14, the learning server 20a can perform an operation of deleting (or excluding) only the signal 1303a corresponding to the specific time td of the second biological signal 1303, or an operation of raising the time of the second biological signal 1303 by the specific time td. Also, for example, the learning server 20a can perform an operation (e.g., time shifting) of moving the first biological signal 1303 backward by the specific time td. On the other hand, although not shown, time synchronization for the third biological signal measured by the external measurement sensor 600 can also be performed.

[0125] According to various embodiments, the learning server 20a can select a reference biosignal from among a plurality of biosignals 1301 and 1303, and synchronize the remaining biosignals in time based on a specific time td that is identified with reference to the selected reference biosignal. For example, the learning server 20a can select, as a reference signal, the first biosignal 1301 related to the specimen S1 closest to the heart among the plurality of biosignals 1301 and 1303. The learning server 20a can identify the specific time td based on the distance difference between the specimen S2 related to the remaining second biosignal 1303 (and / or the biosignal measured by the external measurement sensor 600) and the specimen S2 corresponding to the first reference signal 1301, and perform the above-described time synchronization operation. On the other hand, although not shown, time synchronization for the third biosignal measured by the external measurement sensor 600 can also be performed in this manner.

[0126] According to various embodiments, the learning server 20a can determine the specific time td based on the personal characteristic information of the user. The learning server 20a can store information on a plurality of delay times, and can identify a specific time td corresponding to the personal characteristic information of the user among the plurality of delay times. As an example, the larger the height, the relatively longer specific time td can be selected.

[0127] According to various embodiments, the learning server 20a may implement an artificial intelligence model for the time synchronization, and perform the time synchronization based on the implemented artificial intelligence model.

[0128] Hereinafter, an example of an operation of generating an artificial intelligence model implemented to provide information on a specific type of biosignal having an accuracy sufficient to correspond to a contact type based on a specific type of biosignal obtained by a non-contact method of the learning server 20a according to various embodiments will be described. Hereinafter, an example in which the specific type of biosignal is PPG will be described, but the description is not limited to the described example, and an artificial intelligence model for measuring various types of biosignals can be implemented.

[0129] FIG. 15 is a flowchart showing an example of an operation of generating an artificial intelligence model embodied to provide information on a specific type of biological signal having an accuracy corresponding to a contact method based on a specific type of biological signal acquired by a non-contact method of a learning server 20a according to various embodiments. Operations may be performed regardless of the order of operations shown and / or described, and more operations may be performed and / or fewer operations may be performed. Hereinafter, FIG. 15 will be further described with reference to FIGS. 16 to 18b.

[0130] FIG. 16 is a drawing for explaining an example of an operation of generating an artificial intelligence model of the learning server 20a according to various embodiments. FIG. 17 is a drawing for explaining an example of at least one artificial intelligence model generated by the learning server 20a according to various embodiments. FIG. 18a is a drawing for explaining another example of at least one artificial intelligence model generated by the learning server 20a according to various embodiments. FIG. 18b is a drawing for explaining still another example of at least one artificial intelligence model generated by the learning server 20a according to various embodiments.

[0131] According to various embodiments, in operation 1501, the learning server 20a acquires a specific type of first biological signal based on photographing of a specimen S using the camera 12 of the electronic device 10, and acquires a second biological signal based on the first contact sensor 15. In operation 1503, the learning server 20a acquires the specific type of third biological signal based on the second contact sensor. In operation 1505, the learning server 20a can acquire the specific type of third biological signal based on the second contact sensor. Operations 1501 to 1505 of the learning server 20a may be embodied like operations 501 to 505 of the learning server 20a described above, and thus redundant descriptions are omitted.

[0132] According to various embodiments, the learning server 20a can obtain at least one artificial intelligence model for obtaining the specific type of biological signal based on a plurality of biological signals and the additional learning information in operation 1507. For example, referring to FIG. 16, the artificial intelligence model generation module 25a can generate at least one artificial intelligence model 1700a-1700f and 1800 based on a plurality of biological signals (MPPG1720, rPPG1740, PPG1750) and additional learning information 1730 (e.g., additional information 1731 and environmental data 1733) stored in the database 24a by the data acquisition module 700. At this time, the PPG data stored in the database 24a may or may not be time-synchronized with each other, and is not limited to the described example. The at least one artificial intelligence model 1700a-1700f and 1800 can be implemented to provide a PPG estimated value with accuracy corresponding to the PPG obtained by the contact method based on information (e.g., a plurality of images including the face) for a specimen obtained by the non-contact method. For example, the artificial intelligence model generation module 25a can implement an artificial intelligence model through supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, etc. Also, the artificial intelligence model generation module 25a according to an embodiment can implement an artificial intelligence model with an artificial neural network (ANN). For example, the artificial intelligence model generation module 25a can use, but is not limited to, a feedforward neural network, a radial basis function network, or a kohonen self-organizing network. Also, the artificial intelligence model generation module 25a according to an embodiment can implement an artificial intelligence model with a deep neural network (DNN).For example, the artificial intelligence model generation module 25a can utilize, but is not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), an LSTM (Long Short Term Memory Network), GRUs (Gated Recurrent Units), etc.

[0133] According to various embodiments, the learning server 20a may be embodied to learn models 1700a, 1700c, 1700e for obtaining non-contact PPG (rPPG) and models 1700b, 1700d, 1700f for obtaining contact PPG. Accordingly, PPG 1750 may be obtained by inputting rPPG 1740 output by inputting image information 1710 (e.g., a plurality of images) into the rPPG acquisition models 1700a, 1700c, 1700e into the second PPG acquisition models 1700b, 1700d, 1700f.

[0134] In one embodiment, referring to FIG. 17(a), the learning server 20a can be embodied to learn an artificial intelligence model (e.g., the first rPPG acquisition model 1700a) for acquiring remote photoplethysmography (rPPG) and an artificial intelligence model (e.g., the first PPG acquisition model 1700b) for acquiring contact PPG. For example, the learning server 20a can acquire the first rPPG acquisition model 1700a by using, as input data, the image information 1710 and additional learning information 1730 (e.g., additional information 1731 and environmental data 1733) related to each other and stored in the database 24a, and using rPPG 1740 as output data for learning. The first rPPG acquisition model 1700a can be embodied to output rPPG 1740 when the image information 1710 and the additional learning information 1730 (e.g., additional information 1731 and environmental data 1733) are input. Also, for example, the learning server 20a can acquire the first PPG acquisition model 1700b by using, as input data, rPPG 1740 and MPPG 1720 among the information stored in the database 24a, and using PPG 1750 as output data for learning. The second PPG acquisition model 1700b can be embodied to output PPG 1750 when rPPG 1740 and MPPG 1720 are input. The PPG 1750 can be acquired by inputting the rPPG 1740 output when the image information 1710 and the additional learning information 1730 (e.g., additional information 1731 and environmental data 1733) are input to the first rPPG acquisition model 1700a and the MPPG 1720 acquired by the electronic device 10 into the second PPG acquisition model 1700b and outputting the result.

[0135] Also, in one embodiment, referring to FIG. 17(b), the learning server 20a can be embodied to learn a model (e.g., the second rPPG acquisition model 1700c) for acquiring non-contact PPG (rPPG) and an artificial intelligence model (e.g., the second PPG acquisition model 1700d) for acquiring contact PPG. For example, the learning server 20a uses the image information 1710 captured by the camera 12 as input data and rPPG1740 as output data to learn among the related information stored in the database 24a, thereby obtaining the second rPPG acquisition model 1700c. The second rPPG acquisition model 1700c can be embodied to output rPPG1740 when the image information 1710 is input. On the other hand, without being limited to the described example, the second rPPG acquisition model 1700c can be software and / or an algorithm that obtains a third characteristic value based on G - R and G - B as described above with reference to FIG. 9. Also, for example, the learning server 20a uses rPPG1740, additional learning information 1730, and MPPG1720 as input data and PPG1750 as output data to learn among the information stored in the database 24a, thereby obtaining the second PPG acquisition model 1700d. The second PPG acquisition model 1700d can be embodied to output PPG1750 when rPPG1740, additional learning information 1730, and MPPG1720 are input. The PPG1750 can be obtained by inputting the rPPG1740 output by inputting the image information 1710 into the second rPPG acquisition model 1700c, the MPPG1720 obtained by the electronic device 10, and the additional learning information 1730 (e.g., additional information 1731, environmental data 1733) into the second PPG acquisition model 1700d and outputting the result.

[0136] Also, in one embodiment, referring to FIG. 17(c), the learning server 20a may be embodied to learn an artificial intelligence model (e.g., the 3rd rPPG acquisition model 1700e) for acquiring remote photoplethysmography (rPPG) and an artificial intelligence model (e.g., the 3rd PPG acquisition model 1700f) for acquiring contact PPG. For example, the learning server 20a may use, as input data, the image information 1710, MPPG 1720, and additional learning information 1730 (e.g., additional information 1731, environmental data 1733) related to each other and stored in the database 24a, and learn with the rPPG 1740 as output data to acquire the 3rd rPPG acquisition model 1700e. The 3rd rPPG acquisition model 1700e may be embodied to output the rPPG 1740 when the image information 1710, MPPG 1720, and additional learning information 1730 (e.g., additional information 1731, environmental data 1733) are input. Also, for example, the learning server 20a may learn with the rPPG 1740 as input data and the PPG 1750 as output data among the information stored in the database 24a to acquire the 3rd PPG acquisition model 1700d. The 3rd PPG acquisition model 1700f may be embodied to output the PPG 1750 when the rPPG 1740 is input. The PPG 1750 may be acquired by inputting the rPPG 1740 output by inputting the image information 1710, MPPG 1720, and additional learning information 1730 (e.g., additional information 1731, environmental data 1733) to the 3rd rPPG acquisition model 1700e into the 3rd PPG acquisition model 1700f and outputting the result.

[0137] According to various embodiments, the learning server 20a may learn a single integrated artificial intelligence model (e.g., the first integrated artificial intelligence model 1800) for acquiring PPG. For example, referring to FIG. 18a, the learning server 20a can acquire the first integrated artificial intelligence model 1800 by learning with image information 1710, MPPG 1720, and additional learning information 1730 as input data and PPG 1740 as output data. The first-1 integrated artificial intelligence model 1800a can be implemented to output PPG 1750 when image information 1710, MPPG 1720, and additional learning information 1730 (e.g., additional information 1731, environmental data 1733) are input. On the other hand, without being limited to the described examples, the first-1 integrated artificial intelligence model 1800a can be implemented to output rPPG 1720 when image information 1710, MPPG 1720, and additional learning information 1730 (e.g., additional information 1731, environmental data 1733) are input.

[0138] According to various embodiments, the learning server 20a may learn a single integrated artificial intelligence model (e.g., the first integrated artificial intelligence model 1800) for acquiring PPG. For example, referring to FIG. 18b, the learning server 20a can acquire the first-2 integrated artificial intelligence model 1800b by learning with image information 1710, PPG 1750, and additional learning information 1730 as input data and rPPG 1740 or MPPG 1720 as output data. The first-2 integrated artificial intelligence model 1800b can be implemented to output MPPG 1720 or rPPG 1740 when image information 1710, PPG 1750, and additional learning information 1730 (e.g., additional information 1731, environmental data 1733) are input.

[0139] Hereinafter, as at least a part of the 1507 operation of the learning server 20a according to various embodiments, an example of an operation of generating an artificial intelligence model without time synchronization will be described.

[0140] FIG. 19 is a flowchart showing an example of an operation of generating an artificial intelligence model without time synchronization of the learning server 20a according to various embodiments. Operations may be performed regardless of the order of operations shown and / or described, and more operations may be performed and / or fewer operations may be performed.

[0141] According to various embodiments, in operation 1901, the learning server 20a can acquire a plurality of biological signals (e.g., rPPG, MPPG, PPG) as training data without performing time synchronization between a plurality of different biological signals (e.g., rPPG, MPPG, PPG). Accordingly, the learning server 20a can perform an operation of learning at least one of the aforementioned artificial intelligence models 1700a to 1700f and 1800 based on a plurality of biological signals (e.g., rPPG, MPPG, PPG) for which time synchronization has not been performed. Accordingly, at least one of the implemented artificial intelligence models 1700a to 1700f and 1800 can be implemented to output information on highly accurate PPG even when information acquired by the electronic device 10 in the future is input in a state where time synchronization is not performed.

[0142] The following is a drawing for explaining an example of an operation of generating still other artificial intelligence models of the learning server 20a according to various embodiments. In the following, an example in which the specific type of biological signal is PPG will be described, but the description is not limited to the described example, and an artificial intelligence model for measuring various types of biological signals may be implemented.

[0143] According to various embodiments, the time difference td between the plurality of biological signals (e.g., rPPG, MPPG, PPG) may be used to measure body information such as blood pressure.

[0144] FIG. 20 is a flowchart showing an example of the generation operation of still another artificial intelligence model of the learning server 20a according to various embodiments. Operations may be performed regardless of the order of operations shown and / or described, and more operations may be performed and / or fewer operations may be performed. Hereinafter, FIG. 20 will be further described with reference to FIGS. 21 to 23.

[0145] FIG. 21 is a drawing for explaining an example of an operation of using rPPG and PPG for the learning server 20a to generate an artificial intelligence model according to various embodiments. FIG. 22 is a drawing for explaining an example of an operation of using MPPG and PPG for the learning server 20a to generate an artificial intelligence model according to various embodiments. FIG. 23 is a drawing for explaining an example of at least one artificial intelligence model generated by the learning server 20a according to various embodiments.

[0146] According to various embodiments, the learning server 20a acquires a first type of biological signal based on photographing of a specimen S using the camera 12 of the electronic device 10 in operation 2001, acquires a second biological signal based on a contact sensor (e.g., external measurement sensor 600, contact sensor 15) in operation 2003, and can acquire the additional learning information in operation 2005. For example, the learning server 20a does not acquire all of rPPG, PPG, and MPPG for the generation of the artificial intelligence model, but as shown in FIG. 21, can acquire rPPG and PPG, or as shown in FIG. 22, can acquire rPPG and MPPG. Other overlapping explanations are omitted.

[0147] According to various embodiments, the learning server 20a can acquire at least one of models 2300a, 2300b, 2300c, 2300d, 2300e for acquiring the specific type of biological signal based on at least a part of the plurality of biological signals and the additional learning information in operation 2007. For example, the learning server 20a can generate at least one artificial intelligence model that is configured to output information about MPPG or information about PPG as a result.

[0148] Also, in one embodiment, referring to FIG. 23(a), the learning server 20a can be embodied to learn an artificial intelligence model (e.g., the 4th rPPG acquisition model 2300a) for acquiring non-contact PPG (rPPG) and an artificial intelligence model (e.g., the 4th PPG acquisition model 2300b) for acquiring PPG2301 (e.g., MPPG, or PPG). For example, the learning server 20a can acquire the 4th rPPG acquisition model 2300a by using the image information 1710 captured by the camera 12 as input data and rPPG1740 as output data for learning among the related information stored in the database 24a. The 4th rPPG acquisition model 2300a can be embodied to output rPPG1740 when the image information 1710 is input. On the other hand, without being limited to the described example, the 4th rPPG acquisition model 2300a can be software and / or an algorithm that acquires a third characteristic value based on G-R and G-B as described above with reference to FIG. 9. Also, for example, the learning server 20a can acquire the 4th PPG acquisition model 2300b by using rPPG1740 and additional learning information 1730 as input data and PPG2301 (e.g., MPPG, or PPG) as output data for learning among the information stored in the database 24a. The 4th PPG acquisition model 2300b can be embodied to output PPG2301 (e.g., MPPG, or PPG) when rPPG1740 and additional learning information 1730 (e.g., additional information 1731, environmental data 1733) are input. PPG2301 (e.g., MPPG, or PPG) can be acquired when additional learning information 1730 (e.g., additional information 1731, environmental data 1733) is input to the 4th PPG acquisition model 2300b together with rPPG1740 output by inputting the image information 1710 to the 4th rPPG acquisition model 2300a.

[0149] Also, in one embodiment, referring to FIG. 23(a), the learning server 20a can be embodied to learn an artificial intelligence model (e.g., the 4th rPPG acquisition model 2300a) for acquiring remote photoplethysmography (rPPG) and an artificial intelligence model (e.g., the 4th PPG acquisition model 2300b) for acquiring PPG2301 (e.g., MPPG, or PPG). For example, the learning server 20a can acquire the 4th rPPG acquisition model 2300a by using, as input data, the image information 1710 captured by the camera 12 among the information related to each other stored in the database 24a and learning with the rPPG1740 as output data. The 4th rPPG acquisition model 2300a can be embodied to output the rPPG1740 when the image information 1710 is input. Also, for example, the learning server 20a can acquire the 4th PPG acquisition model 2300b by using, as input data, the rPPG1740 and additional learning information 1730 among the information stored in the database 24a and learning with the PPG2301 (e.g., MPPG, or PPG) as output data. The 4th PPG acquisition model 2300b can be embodied to output the PPG2301 (e.g., MPPG, or PPG) when the rPPG1740 is input. The PPG2301 (e.g., MPPG, or PPG) output by inputting the rPPG1740 output by inputting the image information 1710 into the 4th rPPG acquisition model 2300a can be acquired by inputting it into the 4th PPG acquisition model 2300b.

[0150] Also, in one embodiment, referring to FIG. 23(b), the learning server 20a can be embodied to learn an artificial intelligence model (e.g., the fifth rPPG acquisition model 2300c) for acquiring remote photoplethysmography (rPPG) and an artificial intelligence model (e.g., the fifth PPG acquisition model 2300d) for acquiring PPG2301 (e.g., MPPG, or PPG). For example, the learning server 20a uses, as input data, the image information 1710 and additional learning information 1730 captured by the camera 12 among the information related to each other stored in the database 24a, and learns with rPPG1740 as output data, thereby obtaining the fifth rPPG acquisition model 2300c. The fifth rPPG acquisition model 2300c can be embodied to output rPPG1740 when the image information 1710 and additional learning information 1730 (e.g., additional information 1731, environmental data 1733) are input. Also, for example, the learning server 20a uses, as input data, rPPG1740 among the information stored in the database 24a, and learns with PPG2301 (e.g., MPPG, or PPG) as output data, thereby obtaining the fifth PPG acquisition model 2300d. The fifth PPG acquisition model 2300d can be embodied to output PPG2301 (e.g., MPPG, or PPG) when rPPG1740 is input. PPG2301 (e.g., MPPG, or PPG) can be obtained by inputting rPPG1740 output by inputting the image information 1710 to the fifth rPPG acquisition model 2300c into the fifth PPG acquisition model 2300d and outputting it.

[0151] According to various embodiments, the learning server 20a may learn a single integrated artificial intelligence model (e.g., the second integrated artificial intelligence model 2300e) for acquiring PPG. For example, referring to FIG. 23(c), the learning server 20a uses the image information 1710, MPPG 1720, and additional learning information 1730 as input data and learns with PPG 2301 (e.g., MPPG, or PPG) as output data, thereby acquiring the second integrated artificial intelligence model 2300e. The second integrated artificial intelligence model 2300e may be embodied to output PPG 2301 (e.g., MPPG, or PPG) when the image information 1710, MPPG 1720, and additional learning information 1730 (e.g., additional information 1731, environmental data 1733) are input.

[0152] The following are drawings for explaining an example of an operation of providing a biological signal with accuracy similar to that of a contact method based on information acquired by a non-contact method using an artificial intelligence model of the electronic device 10 according to various embodiments. In the following, an example in which the specific type of biological signal is PPG will be described, but the present invention is not limited to the described example, and an artificial intelligence model for measuring various types of biological signals may be embodied.

[0153] FIG. 24 is a flowchart for explaining an example of an operation of providing a biological signal with accuracy similar to that of a contact method using an artificial intelligence model of the electronic device 10 according to various embodiments. The operations may be performed regardless of the illustrated and / or described order of operations, and more operations may be performed and / or fewer operations may be performed. In the following, FIG. 24 will be further described with reference to FIG. 25.

[0154] FIG. 25 is a drawing for explaining an operation of providing a biological signal with accuracy similar to that of a contact method using an artificial intelligence model of the electronic device 10 according to various embodiments.

[0155] According to various embodiments, the electronic device 10 can execute an application in operation 2401. For example, the application can be embodied to provide information on a biosignal having the accuracy of a contact type and / or biometric information (e.g., blood pressure, blood sugar, etc.) analyzed based on the biosignal, based on obtaining information on a part of the body of the user U (i.e., a specimen) in a non-contact manner.

[0156] According to various embodiments, the electronic device 10 can obtain additional learning information in operation 2403. For example, the electronic device 10 can obtain personal information as additional learning information. An execution screen of an application for inputting information on the personal information (e.g., gender, year, age, race, etc.) can be displayed, the characteristic information of the user input through the execution screen can be stored, and / or transmitted to the server 20 (e.g., the usage server 20b). The execution screen of the application can be an execution screen provided at the time of the user's enrollment and / or an execution screen for inputting the user's personal information. On the other hand, without being limited to the described examples, the electronic device 10 can obtain camera information, shooting information, and / or image information as additional learning information.

[0157] According to various embodiments, in operation 2405, the electronic device 10 can acquire a plurality of images by using the camera 12 of the electronic device 10, and in operation 2407, can acquire sensing data by using the touch sensor 15 of the electronic device 10. For example, as illustrated in FIG. 21, the electronic device 10 can display an execution screen 2501 of an application for guiding a user to touch a part of the body (e.g., a finger) to the touch sensor 15 while photographing the user's face by using the camera 12. The execution screen 2501 can include an area 2501a where the user's face is displayed, an area 2501b where text for guiding (e.g., "Touch your finger to the sensor and take a picture of your face.") is displayed, and an area 2501c where information about the user's photographing environment (e.g., the distance, position, inclination, illuminance, etc. between the electronic device 10 and the face) is displayed. Accordingly, the electronic device 10 can acquire a plurality of images including the user's face by using the camera 12, acquire sensing data by using the touch sensor 15 disposed at the back, and acquire additional learning information (e.g., environmental data acquired by the environmental sensor 16 and additional information) during photographing. On the other hand, as described above, the additional information can include personal information, camera information, photographing information, and / or image information. The electronic device 10 can transmit the acquired information to the server 20 (e.g., the usage server 20b).

[0158] According to various embodiments, the application can be implemented to drive the camera 12, the touch sensor 15, and the environmental sensor 14 of the electronic device 10 during execution and have permissions for each of them.

[0159] According to various embodiments, in operation 2409, the electronic device 10 can acquire a specific type of biological signal based on the plurality of images, the sensing data, and the additional learning information, and in operation 2411, can acquire at least one piece of biological information corresponding to the specific type of biological signal. For example, as described above, the utilization server 20b can acquire the finally output PPG as a response to inputting the received information to at least one learned artificial intelligence model (e.g., artificial intelligence models 1700a to 1700f in FIG. 17, artificial intelligence models 1800a to 1800b in FIGS. 18a to 18b, artificial intelligence models 2300a to 2300e in FIG. 23). As a result, the PPG can have the accuracy of a contact type method. Also, the utilization server 20b can acquire the user's body information (e.g., blood pressure, heart rate information) based on the PPG. As a result, the electronic device 10 can receive information regarding the PPG and / or the user's body information (e.g., blood pressure) from the utilization server 20b and display it on the execution screen 2503 of the application. For example, the execution screen 2503 can include information 2503a regarding the heart rate and information 2503b regarding the blood pressure.

[0160] The following are drawings for explaining an example of an operation for guiding photographing in at least a part of the 2403 operations of the electronic device 10 according to various embodiments.

[0161] FIG. 26 is a flowchart for explaining an operation for guiding photographing of the electronic device 10 according to various embodiments. Operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed and / or fewer operations may be performed. Hereinafter, with reference to FIG. 27, FIG. 26 will be further described.

[0162] FIG. 27 is a drawing for explaining an example of an operation for guiding photographing of the electronic device 10 according to various embodiments.

[0163] According to various embodiments, in operation 2601, the electronic device 10 can display an execution screen of an application for photographing. For example, as illustrated in FIG. 27, the electronic device 10 can provide an execution screen 2701 of an application for photographing.

[0164] According to various embodiments, in operation 2603, the electronic device 10 determines whether a specific condition is satisfied. If the specific condition is satisfied (2603 - Y), in operation 2605, photographing can be performed with at least one camera parameter (e.g., shutter speed, FPS, shooting resolution, etc.) set to a specific value. For example, in at least a part of the operation of determining whether the specific condition is satisfied, the electronic device 10 can use the environmental sensor 16 of the electronic device 10 to determine whether information related to the surrounding environment to be photographed (e.g., illuminance, etc.) and / or information related to the state of the electronic device 10 during photographing (e.g., position, depression period, etc.) satisfies the specific condition. As an example, as illustrated in FIG. 27, the electronic device 10 can determine whether the position of the electronic device 10 satisfies a specific position (e.g., the second position). The electronic device 10 can perform photographing using the camera 12 when the position of the electronic device 10 is a specific position (e.g., the second position). Accordingly, the deviation of the image obtained by the non - contact method can be reduced, and the accuracy of PPG can be further improved.

[0165] According to various embodiments, as illustrated in FIG. 27, the electronic device 10 can update and provide information about the position (e.g., the first position and the second position) on the execution screen 2701 of the application for photographing so that the user can recognize it.

[0166] According to various embodiments, at least one camera parameter (e.g., shutter speed, FPS, etc.) can be set as a specific value during the photographing. For example, the camera parameters can be set so that a video is photographed with the FPS in the range of 20 to 30 frames per second.

[0167] The following are diagrams for explaining examples of operations of the server 20 (e.g., usage server 20b) using an artificial intelligence model according to various embodiments.

[0168] FIG. 28 is a flowchart for explaining an example of an operation of the server 20 (e.g., usage server 20b) using an artificial intelligence model according to various embodiments. Operations may be performed regardless of the order of operations shown and / or described, and more operations may be performed and / or fewer operations may be performed. Hereinafter, FIG. 28 will be further described with reference to FIG. 29.

[0169] FIG. 29 is a diagram for explaining an example of an operation of the server 20 (e.g., usage server 20b) using an artificial intelligence model according to various embodiments.

[0170] According to various embodiments, the usage server 20b can obtain at least one input data corresponding to each of a plurality of artificial intelligence models in operation 2801, and can obtain a plurality of biological signals of a specific type as a response to inputting at least one input data to each of the plurality of artificial intelligence models in operation 2803. For example, the usage server 20b can store at least one learned artificial intelligence model described above (e.g., artificial intelligence models 1700a to 1700f in FIG. 17, artificial intelligence models 1800a to 1800b in FIGS. 18a to 18b, artificial intelligence models 2300a to 2300e in FIG. 23). At this time, the usage server 20b can obtain a plurality of PPGs output by inputting input data related to each artificial intelligence model (e.g., artificial intelligence models 1700a to 1700f in FIG. 17, artificial intelligence models 1800a to 1800b in FIGS. 18a to 18b, artificial intelligence models 2300a to 2300e in FIG. 23) based on the image information 1710, MPPG 1720, and / or additional learning information 1730 received from the electronic device 10.

[0171] According to various embodiments, the usage server 20b can acquire a specific biometric signal of the specific type based on the plurality of biometric signals of the specific type in operation 2805. In one embodiment, the usage server 20b can select a specific PPG determined as the most reliable among the plurality of PPGs and provide information and / or body information for the specific PPG to the electronic device 10. Also, in one embodiment, the usage server 20b can acquire information for a specific PPG by performing a predetermined calculation (e.g., average) based on the plurality of PPGs and provide information and / or body information for the specific PPG to the electronic device 10. Also, in one embodiment, the usage server 20b can select a specific PPG output by an artificial intelligence model that best suits the user among the plurality of PPGs and provide information and / or body information for the specific PPG to the electronic device 10.

Claims

1. An electronic device, comprising: a communication circuit; and at least one processor; wherein the at least one processor is configured to: acquire a plurality of images including a user's face acquired using a camera of a first external electronic device through the communication circuit; acquire shooting information related to shooting acquired by the first external electronic device while acquiring the plurality of images through the communication circuit; acquire a specific type of first biometric signal acquired based on a first sensor of the first external electronic device contacted by a first finger of a first hand of the user and a specific type of second biometric signal acquired based on a second external electronic device contacted by a second finger corresponding to the first finger of the second hand of the user while acquiring the plurality of images through the communication circuit; acquire personal information related to the user's personal characteristics through the communication circuit; be configured to acquire at least one artificial intelligence model by performing learning using the plurality of images, the shooting-related information, the first biometric signal, the second biometric signal, and the personal information as training data; the at least one artificial intelligence model is embodied to provide information on the second biometric signal as a response to receiving an input of the plurality of images, the shooting-related information, information on the first biometric signal acquired based on the first sensor of the first external electronic device, and the personal information.

2. The electronic device according to claim 1, wherein the shooting information includes information related to a state related to shooting of the first external electronic device and information related to an external environment of the first external electronic device.

3. The at least one processor is configured to: perform time synchronization of the first biometric signal and the second biometric signal, the electronic device according to claim 1 or 2.

4. The at least one processor is configured to: perform time synchronization of the first biometric signal and the second biometric signal based on a biometric signal identified based on the plurality of images, the electronic device according to claim 3.

5. The at least one processor is configured to: generate the at least one artificial intelligence model in a state where the time synchronization is not performed, the electronic device according to claim 3.

6. A method for operating an electronic device, comprising: acquiring, through the communication circuit of the electronic device, a plurality of images including a user's face acquired using a camera of a first external electronic device; acquiring, through the communication circuit, shooting information related to shooting acquired by the first external electronic device while acquiring the plurality of images; acquiring, through the communication circuit, a first specific type of biometric signal acquired based on a first sensor of the first external electronic device contacted by a first finger of a first hand of the user and a second specific type of biometric signal acquired based on a second external electronic device contacted by a second finger corresponding to the first finger of the second hand of the user while acquiring the plurality of images; acquiring, through the communication circuit, personal information related to the user's personal characteristics; and acquiring at least one artificial intelligence model by performing learning using the plurality of images, the information related to shooting, the first biometric signal, the second biometric signal, and the personal information as training data; wherein the at least one artificial intelligence model is configured to provide information on the second biometric signal as a response to receiving an input of the plurality of images, the information related to shooting, the information on the first biometric signal acquired based on the first sensor of the first external electronic device, and the personal information, Method of operation.

7. The method of operation according to claim 6, wherein the shooting information related to shooting of the electronic device includes information related to a state related to shooting of the first external electronic device and information related to an external environment of the first external electronic device.

Images