Electronic device for stable rPPG signal measurement under shooting conditions, and its operating method.

Abstract

How an electronic device for estimating pulse rate works. [Solution] An electronic device operation method includes: acquiring multiple images of the user captured using the electronic device's RGB camera; acquiring a specific region for a specific body part of the user from each of the multiple images; generating multiple first data related to an RGB color model for the specific region associated with each of the multiple images; generating multiple second data related to a YCrCb color model based on the multiple first data; generating first time series data related to the green channel based on the multiple first data; generating second time series data related to the color difference channel based on the multiple second data; generating third time series data based on adding the first and second time series data; and estimating the user's pulse rate based on converting the third time series data to the frequency domain.

Description

Technical Field

[0001] The present disclosure relates to an electronic device for providing a plurality of color model-based rPPG measurements and services, and an operation method thereof.

Background Art

[0002] The most common technique for measuring Photoplethysmography (PPG) using light is to analyze the amount of transmitted light with respect to the light irradiated on the human body, and is explained by the Beer-Lambert law that the absorbance is proportional to the concentration of the absorbing substance and the thickness of the absorption layer. According to this law, since the change in transmitted light results in a signal proportional to the change in the volume of the substance being transmitted, the state of the heart can be grasped using PPG even when the absorbance of the substance is unknown.

[0003] Recently, a technology using rPPG (remote Photoplethysmography) has emerged, which is a step further from the technology using PPG. As the most popular technology for grasping signals related to heart beats using PPG, there is a technology that irradiates light by directly contacting a device with a camera and illumination attached in close proximity, such as a smartphone, to the human body and immediately measures the transmitted light to obtain PPG. Recently, technologies related to rPPG (remote Photoplethysmography) for grasping changes in the volume of blood vessels from signals obtained from images taken with a camera have been continuously studied and developed.

[0004] The technology using rPPG can be variously applied in devices and places equipped with cameras, such as immigration control offices at airports and telemedicine, in that no contact between the subject and the measuring equipment is required.

[0005] However, since noise generated by ambient light and the movement of the object during the process of capturing the object with a camera has a significant impact on the signal in rPPG technology, the core technology in measuring biological signals using rPPG can be seen as the technique of extracting only the signal related to the volume change of the object being measured from the captured image. [Overview of the project] [Problems that the invention aims to solve]

[0006] Conventional methods for measuring rPPG typically involve analyzing an image of an object captured using a camera. In this case, rPPG can be measured based on the pixel values ​​of the captured image extracted using various color models such as the RGB color model, YCbCrCg color model, and CIELa*b* color model. However, during camera capture, noise values ​​due to the shooting environment (e.g., shaking, external illumination, shadows, etc.) are reflected in the pixel values, making it difficult to accurately measure the rPPG of an object. According to various embodiments, the electronic device and its operating method can improve the accuracy of rPPG measurement for an object by using multiple color models, including the RGB color model, YCbCrCg color model, and CIELa*b* color model, in a mutually complementary manner to reduce noise values. Furthermore, according to various embodiments, the electronic device and its operating method can improve the accuracy of rPPG measurement for an object based on color model values ​​and IR values ​​by simultaneously using an RGB camera and an IR camera.

[0007] Recently, personnel control kiosks placed at vehicle or worker entrances have been technologically developed to provide diverse services based on biometric information measured using rPPG. However, due to the low accuracy of rPPG and a poor understanding of how to link the services provided by vehicles and personnel control kiosks, the services offered are not highly practical. According to various embodiments, electronic devices and their operating methods can provide practically practical vehicle and personnel control kiosk services based on high-quality rPPG that can be measured in all environments. [Means for solving the problem]

[0008] According to various embodiments, a method for operating an electronic device may be provided, which includes: acquiring a plurality of images of a user captured using the RGB camera of the electronic device; acquiring a specific region of a specific body part of the user from each of the plurality of images; generating a plurality of first data related to an RGB color model for the specific region associated with each of the plurality of images; generating a plurality of second data related to a YCrCb color model based on the plurality of first data; generating first time series data related to the green channel based on the plurality of first data; generating second time series data related to the color difference channel based on the plurality of second data; generating third time series data based on adding the first time series data and the second time series data; and estimating the pulse rate of the user based on converting the third time series data to the frequency domain.

[0009] According to various embodiments, an electronic device may be provided, comprising at least one processor, the at least one processor configured to: acquire a plurality of images of a user captured using an RGB camera; acquire a specific region of a specific body part of the user from each of the plurality of images; generate a plurality of first data related to an RGB color model for the specific region associated with each of the plurality of images; generate a plurality of second data related to a YCrCb color model based on the plurality of first data; generate first time series data related to the green channel based on the plurality of first data; generate second time series data related to the color difference channel based on the plurality of second data; generate third time series data based on adding the first time series data and the second time series data; and estimate the pulse rate of the user based on converting the third time series data to the frequency domain.

[0010] The means of solving the problems in the various embodiments are not limited to the means described above, and means of solving that are not mentioned can be clearly understood by a person skilled in the art to which the present invention pertains from this specification and the accompanying drawings. [Effects of the Invention]

[0011] According to various embodiments, an electronic device and its operating method can be provided that improves the accuracy of rPPG measurement for an object by measuring rPPG using multiple color models, including RGB color models, YCbCrCg color models, and CIELa*b* color models, in a mutually complementary manner to reduce noise values.

[0012] Furthermore, according to various embodiments, an electronic device and its operating method may be provided that improves the accuracy of rPPG measurement for an object based on color model values ​​and IR values ​​by simultaneously utilizing an RGB camera and an IR camera.

[0013] Furthermore, according to various embodiments, electronic devices and methods of operation can be provided that offer substantially usable vehicle and personnel control kiosk services based on high-quality rPPG measurable in all environments. [Brief explanation of the drawing]

[0014] [Figure 1] This diagram shows examples of components of a non-contact biometric information system according to various embodiments. [Figure 2] This diagram shows yet another example of the components of a non-contact biometric information system according to various embodiments. [Figure 3] This block diagram shows examples of server components related to various embodiments. [Figure 4] This is a block diagram showing examples of components of a user device relating to various embodiments. [Figure 5] This flowchart illustrates examples of how electronic devices for rPPG and biometric information measurement operate based on multiple color models relating to various embodiments. [Figure 6] This is a diagram illustrating an example of a module that performs operations for rPPG and biological information measurement using a visible light camera according to various embodiments. [Figure 7] This diagram illustrates an example of the operation for preprocessing RGB data related to various embodiments. [Figure 8] This diagram illustrates an example of operation utilizing multiple color models related to various embodiments. [Figure 9] This flowchart illustrates an example of how an electronic device for measuring rPPG and biometric information operates based on multiple region-specific IR channel values ​​related to various embodiments. [Figure 10] This is a diagram illustrating an example of a module that performs operations for rPPG and biological information measurement using an infrared camera according to various embodiments. [Figure 11] This diagram illustrates an example of operation that utilizes the values ​​of multiple region-specific IR channels related to various embodiments. [Figure 12] A flowchart for explaining an example of an operation method of an electronic device for measuring rPPG using a visible light camera and an IR camera based on shooting environment conditions according to various embodiments. [Figure 13] A drawing for explaining an example of an embodiment for measuring rPPG using a visible light camera and an IR camera according to various embodiments. [Figure 14] A flowchart for explaining an example of an operation method of an electronic device for measuring blood pressure AI model-based blood pressure according to various embodiments. [Figure 15] A flowchart for explaining an example of an operation method of an electronic device for measuring stress analysis AI model-based stress according to various embodiments. [Figure 16] A drawing for explaining an example of a module for performing a stress analysis operation according to various embodiments. [Figure 17] A drawing for explaining an example of a transportation means for providing a service based on non-contact biometric information according to various embodiments. [Figure 18] A flowchart for explaining an example of an operation method of an electronic device for providing a non-contact biometric information and external information-based mobility service according to various embodiments. [Figure 19] A drawing for explaining an example of a mobility service provided by non-contact biometric information and external information according to various embodiments. [Figure 20] A flowchart for explaining an example of an operation method of an electronic device for providing a mobility service that further considers passenger information according to various embodiments. [Figure 21] A drawing for explaining an example of measuring rPPG / biometric information for a passenger according to various embodiments. [Figure 22] A flowchart for explaining an example of an operation method of an electronic device for providing a mobility service that further considers the mode of a vehicle according to various embodiments. [Figure 23]This is a diagram illustrating examples of mobility services that continuously change the shape of a seat during autonomous driving, according to various embodiments. [Figure 24] This flowchart illustrates an example of how an electronic device operates to provide a sustainable bio-information storage platform service related to various embodiments. [Figure 25a] This diagram illustrates an embodiment in which biological information measured for each driving path over a predetermined period of time related to various embodiments is accumulated in a database. [Figure 25b] This is a diagram illustrating an example of a service that provides driver recommendation information based on accumulated biological data related to various embodiments. [Figure 26] This is a diagram illustrating an example of a kiosk that provides contactless biometric information-based services in various embodiments. [Figure 27] This flowchart illustrates an example of how an electronic device operates to provide multiple location-based bio-information infrastructure services related to various embodiments. [Figure 28] This diagram illustrates an example of collecting biometric information from a specific user from multiple kiosks located at different locations in various embodiments. [Figure 29] This is a diagram illustrating examples of services based on multiple location-specific biometric information related to various embodiments. [Modes for carrying out the invention]

[0015] The various embodiments and terminology used in this document are not intended to limit the technical features described herein to any particular embodiment, but should be understood to include various modifications, equivalents, or substitutes for the embodiments in question. In relation to the description of the drawings, similar or related reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items unless the context clearly indicates otherwise. In this document, each phrase such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B or C" may include any one of the items listed together in the phrase in question, or any possible combination thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish the component in question from other components in question, and do not limit the component in question in any other respect (e.g., importance or order). When a component (e.g., the first) is referred to as "coupled" or "connected" with or without the terms "functionally" or "communically" to another component (e.g., the second), it means that the first component can be connected to the other component directly (e.g., by wire), wirelessly, or through the third component.

[0016] The term "module," as used in the various embodiments of this document, can include units embodied in hardware, software, or firmware, and can be used interchangeably with terms such as logic, logic block, component, or circuit. A module can be a component configured as a whole or the smallest unit or part of such component that performs one or more functions. For example, according to one embodiment, a module can be embodied in the form of an ASIC (application-specific integrated circuit).

[0017] The various embodiments of this document may be embodied in software (e.g., a program) containing one or more instruction sets stored in a storage medium (e.g., internal memory) or external memory) readable by a machine (e.g., an electronic device). For example, the processor (e.g., a processor) of the machine (e.g., an electronic device) can invoke and execute at least one instruction set from the storage medium. This allows the machine to be operated to perform at least one function by the invoked instruction set. The one or more instruction sets may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily on the storage medium.

[0018] According to one embodiment, the methods relating to the various embodiments disclosed herein may be provided in a computer program product. The computer program product may be traded as a commodity between sellers and buyers. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or online (e.g., downloaded or uploaded) through an application store (e.g., PlayStore™), or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily generated on a device-readable storage medium such as the memory of a manufacturer's server, an application store server, or an intermediary server.

[0019] According to various embodiments, each component of the aforementioned components (e.g., a module or program) may contain one or more individuals, some of which may be separated and placed in other components. According to various embodiments, one or more components or operations within the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., a module or program) may be integrated into a single component. In such cases, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the respective components within the multiple components prior to the integration. According to various embodiments, operations performed by modules, programs or other components may be executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0020] According to various embodiments, an operating method for an electronic device may be provided, which includes: acquiring a plurality of images of a user captured using the RGB camera of the electronic device; acquiring a specific region of a specific body part of the user from each of the plurality of images; generating a plurality of first data related to an RGB color model for the specific region associated with each of the plurality of images; generating a plurality of second data related to a YCrCb color model based on the plurality of first data; generating first time series data related to the green channel based on the plurality of first data; generating second time series data related to the color difference channel based on the plurality of second data; generating third time series data based on adding the first time series data and the second time series data; and estimating the pulse rate of the user based on converting the third time series data to the frequency domain.

[0021] According to various embodiments, the operation of obtaining a specific region for a specific body part of the user from each of the multiple images may include: identifying a value for a specific color model for each of the multiple images; and extracting a region from the identified value that is greater than or equal to a preset value.

[0022] According to various embodiments, an operation method may be provided in which the operation to generate a plurality of first data related to the RGB color model includes: an operation to obtain a plurality of values ​​for a plurality of color channels for the plurality of image-specific RGB color models, each of which corresponds to a specific point in time; an operation to identify a plurality of interval values ​​for a plurality of time windows among the plurality of values ​​for the plurality of color channels; and an operation to generate the plurality of first data by adjusting the plurality of values ​​based on the average of the plurality of interval values.

[0023] According to various embodiments, an operating method may be provided that further includes performing at least one of the following operations on a plurality of first data: noise reduction operation using a moving average filter, high-frequency noise reduction operation, or trend correction operation using detrend.

[0024] According to various embodiments, the operation to generate the first time series data related to the green channel may further include: generating first sub-time series data by subtracting the value of the red channel from the value of the green channel; generating second sub-time series data by subtracting the value of the blue channel from the value of the green color channel; and generating first time series data by adding the first sub-time series data and the second sub-time series data.

[0025] According to various embodiments, a method of operation may be provided that further includes the operation of generating the plurality of first data with the plurality of second data for a YCrCb color model; and the operation of generating second time series data by adding a third sub-time series data of the Cr channel and a fourth sub-time series data of the Cb channel based on the plurality of second data.

[0026] According to various embodiments, an electronic device may be provided, comprising at least one processor, the at least one processor configured to: acquire a plurality of images of a user captured using an RGB camera; acquire a specific region of a specific body part of the user from each of the plurality of images; generate a plurality of first data related to an RGB color model for the specific region associated with each of the plurality of images; generate a plurality of second data related to a YCrCb color model based on the plurality of first data; generate first time series data related to the green channel based on the plurality of first data; generate second time series data related to the color difference channel based on the plurality of second data; generate third time series data based on adding the first time series data and the second time series data; and estimate the pulse rate of the user based on converting the third time series data to the frequency domain.

[0027] According to various embodiments, an electronic device may be provided in which at least one processor is configured, in at least part of the operation of obtaining a specific area for a specific body part of a user from each of the plurality of images: to identify a value for a specific color model for each of the plurality of images, and to extract an area from the identified value that is greater than or equal to a preset value.

[0028] According to various embodiments, an electronic device may be provided in which at least one processor is configured to generate the plurality of first data by, in at least part of an operation that generates a plurality of first data related to the RGB color model: obtaining a plurality of values ​​for a plurality of color channels for the plurality of image-specific RGB color models, each of the plurality of values ​​corresponding to a specific point in time, identifying a plurality of interval values ​​for a plurality of time windows among the plurality of values ​​for the plurality of color channels, and adjusting the plurality of values ​​based on the average of the plurality of interval values.

[0029] According to various embodiments, an electronic device may be provided in which the at least one processor is further configured to perform at least one of the following operations on the plurality of first data: a noise reduction operation using a moving average filter, a high-frequency noise reduction operation, or a trend correction operation using detrend.

[0030] 1. Non-contact biometric information system 1 The non-contact biometric information system 1, according to various embodiments, may be a system that measures rPPG (remote photoplethysmography) of an object based on an image of the object captured by a camera, acquires biometric information (e.g., blood pressure, pulse rate, stress index, etc.) measured based on the rPPG, and provides a variety of services. The rPPG may mean blood flow rate (PPG) measured non-contact (i.e., remotely) from blood vessels near the skin. The non-contact biometric information system 1 improves the quality of the measured rPPG by reducing the influence of noise caused by the shooting environment (e.g., illumination, shadows, shaking) reflected in the image captured by the camera, thereby enabling the provision of highly practical services, which will be described in detail below.

[0031] 2. Components of the non-contact biometric information system 1 Figure 1 is a diagram showing examples of components of a non-contact biometric information system 1 according to various embodiments.

[0032] Referring to Figure 1, the non-contact biometric information system 1 according to various embodiments may include a server 10 and a user device 20. Both the server 10 and the user device 20 can be defined as "electronic devices".

[0033] According to various embodiments, the server 10 may be implemented to measure rPPG and resulting biometric information based on an image of user U (e.g., an image of a part of the body) captured by the camera C of user device 20, and to provide various types of services. For example, referring to Figure 1, the server 10 stores a first program 30a and, based on the first program 30a, measures the user U's rPPG and resulting biometric information based on an image of user U received from user device 20, and can provide various types of services. The first program 30a may be implemented to include at least one of a software module, program, various types of information (parameters), or artificial intelligence model for analyzing the image of user U.

[0034] According to various embodiments, the user device 20 may be an electronic device of user U. The user device 20 may include not only electronic devices that can be carried by user U, such as smartphones, tablets, laptops, wearable devices, and HMDs (head-mounted displays), but also stationary electronic devices such as kiosks, PCs, and televisions (TVs), and may further include a variety of electronic devices that include a camera C and other hardware capable of taking pictures of user U, not limited to the examples described. The user device 20 may display a graphic user interface (e.g., display an execution screen that provides a shooting function) that is embodied to provide user U with a shooting function based on the execution of the second program 30b, and may provide at least one of rPPG, biometric information, or information for a variety of services received from the server 10.

[0035] On the other hand, Figure 2 is a diagram showing yet another example of the components of the non-contact biometric information system 1 according to various embodiments. Referring to Figure 2, as shown in Figure 1, the non-contact biometric information system 1 includes a server 10 and a user device 20, but only the program 30 may be implemented. The program 30 includes both the first program 30a and the second program 30b described above, and accordingly, the functions of the server 10 described above may be driven independently (or on-device) by the user device 20. In other words, the user device 20 may be implemented to take an image of the user U using a camera C based on the execution of the program 30, analyze the captured image to measure rPPG and the resulting biometric information, and provide various types of services.

[0036] It will be obvious to those skilled in the art that the operation of the electronic devices in the various embodiments described below can be understood as the operation of the server 10, the operation of the user device 20, or the operation resulting from the cooperation of the server 10 and the user device 20.

[0037] 2.1 Components of Electronic Devices The following describes examples of electronic device configurations for various embodiments of the non-contact bio-information system 1, with reference to Figures 3 and 4.

[0038] 2.1.1 Components of Server 10 Figure 3 is a block diagram showing examples of components of server 10 according to various embodiments.

[0039] Referring to Figure 3, the server 10 in various embodiments may include a first processor 210, a first communication circuit 220, and a first memory 230. However, the server 10 may include more configurations, not limited to the examples described and / or illustrated.

[0040] According to various embodiments, the first processor 210 can control the overall operation of the server 10. To this end, the first processor 210 can perform calculations and processing of various types of information and control the operation of components of the server 10 (e.g., the first communication circuit 220). According to one embodiment, as at least part of data processing or calculations, the first processor 210 can store instructions or data received from other components in volatile memory, process the instructions or data stored in volatile memory, and store the resulting data in non-volatile memory. According to one embodiment, the first processor 210 may include a main processor (not shown) (e.g., a central processing unit or application processor) or an auxiliary processor (not shown) that can operate independently or together with it (e.g., a graphics processing unit, a neural network processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the server 10 includes a main processor (not shown) and an auxiliary processor (not shown), the auxiliary processor (not shown) may use less power than the main processor (not shown) or be configured to specialize in a specified function. An auxiliary processor (not shown) may be implemented separately from or as part of the main processor (not shown).

[0041] According to one embodiment of the present application, an auxiliary processor (not shown) can, for example, control at least a portion of a function or state related to at least one component of the server 10 (e.g., a first communication circuit 220) on behalf of the main processor (not shown) while the main processor (not shown) is in an inactive state (e.g., slipped), or together with the main processor (not shown) while the main processor (not shown) is in an active state (e.g., application execution). According to one embodiment, the auxiliary processor (not shown) (e.g., an image signal processor or a communication processor) may be embodied as part of another functionally related component (e.g., a first communication circuit 220). According to one embodiment, the auxiliary processor (not shown) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. Artificial intelligence models may be generated through machine learning. Such learning may be performed, for example, on the server 10 on which the artificial intelligence is performed, or through a separate server (e.g., a learning server). Learning algorithms can include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Artificial intelligence models can include multiple artificial neural network layers. Artificial neural networks can be deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), deep Q-networks, or any combination of two or more of the above, but are not limited to the examples mentioned above.Artificial intelligence models can include, or largely include, software structures in addition to hardware structures.

[0042] On the other hand, unless otherwise specified in the following explanation, the operation of the server 10 may be interpreted as being carried out under the control of the first processor 210.

[0043] According to various embodiments, the first communication circuit 220 can communicate with an external device (e.g., user device 20). For example, the first communication circuit 220 can be connected to a network via wireless or wired communication to establish communication with an external device (e.g., user device 20) and exchange information and / or data through the established communication. The wireless communication may include cellular communication using at least one of the following: LTE, LTE-A (LTE Advance), CDMA (code division multiple access), WCDMA (wideband CDMA), UMTS (universal mobile telecommunications system), WiBro (Wireless Broadband), or GSM (Global System for Mobile Communications). According to one embodiment, the wireless communication may include at least one of the following: WiFi (wireless fidelity), Bluetooth (registered trademark), Bluetooth Low Power (BLE), Zigbee, NFC (near field communication), Magnetic Secure Transmission, Radio Frequency (RF), or Body Area Network (BAN). According to one embodiment, wireless communication may include GNSS. GNSS may be, for example, GPS (Global Positioning System), Glonass (Global Navigation Satellite System), Beidou Navigation Satellite System (hereinafter, "Beidou"), or Galileo, the European global satellite-based navigation system. Hereinafter, in this document, "GPS" may be used interchangeably with "GNSS".Wired communication may include at least one of the following: USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), RS-232 (Recommended Standard 232), power line communication, or POTS (Plain Old Telephone Service). Networks may include at least one of the following: telecommunications networks, such as computer networks (e.g., LAN or WAN), the Internet, or telephone networks.

[0044] According to one embodiment of this application, the memory 230 can store various types of information. The memory 230 can store data temporarily or semi-permanently. For example, the memory 230 can store a copyright module 200 for generating an active experience file. The server 10 (e.g., first processor 210) can perform operations to generate an active experience file based on the copyright module 200.

[0045] 2.1.2 Components of the user device 20 Figure 4 is a block diagram showing examples of components of the user device 20 according to various embodiments.

[0046] According to various embodiments, referring to Figure 4, the user device 20 may include a second processor 410, a second communication circuit 420, a camera 430, a touchscreen 440, and a second memory 550. The second processor 410 may be implemented as described above for the first processor 210, the second communication circuit 420 as described above for the first communication circuit 220, and the second memory 450 as described above for the first memory 230; therefore, redundant explanations will be omitted.

[0047] According to various embodiments, the camera 430 may be implemented to capture an image of the user of the user device 20. For example, the camera 430 may include at least one of a visible light camera (or RGB camera) or an infrared camera (or IR (infrared) camera). The visible light camera may be a camera capable of capturing images in the general visible light region (e.g., the 380-780 nm wavelength range) (i.e., image capture based on visible light), and the infrared camera may be a camera capable of capturing images in the infrared region (e.g., the 700 nm-1000 nm wavelength range) (i.e., image capture based on infrared light). Depending on the type of user device 20, the user device 20 may be implemented in a form that includes a visible light camera, an infrared camera, or both the visible light camera and the infrared camera. When implemented in a form that includes both the visible light camera and the infrared camera, the user device 20 can measure rPPG based on the visible light image and the infrared image, which will be described later.

[0048] According to various embodiments, the touchscreen 440 may be embodied to display a graphic user interface containing predetermined information and to acquire user input received on the graphic user interface. For example, the user device 20 (e.g., a second processor 410) may display a graphic user interface (e.g., an execution screen) that includes at least one of rPPG, biometric information, or various types of services received from the server 10. Alternatively, for example, the user device 20 (e.g., a second processor 410) may display a graphic user interface (e.g., an execution screen) that includes a menu screen and / or icons for controlling the camera 430, and activate the camera 430 to capture an image of the user based on user input to the icons received through the touchscreen 440.

[0049] 3. rPPG and Methods for Measuring Biological Information 3.1. rPPG and biometric measurement based on multiple color models Figure 5 is a flowchart illustrating an example of how an electronic device (e.g., server 10, user device 20) for rPPG and biometric information measurement based on multiple color models relating to various embodiments operates. However, the operations may be performed in a different order than described and / or illustrated, and more or fewer operations may be performed than described and / or illustrated. Figure 5 will be further described below with reference to Figures 6 to 8.

[0050] Figure 6 is a diagram illustrating an example of a module that performs operations for rPPG and biometric information measurement using a visible light camera according to various embodiments. Figure 7 is a diagram illustrating an example of operations for preprocessing RGB data according to various embodiments. Figure 8 is a diagram illustrating an example of operations using multiple color models according to various embodiments.

[0051] According to various embodiments, an electronic device (e.g., server 10, user device 20) can acquire multiple images of the user using an RGB camera in operation 501, identify a specific region from each of the multiple images in operation 503, and generate multiple first data related to a first color model for the specific region in each of the multiple images in operation 505. For example, the electronic device (e.g., server 10, user device 20) can acquire RGB data for a specific part of the user's body (e.g., face) over a preset period of time using a visible light camera among the cameras 430 for rPPG measurement of the user. For example, user device 20 can capture multiple images of the user over a preset period of time using a visible light camera at the user's request. Referring to Figures 6 and 7, the electronic device (e.g., server 10, user device 20) can use the face region extraction module 610 to identify the skin region R2 within each of the multiple images (or image frames) (Frame#1, ..., Frame#n) captured by the user device 20, based on the execution of program 300 (e.g., second program 30b or program 30), and use the RGB data generation module 620 to obtain values ​​for the RGB color model for the skin region R2. On the other hand, not limited to the above example, skin regions of other body parts (e.g., thighs, arms, etc.) may be extracted instead of skin region R2 within the user's face region R1.

[0052] According to various embodiments, an electronic device (e.g., server 10, user device 20) uses a face region extraction module 610 to identify skin regions within a face. As part of this operation, it identifies a face region R1 from a plurality of captured images (Frame#1, ..., Frame#n) based on an object identification algorithm. After converting the multiple pixel-specific RGB color model values ​​contained in the identified face region R1 to YCrCb color model values ​​or HSV color model values, it can identify at least some pixels having a value greater than or equal to a preset value as skin region R2 within the face region R1.

[0053] According to various embodiments, an electronic device (e.g., server 10, user device 20) can use the RGB data generation module 620 to obtain values ​​for the RGB color model for the skin region R2. As part of this operation, it can apply Gaussian blur to the skin region R2 of each of the multiple images (Frame#1, ..., Frame#n) to remove noise, and as shown in Figure 7, obtain the average values ​​of the B channel (blue channel), R channel (red channel), and G channel (green channel) of multiple pixels contained in the skin region R2 of each of the multiple images (Frame#1, Frame#2, Frame#3, Frame#4, ..., Frame#n) from which the noise has been removed (i.e., obtain the average values ​​for the RGB color model for the skin region R2). Also, referring to Figure 7, the electronic device (e.g., server 10, user device 20) can perform mean centering on the average values ​​for the RGB color model for the skin region R2 of each of the multiple images (Frame#1, ..., Frame#n). For example, an electronic device (e.g., server 10, user device 20) can adjust (or change, adjust, or correct) the average value of the RGB color model for the skin region R2 of some of the multiple images (Frame#1, ..., Frame#n) contained within time windows W1 and W2 based on the average value of each BGR channel between time windows W1 and W2 for a predetermined number of images (or predetermined time). For example, the average centering of the RGB color model for the skin region R2 can be performed by the following [Mathematical Formula 1]. [Mathematical formula 1] JPEG2026056524000002.jpg15170

[0054] Here, Va represents the value of a specific channel (B channel, G channel, R channel) of a specific image frame that has been mean-centered, Va represents the value of a specific channel of a specific image frame before mean-centering, Vi represents the value of a specific channel for each image frame included in a specific time window, where i represents the identification number of the image frame within the specific time window, n represents the number of image frames included in the specific time window, and m may represent the number of time windows overlapping the specific image frame.

[0055] In other words, according to [Mathematical Formula 1] above, the value of a specific channel for the skin region R2 of a specific image may be obtained by subtracting the average value of the specific channel in the specific time window in which the specific image is contained. In this case, referring to Figure 7, multiple time windows may be realized and set to overlap each other. Accordingly, when multiple time windows are set for a specific image, the average value of the specific channel for each of the multiple time windows is subtracted from the value of the specific channel for the skin region R2 of the specific image. However, the average value of the specific channel that is subtracted may be divided by the number of overlapping time windows (m) before being subtracted.

[0056] Furthermore, electronic devices (e.g., server 10, user device 20) can perform at least one of the following operations: amplitude correction or trend correction, based on the average value rather than mean centering.

[0057] According to various embodiments, an electronic device (e.g., server 10, user device 20) can utilize a first preprocessing module 630, as shown in Figure 6, to perform preprocessing operations on the RGB color model values ​​for the skin region R2 of multiple acquired images. The preprocessing operations may include at least one of the following: removing noise (e.g., high-frequency noise, low-frequency noise) using a signal filter (e.g., moving average filter) or trend correction operations including detrend removal.

[0058] According to various embodiments, an electronic device (e.g., server 10, user device 20) can generate a plurality of second data related to a second color model (e.g., YCrCb, CIE La*b*) based on the plurality of first data (e.g., RGB color model values ​​for skin region R2) in operation 507. Referring, for example, to Figures 6 and 8, the electronic device (e.g., server 10, user device 20) can use a time-series data generation module 640 (e.g., first to third time-series data generation modules 640a, 640b, 640c) to convert RGB color model values ​​for skin region R2 for multiple images to values ​​for other color models. The other color models may include, but are not limited to, the YCrCb color model or the CIE La*b* color model.

[0059] According to various embodiments, an electronic device (e.g., server 10, user device 20) can generate first time series data related to the green channel based on a plurality of first data (e.g., RGB color model values) in operation 509, generate at least one second time series data related to the color difference channel based on a plurality of second data (e.g., YCrCb color model values) in operation 511, and generate integrated time series data based on adding the first and second time series data in operation 513. For example, the electronic device (e.g., server 10, user device 20) can generate final time series data (e.g., integrated time series data) by combining the values ​​of multiple color models (e.g., RGB color model, YCrCb color model, or CIE La*b* color model) for the skin region R2 that were acquired. By combining these, the disadvantages of each different color model are complemented, and noise caused by the shooting environment (e.g., illuminance, shadows, tremors) that occurs when photographing the user is reduced, thereby enabling the acquisition of even higher quality rPPG.

[0060] In one embodiment, referring to Figure 8, the electronic device (e.g., server 10, user device 20) can use the first time-series data generation module 640a to generate first time-series data 810 for multiple images by using RGB color model values ​​for the skin region R2 and adding a first value obtained by subtracting the red channel from the green channel and a second value obtained by subtracting the blue channel from the green channel. The green channel best reflects the amount of blood, and there is an error due to the red and blue channels included in the ambient light source during shooting. Therefore, the first time-series data value obtained by subtracting the red and blue channels from the green channel may better represent the amount of blood while reducing the error due to the red and blue channels of the external light source.

[0061] In one embodiment, referring to Figure 8, the electronic device (e.g., server 10, user device 20) can use the second time-series data generation module 640b to generate second time-series data 810 by summing all color difference channels (Cr channel, Cb channel) for multiple images using the YCrCb color model values ​​for the skin region R2.

[0062] In one embodiment, referring to Figure 8, an electronic device (e.g., server 10, user device 20) can use a third time series data generation module 640c to generate a summed third time series data 830 containing only a values, using CIE La*b* color model values ​​for the skin region R2.

[0063] An electronic device (e.g., server 10, user device 20) can generate a third time series data by combining at least some of the multiple time series data 810, 820, and 830 generated using the rPPG generation module 650. For example, an electronic device (e.g., server 10, user device 20) can generate integrated time series data by adding the first time series data 810 and the second time series data 820. Accordingly, the values ​​of the first time series data 810, from which the red and blue channels have been removed, can be complemented by the second time series data 820, which includes the chrominance channel. Furthermore, electronic devices (e.g., server 10, user device 20) can generate integrated time series data by adding the first time series data 810, the second time series data 820, and the third time series data 830. The generated integrated time series data represents blood flow and can therefore be defined as a signal in rPPG.

[0064] According to various embodiments, an electronic device (e.g., server 10, user device 20) can perform preprocessing operations on multiple time-series data 810, 820, and 830 before combining them. For example, the preprocessing operations may include Z-Score operations.

[0065] According to various embodiments, an electronic device (e.g., server 10, user device 20) can acquire the user's biometric information based on integrated time-series data in operation 515. For example, the electronic device (e.g., server 10, user device 20) can acquire biometric information by performing analysis on the integrated time-series data using a biometric information acquisition module 660. The biometric information may include pulse rate, blood pressure, stress index, etc. For example, the electronic device (e.g., server 10, user device 20) can convert the integrated time-series data into the frequency domain in operation 515, extract frequency values ​​in the frequency domain corresponding to the pulse rate domain (40 bpm to 240 bpm), extract a preset frequency band (e.g., a frequency band corresponding to 20 bpm) containing the highest frequency among the extracted frequency values ​​using a band filter, and extract the pulse rate based on the distance between peaks within the extracted frequency band. The electronic device (e.g., server 10, user device 20) can estimate other biometric information such as blood pressure based on the extracted pulse rate.

[0066] 3.2. rPPG and biometric information measurement based on multiple region-specific IR channel values Figure 9 is a flowchart illustrating an example of how an electronic device (e.g., server 10, user device 20) for rPPG and biometric information measurement based on multiple region-specific IR channel values ​​in various embodiments operates. However, the operations may be performed in a different order than described and / or illustrated, and more or fewer operations may be performed than described and / or illustrated. Figure 9 will be further described below with reference to Figures 10 to 11.

[0067] Figure 10 is a diagram illustrating an example of a module that performs operations for rPPG and biological information measurement using an infrared camera according to various embodiments. Figure 11 is a diagram illustrating an example of operation that utilizes the values ​​of multiple region-specific IR channels according to various embodiments.

[0068] According to various embodiments, an electronic device (e.g., server 10, user device 20) can acquire multiple images of the user using an infrared camera in operation 901. For example, the electronic device (e.g., server 10, user device 20) can acquire IR data for a predetermined period of time on a specific part of the user's body (e.g., face) using the infrared camera of camera 430 for rPPG measurement of the user. For example, user device 20 can take multiple images of the user using the infrared camera for a predetermined period of time at the user's request.

[0069] According to various embodiments, an electronic device (e.g., server 10, user device 20) can identify multiple regions of the face based on facial feature information extracted from each of the multiple images in operation 903. For example, referring to Figures 10 to 11, the electronic device (e.g., server 10, user device 20) can use a feature point extraction module 1010 to extract feature points representing the user's face from the multiple images, and use a landmark identification module 1020 to identify the regions corresponding to feature points defined as landmarks among the extracted feature points as multiple landmark regions L1, L2, L3, L4. As an example, the multiple landmark regions L1, L2, L3, L4 may include the upper part of the left cheek L1, the lower part of the left cheek L2, the upper part of the right cheek L3, and the lower part of the right cheek L4. The landmark identification module 1020 is an artificial intelligence model that has been trained to pre-identify feature points that represent parts (e.g., specific skeletons) that define each of the multiple landmark regions L1, L2, L3, and L4. Based on the input of information (e.g., coordinates, vectors) for the extracted feature points, it can identify and output feature points that represent parts (e.g., specific skeletons) that define each of the landmark regions L1, L2, L3, and L4. An electronic device (e.g., server 10, user device 20) can define the regions corresponding to the outputted feature points as landmark regions L1, L2, L3, and L4.

[0070] According to various embodiments, an electronic device (e.g., server 10, user device 20) can acquire multiple time-series data for IR channels from the multiple regions L1, L2, L3, and L4 in operation 905, and acquire integrated time-series data based on at least one of the mean or variance values ​​of the multiple time-series data in operation 907. For example, referring to Figure 10, the electronic device (e.g., server 10, user device 20) can use a landmark region time-series data extraction module 1030 (e.g., first to fourth region extraction modules 1030a, 1030b, 1030c, 1030d) to extract the average time-series signals of the IR channels of each of the multiple regions L1, L2, L3, and L4, preprocess the extracted average time-series signals of the IR channels of each of the multiple regions L1, L2, L3, and L4, and combine the preprocessed IR channel signals of each of the multiple regions L1, L2, L3, and L4 to acquire integrated time-series data.

[0071] For example, an electronic device (e.g., server 10, user device 20) can utilize multiple region extraction modules 1030a, 1030b, 1030c, and 1030d as at least part of the operation to preprocess the average time-series signals of the IR channels. For example, the electronic device (e.g., server 10, user device 20) can use each of the multiple region extraction modules 1030a, 1030b, 1030c, and 1030d to generate average time-series signals of the IR channels of multiple regions L1, L2, L3, and L4 for multiple image frames sequentially captured using deque, and then perform at least one of average centering or Z-score on the generated average time-series signals of the IR channels of multiple regions L1, L2, L3, and L4 to generate time-series signals for the IR channels of multiple regions L1, L2, L3, and L4. The generated time-series signals can be defined as rPPG signals. Thereafter, electronic devices (e.g., server 10, user device 20) can use the preprocessing module 1040 to perform preprocessing on the rPPG signals of each of the multiple regions L1, L2, L3, and L4 by performing at least one of the following operations: moving average filtering / moving time window-based trend removal, or Bézier curve-based filtering.

[0072] As a result, electronic devices (e.g., server 10, user device 20) can generate integrated time-series data by combining some of the rPPG signals for the IR channels of multiple pre-processed regions L1, L2, L3, and L4. For example, the combination means generating integrated time-series data by performing arithmetic operations on each of the rPPG signals for the IR channels of the multiple pre-processed regions L1, L2, L3, and L4, and the combination may be determined by the shooting environment (e.g., illumination, shadow, shaking). For example, under first imaging conditions (e.g., shadow conditions), integrated time-series data can be generated by adding a first result signal obtained by subtracting the rPPG of the second region L2 from the rPPG of the first region L1, and a second result signal obtained by subtracting the rPPG of the fourth region L4 from the rPPG of the third region L3. Under second imaging conditions (e.g., shadow conditions), integrated time-series data can be generated by adding a third result signal obtained by subtracting the rPPG of the third region L3 from the rPPG of the first region L1, and a fourth result signal obtained by subtracting the rPPG of the fourth region L4 from the rPPG of the second region L2. On the other hand, not limited to the examples described, rPPG signals for multiple IR channels of regions L1, L2, L3, and L4, preprocessed in various ways, can be combined.

[0073] According to various embodiments, an electronic device (e.g., server 10, user device 20) can acquire the user's biometric information based on integrated time-series data in operation 909. For example, the electronic device (e.g., server 10, user device 20) can use the biometric information acquisition module 1050 to estimate the pulse rate through frequency interpretation of the integrated time-series data, as described in operation 515 above, and redundant explanations are omitted.

[0074] For example, an electronic device (e.g., server 10, user device 20) can use the visible light camera among the cameras 430 to acquire RGB data for a predetermined period of time on a specific part of the user's body (e.g., face) for rPPG measurement of the user. For example, the user device 20 can, at the user's request, use the visible light camera to capture multiple images of the user for a predetermined period of time. Referring to Figures 6-7, based on the execution of program 300 (e.g., second program 30b or program 30), the electronic device (e.g., server 10, user device 20) can use the face region extraction module 610 to identify the skin region R2 within each of the multiple images (or image frames) (Frame#1, ..., Frame#n) captured by the user device 20, and use the RGB data generation module 620 to acquire values ​​for the RGB color model for the skin region R2. On the other hand, not limited to the above example, skin regions of other body parts (e.g., thighs, arms, etc.) may be extracted instead of skin region R2 within the user's face region R1.

[0075] 3.3 Hybrid rPPG and Biometric Measurement Figure 12 is a flowchart illustrating an example of how an electronic device (e.g., server 10, user device 20) operates for measuring rPPG using a visible light camera and an IR camera based on various imaging environment conditions. However, the operations may be performed in a different order than described and / or illustrated, and more or fewer operations may be performed than described and / or illustrated. Figure 12 will be further described below with reference to Figure 13.

[0076] Figure 13 is a diagram illustrating an example of measuring rPPG using a visible light camera and an IR camera according to various embodiments.

[0077] According to various embodiments, an electronic device (e.g., server 10, user device 20) can acquire shooting environment conditions in operation 1201 and determine whether the merging conditions are satisfied in operation 1203. For example, non-contact rPPG / biometric information measurement and service can be performed in a variety of shooting environments. Accordingly, the electronic device (e.g., server 10, user device 20) can acquire information that affects camera shooting, such as information on current illuminance and information on shaking, as shooting environment conditions, and can determine whether the merging conditions are satisfied based on whether the acquired information exceeds (or falls below) a preset value. For example, if the illuminance value exceeds a preset value, it may be determined that the merging conditions are satisfied, and if the illuminance value is less than the preset value, it may be determined that the merging conditions are not satisfied. Also, for example, if the shaking value exceeds a preset value, it may be determined that the merging conditions are satisfied, and if the shaking value is less than the preset value, it may be determined that the merging conditions are not satisfied.

[0078] As an example, referring to Figure 13, non-contact rPPG / biometric information measurement and service can be performed for passengers (e.g., the driver or passengers) within the means of transport. Referring to Figure 13(a), non-contact rPPG / biometric information measurement and service can be performed based on images captured by a camera installed in the vehicle, and referring to Figure 13(b), non-contact rPPG / biometric information measurement and service can be performed based on images captured by a camera on a user device 20 (e.g., mobile) installed in the vehicle. Accordingly, electronic devices (e.g., server 10, user device 20) can acquire information that affects camera shooting, such as information on illuminance and information on vibration, which is obtained by sensors installed in the means of transport (e.g., illuminance sensor, motion sensor) or sensors on the user device 20 (e.g., illuminance sensor, motion sensor), and can determine whether the merger conditions are met based on this information. Furthermore, referring to Figure 13(b), the means of transport may be an autonomous means of transport that does not require a driver (e.g., an autonomous vehicle, an autonomous vessel, etc.).

[0079] Furthermore, as an example, although not illustrated, cameras for non-contact rPPG / biometric information measurement and service may be installed in a variety of locations (or spaces) where cameras can be mounted, such as inside homes, buildings, offices, and corridors, in addition to transportation methods, and information on shooting conditions using sensors may be collected.

[0080] According to various embodiments, if the merging conditions are met (1203-Y), the electronic device (e.g., server 10, user device 20) can acquire first time-series data from an RGB camera in operation 1205, second time-series data from an IR camera in operation 1207, and acquire biological information based on the first and second time-series data in operation 1209. For example, if the merging conditions are met, the device recognizes that the imaging environment may reduce the accuracy of rPPG, and accordingly, the electronic device (e.g., server 10, user device 20) can generate first integrated time-series data based on an image captured by a visible light camera according to the operation method in Figure 5 described above, and second integrated time-series data based on an image captured by an infrared camera according to the operation method in Figure 9. Electronic devices (e.g., server 10, user device 20) can generate time-series data by reflecting the first and second integrated time-series data together (e.g., by adding them up and taking the average, or by simply summing them), and then measure biological information (e.g., pulse rate) based on the generated time-series data, or measure final biological information by complementing the biological information (e.g., pulse rate) measured based on the first and second integrated time-series data respectively.

[0081] According to various embodiments, if the merging conditions are not satisfied (1203-N), the electronic device (e.g., server 10, user device 20) can acquire RGB camera-based time-series data in operation 1211 and acquire biological information based on the time-series data in operation 1213. For example, the electronic device (e.g., server 10, user device 20) can then generate integrated time-series data based on images captured by a visible light camera according to the operation method described in Figure 5 and estimate biological information. Alternatively, for example, contrary to the description, the electronic device (e.g., server 10, user device 20) can then generate integrated time-series data based on images captured by an infrared camera according to the operation method described in Figure 9 and estimate biological information.

[0082] According to various embodiments, operation may be performed based on an RGB camera when the information regarding the shooting conditions is within a first range, operation may be performed based on an IR camera when it is within a second range lower than the first range, and hybrid operation may be performed based on both an RGB camera and an IR camera in a third range between the first and second ranges, but is not limited to the examples described. For example, operation may be performed based on an RGB camera when the illuminance is within a first range, operation may be performed based on an IR camera when the illuminance is within a second range lower than the first range, and hybrid operation may be performed in a third range in between.

[0083] 3.4 rPPG / Biometric Information Measurement Operation Based on AI Model 3.4.1 Blood Pressure AI Model-Based Blood Pressure Measurement Figure 14 is a flowchart illustrating an example of how an electronic device (e.g., server 10, user device 20) for measuring blood pressure, based on a blood pressure AI model according to various embodiments, operates. However, the operations may be performed in a different order than those described and / or illustrated, and more or fewer operations may be performed than those described and / or illustrated.

[0084] According to various embodiments, an electronic device (e.g., server 10, user device 20) can acquire multiple images of the user using an RGB camera in operation 1401, identify a specific region from each of the multiple images in operation 1403, and generate multiple first data related to a first color model for the specific region in each of the multiple images in operation 1405. For example, in operations 501 to 505, the electronic device (e.g., server 10, user device 20) can acquire RGB color model values ​​for the skin region R2 for multiple images captured over a preset period of time, as described above.

[0085] According to various embodiments, an electronic device (e.g., server 10, user device 20) can generate a plurality of second data related to a second color model based on the plurality of first data in operation 1407, generate a first time series data related to the green channel based on the plurality of first data in operation 1409, and generate at least one second time series data related to the chrominance channel based on the plurality of second data in operation 1411. For example, an electronic device (e.g., server 10, user device 20) can perform operations 1407 to 1411 to generate time series data (rPPG) for multiple color models (e.g., RGB, YCgCr, CIE La*b*) as described above.

[0086] According to various embodiments, an electronic device (e.g., server 10, user device 20) can obtain blood pressure based on the input of first and second time-series data to a blood pressure measurement AI model in operation 1413. For example, the blood pressure measurement AI model may be an artificial intelligence model trained to output blood pressure values ​​based on input of information for at least some of the time-series data from multiple color models (e.g., RGB, YCgCr, CIE La*b*). Since this training can be carried out based on various learning algorithms such as supervised learning, unsupervised learning, and machine learning, a specific explanation is omitted.

[0087] In one embodiment, the blood pressure measurement AI model may be an artificial intelligence model that has been trained by setting at least some time-series data from multiple color models (e.g., RGB, YCgCr, CIE La*b*) for a specific time interval as input data, and setting blood pressure values ​​measured during a specific time period as output data.

[0088] In one embodiment, the blood pressure measurement AI model may be an artificial intelligence model trained by setting at least some frequency feature values ​​from multiple color models (e.g., RGB, YCgCr, CIE La*b*) for a specific time interval as input data, and setting blood pressure values ​​measured during that specific time as output data. The frequency feature values ​​may include information about the distance between peaks.

[0089] 3.4.2 Stress Analysis AI Model-Based Stress Measurement Figure 15 is a flowchart illustrating an example of how an electronic device (e.g., server 10, user device 20) for measuring stress on a stress analysis AI model base according to various embodiments operates. However, the operations may be performed in a different order than described and / or illustrated, and more or fewer operations may be performed than described and / or illustrated. Figure 15 will be further described below with reference to Figure 16.

[0090] Figure 16 is a diagram illustrating examples of modules for performing stress analysis operations in various embodiments.

[0091] According to various embodiments, an electronic device (e.g., server 10, user device 20) can acquire multiple images of the user using an RGB camera in operation 1501, and in operation 1503 generate a first time-series signal (e.g., RGB color model values) related to an RGB color model for the specific region (e.g., skin region R2 in Figure 5) based on the multiple images. Operations 1501 to 1503 of the electronic device (e.g., server 10, user device 20) can be performed based on the RGB generation data module 1610, as described above for operations 501 to 503 of the electronic device (e.g., server 10, user device 20), so redundant explanations are omitted.

[0092] According to various embodiments, an electronic device (e.g., server 10, user device 20) can generate a second time-series signal related to the CIE La*b* color model for a specific region based on the multiple images in operation 1505, and correct the first time-series signal based on the second time-series signal in operation 1507. For example, referring to Figure 16, the electronic device (e.g., server 10, user device 20) can perform interpolation on the first time-series signal related to the RGB color model for the skin region R2 obtained using the correction module 1620, and can correct (or pre-process) the brightness of the first time-series signal using the second time-series signal for the L channel measured from the multiple images converted to the CIE La*b* color model.

[0093] According to various embodiments, an electronic device (e.g., server 10, user device 20) can generate a third time-series signal related to a YCrCgCb color model based on a first time-series signal corrected by operation 1509. The electronic device (e.g., server 10, user device 20) can use the YCrCgCb time-series signal generation module 1630 to generate a first time-series signal for a brightness-preprocessed RGB model using a YCrCgCb time-series signal, and can use the pre-processing model 1640 to perform pre-processing on the YCrCgCb time-series signal. The pre-processing operation may include at least one of the following: moving average filter-based denoising, sliding window-based trend denoising, Butterworth bandpass filtering, amplitude correction, or, after detecting signal peaks for each channel (Y, Cr, Cg, Cb), extracting (or cropping) the signal for a preset time (e.g., 8 seconds) based on a down-peak reference.

[0094] According to various embodiments, an electronic device (e.g., server 10, user device 20) can acquire a stress index for the user based on a third time-series signal and additional information in operation 1511. For example, the electronic device (e.g., server 10, user device 20) can acquire blood pressure feature values ​​(e.g., SDNN, SDSD, RMSSD) extracted based on inputting a pre-processed third time-series signal (YCrCgCb signal) into the HRV analysis AI model 1650. The electronic device (e.g., server 10, user device 20) can acquire a stress index based on inputting the acquired blood pressure feature values, along with rPPG acquired based on the rPPG extraction module and biometric information measured by rPPG based on the biometric information acquisition module 1680 (e.g., pulse rate, oxygen saturation), into a stress index analysis algorithm (mathematical formula). The measured rPPG and biometric information can be measured as described above in Figures 5, 9, and 12, so redundant explanations are omitted. The aforementioned electronic devices (e.g., server 10, user device 20) can post-process the acquired stress index based on a post-processing algorithm and provide it to the user.

[0095] 4. Mobility Services Figure 17 is a diagram illustrating an example of a transportation means V that provides a contactless biometric information-based service according to various embodiments.

[0096] According to various embodiments, electronic devices (e.g., server 10, user device 20) can be embodied to measure rPPG and biometric information based on analyzing captured images of passengers (e.g., driver or passengers) aboard the transport means V, as described in Section 3, Table of Contents, to provide predetermined services.

[0097] According to various embodiments, referring to Figure 17, the transport means V may include a communication circuit 1710, a processor 1720, a camera 1730, and a component device 1740, and may be embodied to include many more devices, not limited to the illustrated example. On the other hand, the camera 1730 may not be provided on the transport means V, and the service may be performed based on images captured by a camera 430 of a user device 20 installed on the transport means V.

[0098] According to various embodiments, electronic devices (e.g., server 10, user device 20) can receive various types of information from the communication circuit 1710 of the transport means V. For example, the various types of information may include images of passengers (e.g., driver or passengers) inside the transport means V captured by the camera 1730, and information about the shooting environment conditions.

[0099] According to various embodiments, an electronic device (e.g., server 10, user device 20) can transmit a control signal to the transport means V's processor 1720 to control the component device 1740 based on the consequently analyzed rPPG and biometric information. The transport means V's processor 1720 can then control the component device 1740 based on the received control signal. For example, the component device 1740 may include not only devices that directly affect driving, such as steering devices 1740a and engines 1740b, but also devices that do not directly affect driving but provide convenience, such as seats 1740c and lighting (not shown).

[0100] 4.1 Provision of contactless biometric information and external information infrastructure mobility services Figure 18 is a flowchart illustrating an example of the operation method of an electronic device (e.g., server 10, user device 20) for providing contactless biometric and external information infrastructure mobility services according to various embodiments. However, the operations may be performed in an order different from that described and / or illustrated, and more or fewer operations may be performed than those described and / or illustrated. Figure 18 will be further described below with reference to Figure 19.

[0101] Figure 19 is a diagram illustrating examples of mobility services provided by contactless biometric and external information according to various embodiments.

[0102] According to various embodiments, an electronic device (e.g., server 10, user device 20) can identify a vehicle start event in operation 1801, capture multiple images of the user using a camera in operation 1803, and acquire biometric information based on the captured multiple images in operation 1805. For example, the electronic device (e.g., server 10, user device 20) can acquire multiple images and acquire biometric information using a camera 1730 provided on the transport means (e.g., vehicle) or a camera 430 provided on the user device 20 installed inside the transport means (e.g., vehicle). The cameras 1730 and 430 may include at least one of visible light cameras or infrared cameras, and as described in "3. Table of Contents" above, operations to measure rPPG and biometric information can be performed, so redundant explanations are omitted.

[0103] According to various embodiments, an electronic device (e.g., server 10, user device 20) can acquire information about the vehicle's external environment in operation 1807 and control devices within the vehicle based on biometric information and information about the external environment in operation 1809. For example, the electronic device (e.g., server 10, user device 20) can acquire information about the external environment related to the location of the currently started vehicle. The information about the external environment may include climatic information such as temperature, wind direction, wind speed, and weather at the vehicle's location, and various other types of information that can be collected from an external server. Consequently, the electronic device (e.g., server 10, user device 20) can provide the vehicle with control signals to control the vehicle's components 1740 based on biometric information measured in a non-contact manner and the collected information about the external environment. For example, referring to Figure 19, depending on blood pressure measured in a non-contact manner and external temperature measured as external environment information, the electronic device (e.g., server 10, user device 20) can provide control signals to control different types of components. As an example, referring to Figures 19(a) and 19(b), an electronic device (e.g., server 10, user device 20) may determine that an event has occurred to control the vehicle's components 1740 if the blood pressure value is a specific value (e.g., 130). When the event occurs, the electronic device (e.g., server 10, user device 20) acquires information on the external temperature as external environmental information. As shown in Figure 19(a), if the external temperature is higher than a preset value, it generates a control signal to activate the air conditioner in the components 1740. As shown in Figure 19(b), if the external temperature is lower than a preset value, it generates a control signal to open the windows in the components 1740 and provides these signals to the vehicle's processor 1720. Based on these control signals, the vehicle's processor 1720 can control the vehicle's components 1740. On the other hand, without being limited to the examples described and / or illustrated, the operations controlling the vehicle's components 1740 can be performed in a variety of ways, and operations controlling other components 1740 can be performed in a similar manner, so a specific explanation is omitted.

[0104] According to various embodiments, the electronic device (e.g., server 10, user device 20) may, as part of the operation to generate the aforementioned control signals, either have information on the types and control methods of pre-controlled configuration devices 1740, separated by biometric information and information about the external environment, stored in the form of a lookup table, or utilize at least one of artificial intelligence models that have been pre-trained with information on the types and control methods of pre-controlled configuration devices 1740, separated by biometric information and information about the external environment.

[0105] 4.1.1 Provision of mobility services that further consider passenger information Figure 20 is a flowchart illustrating an example of how an electronic device (e.g., server 10, user device 20) operates to provide a mobility service that further considers passenger information in various embodiments. However, the operations may be performed in a different order than described and / or illustrated, and more or fewer operations may be performed than described and / or illustrated. Figure 20 will be further described below with reference to Figure 21.

[0106] Figure 21 is a diagram illustrating examples of rPPG / biometric information measurement for passengers in various embodiments.

[0107] According to various embodiments, electronic devices (e.g., server 10, user device 20) can provide mobility services by further considering biometric information of passengers in the vehicle in addition to the mobility service provision operation described in "4.1 Table of Contents".

[0108] According to various embodiments, the electronic device (e.g., server 10, user device 20) uses a camera in operation 2001 to capture multiple images of multiple passengers, and in operation 2003 acquires biometric information of multiple passengers based on the captured images, thereby enabling the execution of operations 1807 to 1809 to provide mobility services. For example, referring to Figure 21, a vehicle can carry not only a driver but also passengers, and in the case of an autonomous vehicle, multiple passengers can be on board. In this case, if mobility services are provided considering only the driver's biometric information, inconvenience may be caused to the remaining passengers. Therefore, the electronic device (e.g., server 10, user device 20) can further consider the biometric information of other passengers when providing mobility services.

[0109] For example, an electronic device (e.g., server 10, user device 20) can acquire multiple images taken for a predetermined time for each of several passengers by utilizing a camera 1730 installed in a means of transport (e.g., a vehicle) or a camera 430 installed in a user device 20 located inside a means of transport (e.g., a vehicle). In one embodiment, the camera 1730 may be designed to have an FOV capable of capturing all of the multiple passengers, or it may be installed in a position (e.g., rearview mirror, ceiling, etc.) capable of capturing all of the multiple passengers. Alternatively, multiple cameras 1730 may be installed in positions adjacent to the seats of each of the multiple passengers. In another embodiment, the camera 430 may be designed to have an FOV capable of capturing all of the multiple passengers, or individual passenger images may be captured by installing a user device 20, each carried by each of the multiple passengers, in a position adjacent to each seat.

[0110] Accordingly, the electronic devices (e.g., server 10, user device 20) can further consider the biometric information measured based on each passenger's image, and as a result generate different control signals. For example, if the biometric information of multiple passengers falls within a similar range, the electronic devices (e.g., server 10, user device 20) can generate the determined control signals for the configured devices 1740 as is. Also, as an example, as shown in Figure 21, if the biometric information of multiple passengers does not fall within a similar range (i.e., they are different from each other), the electronic devices (e.g., server 10, user device 20) can generate a control signal to open the windows, but only open the windows around the driver for the driver with high blood pressure, while keeping the windows around passengers with relatively normal blood pressure unchanged.

[0111] 4.1.2 Providing mobility services that take vehicle modes into further consideration Figure 22 is a flowchart illustrating an example of how an electronic device (e.g., server 10, user device 20) operates to provide mobility services that further consider the modes of vehicles according to various embodiments. However, the operations may be performed in a different order than those described and / or illustrated, and more or fewer operations may be performed than those described and / or illustrated. Figure 22 will be further described below with reference to Figure 23.

[0112] Figure 23 is a diagram illustrating examples of mobility services that continuously change the shape of the seat during autonomous driving, according to various embodiments.

[0113] According to various embodiments, an electronic device (e.g., server 10, user device 20) can identify in operation 2201 that the vehicle's driving mode is autonomous driving mode, and in operation 2203 control the shape of the vehicle's seat based on the biometric information. For example, the vehicle may be set to autonomous driving mode by the driver's control after starting. As shown in Figure 23, the electronic device (e.g., server 10, user device 20) can continuously (or periodically) measure the user's biometric information (e.g., blood pressure) based on an image taken while the vehicle is set to autonomous driving mode, and further improve user convenience by controlling the shape of the vehicle's seat to a comfortable shape based on the biometric information measured while the vehicle's mode is set to autonomous driving mode. As an example, as shown in Figure 23, if it is identified that the blood pressure value measured in a non-contact manner exceeds a preset value (E), the electronic device (e.g., server 10, user device 20) can generate a control signal to change the seat angle from the current angle to another angle and provide it to the vehicle's processor 1820.

[0114] According to various embodiments, an electronic device (e.g., server 10, user device 20) can control other devices within the vehicle based on the biometric information and information about the external environment in operation 2205. Since operation 2205 of the electronic device (e.g., server 10, user device 20) can be performed in the same way as operation 1809 of the electronic device (e.g., server 10, user device 20) described above, a redundant explanation will be omitted.

[0115] 5. Provision of a sustainable biometric information storage platform service. Figure 24 is a flowchart illustrating an example of the operation method of an electronic device (e.g., server 10, user device 20) for providing a continuous bio-information storage infrastructure service according to various embodiments. However, the operations may be performed in an order different from that described and / or illustrated, and more or fewer operations may be performed than those described and / or illustrated. Below, Figure 24 will be further explained with reference to Figures 25a to 25b.

[0116] Figure 25a is a diagram illustrating an example in which biological information measured for each driving route over a predetermined period of time related to various embodiments is accumulated in a database 2500. Figure 25b is a diagram illustrating an example of a service that provides driving recommendation information based on the accumulated biological information related to various embodiments.

[0117] In various embodiments, an electronic device (e.g., server 10, user device 20) can identify a specific type of life event in operation 2401, and use a camera in operation 2403 to capture multiple images of the user while the specific type of life event is ongoing. For example, the specific type of life event refers to a variety of activities that the user can perform, and may include, for example, a variety of activities such as driving, exercising, studying, and working. In this case, the electronic device (e.g., server 10, user device 20) can receive input from the user by providing a graphical user interface for receiving information about what kind of life event it is, or it may predict the type of life event the user is currently performing based on pattern information of sensor values ​​collected using a variety of sensors (e.g., motion sensors, angular velocity sensors, illuminance sensors, etc.). The electronic device (e.g., server 10, user device 20) periodically captures images of the user at multiple time intervals based on the camera 430 of the user device 20 or a camera placed in the space where the specific type of life event is performed (e.g., camera 1730 in the vehicle) while a specific type of life event is performed, and as described in "Table of Contents 3.", the user's rPPG and biometric information can be measured at multiple time intervals based on the captured images. For example, as shown in Figure 25a, if the user drives along a specific route on multiple days of the week over a predetermined period, biometric information (e.g., blood pressure, stress) during the drive along the relevant route can be stored based on time-interval images captured by the camera 1730 installed in the vehicle or the camera 430 of the user device 20 installed in the vehicle. The stored biometric information may include at least one of time-interval biometric information or average biometric information for the drive route. On the other hand, it is obvious to those skilled in the art that, not limited to the examples described and / or illustrated, other types of life events (e.g., exercise, studying, work life, etc.) and / or various types of biometric information (e.g., pulse rate, SpO2, etc.) may be stored.

[0118] According to various embodiments, an electronic device (e.g., server 10, user device 20) can store biometric information in a form related to attribute information for the life event based on the multiple images captured in operation 2405. For example, as shown in Figure 25a, an electronic device (e.g., server 10, user device 20) can store blood pressure information for different driving routes in the life event category of the user driving. That is, the attribute information is an element that defines the same life event category, and in the case of driving, it may be information such as the driving route (e.g., starting point, destination, route) and driving time. In the case of exercise, information such as the type of exercise and the duration of exercise may become attribute information for the exercise life event.

[0119] According to various embodiments, an electronic device (e.g., server 10, user device 20) can provide services based on the stored information in operation 2407. For example, the electronic device (e.g., server 10, user device 20) can provide cumulative information of the user by lifestyle category stored for a predetermined period to another external server, or it can autonomously provide services based on the predetermined cumulative information. The accumulated information is personalized information unique to the user and can be used as objective grounds for providing services optimized for the user by lifestyle category. As an example, when the electronic device (e.g., server 10, user device 20) recommends a route for traveling from a first location to a second location, as shown in Figure 25b, it can provide information on the stress index for each travel route from the first location to the second location based on the biometric information (e.g., stress index) for each route accumulated in the database 2500. Accordingly, the user can select a travel route that takes into account not only the travel time and distance, but also the stress index (or changes in biometric information) that the user can experience.

[0120] According to various embodiments, an electronic device (e.g., server 10, user device 20) can determine whether or not to provide a service based on the stored information, based on the user's current biometric information. For example, if the rPPG and / or biometric information values ​​currently measured in a non-contact manner satisfy preset conditions, the electronic device (e.g., server 10, user device 20) can initiate service provision based on the stored information. As an example, if at least one of the blood pressure or stress index is above a preset value, the need for reference to the user's biometric information during their daily life increases, and accordingly, the electronic device (e.g., server 10, user device 20) can perform service provision operations based on stored biometric information categorized by lifestyle. In other words, if the need for reference to the user's biometric information during their daily life is low, a general service may be provided.

[0121] 6. Kiosk Service Figure 26 is a diagram illustrating an example of a kiosk 2600 that provides contactless biometric information-based services according to various embodiments.

[0122] According to various embodiments, electronic devices (e.g., server 10, user device 20) can be embodied in providing predetermined services by measuring rPPG and biometric information based on the analysis of user images captured by kiosks 2600 placed in various locations, as described in "3. Table of Contents" regarding the rPPG and biometric information measurement methods. The kiosk 2600 can be placed in places where workers move, and can be placed in various locations such as entrances, exits, and corridors.

[0123] According to various embodiments, referring to Figure 26, the kiosk 2600 may include a communication circuit 2610, a processor 2620, a camera 2630, and a display 2640, and may be embodied to include many more devices, not limited to the illustrated example.

[0124] According to various embodiments, electronic devices (e.g., server 10, user device 20) can receive various types of information from the communication circuit 2610 of the kiosk 2600. For example, the various types of information may include images of workers captured by the camera 2630 and information about the shooting environment conditions.

[0125] According to various embodiments, an electronic device (e.g., server 10, user device 20) can transmit a control signal to the processor 2620 of the kiosk 2600 to output predetermined information (e.g., construction site-related information I) via the display 2640, based on the rPPG and biometric information that is subsequently analyzed. In this case, the construction site-related information I displayed via the display 2640 of the kiosk 2600 may include not only information received from the electronic device (e.g., server 10, user device 20) but also information received from other external servers (e.g., weather information).

[0126] 6.1 Provision of multiple location-based biometric information infrastructure services Figure 27 is a flowchart illustrating an example of the operation method of an electronic device (e.g., server 10, user device 20) for providing multiple location-based bio-information infrastructure services according to various embodiments. However, the operations may be performed in an order different from that described and / or illustrated, and more or fewer operations may be performed than those described and / or illustrated. Below, Figure 27 will be further described with reference to Figures 28 to 29.

[0127] Figure 28 is a diagram illustrating an example of collecting biometric information from a specific user from multiple kiosks located at different locations according to various embodiments. Figure 29 is a diagram illustrating an example of a service based on multiple location-based biometric information according to various embodiments.

[0128] According to various embodiments, an electronic device (e.g., server 10, user device 20) can acquire identification information for a specific user in operation 2701, identify multiple locations where the specific user is located in operation 2703, and store biometric information of the specific user for each of the multiple locations based on multiple images of the specific user taken using kiosks arranged at each of the multiple locations in operation 2705. For example, referring to Figure 29, biometric information for a specific user (or worker) can be acquired based on kiosks 2600 arranged at each of the multiple construction areas. For example, if a specific user enters and exits a first construction area at a first time, the first kiosk 2600a can provide the electronic device (e.g., server 10, user device 20) with an image of the specific user taken along with reference information (e.g., time, area). Furthermore, for example, if a specific user enters or leaves another construction area (e.g., second construction area, third construction area) at a different time after the first time, the electronic device (e.g., server 10, user device 20) can receive reference information (e.g., time, area) along with an image of the specific user taken by kiosks 2600b and 2600c located in the other construction areas (e.g., second construction area, third construction area). The electronic device (e.g., server 10, user device 20) can identify the specific user based on the face image included in the captured image, using pre-registered face images, and can store biometric information measured based on the captured image for each location of the identified specific user based on the reference information. The operation of measuring biometric information based on the captured image can be performed as described above in "3. Table of Contents," so redundant explanations are omitted.

[0129] According to various embodiments, the electronic device (e.g., server 10, user device 20) can provide services based on the multiple location-specific biometric data stored in operation 2707. For example, as shown in Figure 29(a), the electronic device (e.g., server 10, user device 20) can provide information on the progression of biometric data (e.g., blood pressure) for each construction area where a specific user has been sequentially located, via the kiosk 2600. Furthermore, the electronic device (e.g., server 10, user device 20) can estimate the biometric data that will be elevated in other construction areas where the specific user next works, based on the progression information of the biometric data. If the estimated biometric data exceeds a preset value, accidents can be prevented by preventing the specific user from entering other construction areas. Also, for example, as shown in Figure 29(b), the electronic device (e.g., server 10, user device 20) can select a location (e.g., work area) suitable for the user based on the location-specific biometric data and provide information on it, thereby ensuring that the user is placed in a work area where they can work more smoothly. For example, the work area in which the user's blood pressure was lowest can be identified as the work area that best suits the user. Furthermore, if there are multiple workers, the work area can be judged as suitable if the blood pressure is lower than the average blood pressure of each worker.

Claims

1. In the operation method of an electronic device, An operation to acquire multiple images of the user taken using the RGB camera of the aforementioned electronic device; An action to acquire a specific region for a specific body part of the user from each of the aforementioned multiple images; An operation to generate a plurality of first data related to the RGB color model for a specific region associated with each of the plurality of images; An operation to generate a plurality of second data related to the YCrCb color model based on the plurality of first data; An operation to generate first time series data related to the green channel based on the aforementioned plurality of first data; An operation to generate second time-series data related to the color difference channel based on the aforementioned plurality of second data; An operation to generate a third time series data based on adding the first time series data and the second time series data; and A method for operating an electronic device, including the operation of estimating the pulse rate of the user based on converting the third time-series data into the frequency domain.

2. The process of acquiring a specific area for a specific body part of the user from each of the aforementioned multiple images is as follows: An operation to identify a value for a specific color model for each of the aforementioned multiple images; and A method for operating an electronic device according to claim 1, comprising the operation of extracting a region from the identified values ​​that corresponds to a predetermined range of values.

3. The operation of generating multiple first data related to the RGB color model is as follows: The operation includes obtaining multiple values ​​for each of the multiple color channels of the RGB color model for each of the multiple images, wherein each of the multiple values ​​corresponds to a specific point in time. An operation to identify multiple interval values ​​for multiple time windows among the multiple values ​​for each of the multiple color channels; and A method for operating an electronic device according to claim 1, comprising the operation of generating the plurality of first data by adjusting the plurality of values ​​based on the average of the plurality of interval values.

4. A method for operating an electronic device according to claim 1, further comprising: performing at least one of the following operations on the plurality of first data: noise reduction operation using a signal filter or trend correction operation.

5. The operation for generating the first time-series data related to the green channel is as follows: The operation of generating a first sub-time series data by subtracting the value of the red channel from the value of the green channel, and generating a second sub-time series data by subtracting the value of the blue channel from the value of the green channel; and A method for operating an electronic device according to claim 1, further comprising the operation of generating first time series data by adding the first sub-time series data and the second sub-time series data.

6. The operation of generating the plurality of first data with the plurality of second data for the YCrCb color model; and A method for operating an electronic device according to claim 5, further comprising the operation of generating a second time series data by adding a third sub-time series data of a Cr channel and a fourth sub-time series data of a Cb channel based on the plurality of second data.

7. The operation of generating the aforementioned plurality of first data with the aforementioned plurality of second data for the CIE La*b* color model; and A method for operating an electronic device according to claim 5, further comprising the operation of generating a second time series data based on a fifth sub-time series data of channel a based on the plurality of second data.

8. A computer-readable recording medium that stores a program for executing the method of operating an electronic device described in any one of claims 1 to 7.

9. An electronic device, Includes at least one processor; wherein the at least one processor is: Multiple images of the user are obtained using an RGB camera. From each of the aforementioned multiple images, a specific region is obtained for a specific body part of the user. Multiple first data related to the RGB color model for a specific region associated with each of the multiple images are generated. Based on the aforementioned plurality of first data, a plurality of second data related to the YCrCb color model is generated. Based on the aforementioned plurality of first data, first time series data related to the green channel is generated. Based on the aforementioned plurality of second data, a second time-series data related to the color difference channel is generated. A third time series data is generated by adding the first time series data and the second time series data. An electronic device configured to estimate the pulse rate of the user based on converting the third time-series data into the frequency domain.

10. The at least one processor, as part of an operation to acquire a specific region for a specific body part of the user from each of the plurality of images: Identify the values ​​for a specific color model for each of the aforementioned multiple images, The electronic device according to claim 9, configured to extract a region from the identified values ​​that corresponds to a preset range of values.

11. The at least one processor, as part of an operation that generates a plurality of first data related to the RGB color model: Multiple values ​​are obtained for each of the multiple color channels in the RGB color model for each of the multiple images, and each of the multiple values ​​corresponds to a specific point in time. Among the multiple values ​​for each of the multiple color channels, multiple interval values ​​for each of the multiple time windows are identified, The electronic device according to claim 9, configured to generate the plurality of first data by adjusting the plurality of values ​​based on the average of the plurality of interval values.

12. The aforementioned at least one processor: The electronic device according to claim 9, further configured to perform noise reduction operations on the plurality of first data.

13. The aforementioned at least one processor: The first sub-time series data is generated by subtracting the value of the red channel from the value of the green channel, and the second sub-time series data is generated by subtracting the value of the blue channel from the value of the green channel. The electronic device according to claim 9, configured to generate first time series data by adding the first sub-time series data and the second sub-time series data.

14. The aforementioned at least one processor: The plurality of first data are generated using the plurality of second data for the YCrCb color model, The electronic device according to claim 13, configured to generate a second time series data by adding a third sub-time series data of the Cr channel and a fourth sub-time series data of the Cb channel based on the plurality of second data.

15. The aforementioned at least one processor: The plurality of first data are generated using the plurality of second data for the CIE La*b* color model, The electronic device according to claim 13, configured to generate a second time series data based on a fifth sub-time series data of channel a based on the plurality of second data.

Images