Electronic device for measuring remote photovolume pulse wave and providing service based on

Abstract

According to various embodiments, provided is an operation method of an electronic device for controlling a device in a vehicle, the operation method comprising: acquiring a plurality of images of at least one passenger in the vehicle captured by a camera; acquiring biological characteristic information of the at least one passenger based on the plurality of images; acquiring external environment information of the vehicle; and controlling the device in the vehicle based on the biometric information and the external environment information.

Description

Technical Field

[0001] [Cross-references to related applications] This application claims priority and benefit to Korean Patent Application No. 10-2024-0126412, filed with the Korean Intellectual Property Office on September 19, 2024, the entire contents of which are incorporated herein by reference.

[0002] This invention relates to an electronic device and its operation method for measuring and providing services based on multiple color models of remote photoplethysmography (rPPG). Background Technology

[0003] The most common technique for measuring photoplethysmography (PPG) using light involves analyzing the amount of transmitted light projected onto the human body. This method is explained by Beer-Lambert's law, which states that absorbance is proportional to the concentration of the absorbing material and the thickness of the absorbing layer. According to this law, changes in transmitted light produce a signal proportional to changes in the volume of the light-transmitting material. Therefore, even without knowing the absorbance of the material, PPG can be used to identify the state of organs such as the heart.

[0004] In recent years, based on PPG technology, rPPG technology has emerged. As the most popular technology for identifying heartbeat-related signals using PPG, there is a method that involves directly contacting a device with a camera and a near-field light source (such as a smartphone) with the human body and illuminating it with light, then immediately measuring the transmitted light to obtain PPG data. Currently, rPPG-related technologies for identifying changes in blood vessel volume using signals obtained from images captured by a camera are being researched and developed.

[0005] Because rPPG technology does not require the target object to come into contact with the measuring device, it is widely used in devices and locations equipped with cameras, such as airport immigration checkpoints and telemedicine hospitals.

[0006] However, during the process of using a camera to capture the target object, the signal of rPPG technology is easily affected by ambient light and noise generated by the movement of the target object. Therefore, the technology of extracting only the signal related to the volume change of the target object from the captured image can be regarded as the core technology for measuring biometric signals using rPPG. Summary of the Invention

[0007] Technical issues When measuring rPPG using relevant technical methods, images of the target object are typically analyzed using a camera. In this case, rPPG can be measured based on pixel values ​​extracted from various color models such as RGB, YCbCrCg, and CIELa*b*. However, noise from the shooting environment (such as camera shake, external illumination, and shadows) is reflected in the pixel values, making accurate measurement of the target object's rPPG difficult. According to various embodiments, electronic devices and their operating methods can use multiple color models, including RGB, YCbCrCg, and CIELa*b*, in a complementary manner to measure rPPG, thereby reducing noise and improving the accuracy of rPPG measurement of the target object. According to various embodiments, electronic devices and their operating methods can also improve the accuracy of rPPG measurement on the target object by simultaneously using an RGB camera and an IR camera, based on color model values ​​and infrared (IR) values.

[0008] Currently, technologies are being developed for self-service kiosks for vehicle or human resource management placed at doorways to provide various services based on biometric information measured using rPPG. However, due to the low accuracy of rPPG measurements and insufficient research on methods for associating the services provided by the vehicle or human resource management self-service kiosks, the practically available services are very limited. According to various embodiments, electronic devices and their operating methods can provide highly utilized vehicle and human resource management self-service kiosks based on high-quality rPPG measurements that can be measured in all environments.

[0009] Technical solution According to various embodiments, an operating method for controlling an electronic device within a vehicle is provided, the operating method comprising: acquiring multiple images of at least one passenger inside the vehicle captured by a camera; acquiring biometric information of the at least one passenger based on the multiple images; acquiring external environmental information of the vehicle; and controlling the device inside the vehicle based on the biometric information and the external environmental information.

[0010] The biometric information may include at least one of the following: heart rate, blood oxygen saturation, blood pressure, and stress index of the at least one passenger.

[0011] The external environment information may include at least one of the following: temperature, wind direction, wind speed, and weather conditions corresponding to the vehicle's location.

[0012] The equipment within the vehicle may include at least one of the following: steering equipment, engine, seats, lighting equipment, air conditioning and heating system, speakers, windows, and doors.

[0013] When controlling the devices inside the vehicle based on the biometric information and the external environment information, the controlled devices inside the vehicle can change according to the values ​​corresponding to the biometric information and the values ​​corresponding to the external environment information.

[0014] The at least one passenger includes a driver and a fellow passenger. The operation method may further include: acquiring multiple first images of the driver and multiple second images of the fellow passenger using the camera; acquiring first biometric information of the driver based on the multiple first images; acquiring second biometric information of the fellow passenger based on the multiple second images; and controlling the devices within the vehicle based on the first and second biometric information.

[0015] When controlling the device inside the vehicle based on the biometric information and the external environment information, the controlled device inside the vehicle can change according to the value corresponding to the first biometric information and the value corresponding to the second biometric information.

[0016] The operation method may further include: identifying that the vehicle's mode is set to autonomous driving mode; and changing the seat angle of the at least one passenger based on the biometric information.

[0017] The operation method may further include: acquiring the biometric information based on the multiple images at multiple time periods during the vehicle's operation; storing the biometric information and the vehicle's driving route in an associated form; and providing a predetermined service based on the stored biometric information and the driving route.

[0018] The acquisition of the biometric information may include: identifying shooting environment conditions; when the identified shooting environment conditions do not meet predetermined data fusion conditions, acquiring a predetermined body region of the at least one passenger from each of the plurality of images; generating a plurality of first data associated with an RGB color model of the body region from each of the plurality of images; generating a plurality of second data associated with a YCrCb color model based on the plurality of first data; generating a first time series data associated with a green channel based on the plurality of first data; generating a second time series data associated with a color difference channel based on the plurality of second data; generating a first integrated time series data by combining the first time series data and the second time series data; estimating the heart rate of the at least one passenger by converting the first integrated time series data into the frequency domain; and acquiring the biometric information based on the estimated heart rate.

[0019] The acquisition of the biometric information may include: when the identified shooting environment conditions meet the predetermined data fusion conditions, acquiring multiple images of the at least one passenger using an infrared camera; acquiring multiple body regions of the at least one passenger from the multiple images; extracting time-series data of each body region from the multiple body regions; generating second integrated time-series data based on the time-series data of each body region from the multiple body regions; generating final time-series data based on the first integrated time-series data and the second integrated time-series data; and acquiring the biometric information based on the final time-series data.

[0020] According to various embodiments, an electronic device for controlling devices inside a vehicle is provided, the electronic device comprising: at least one processor; the at least one processor being configured to: acquire multiple images of at least one passenger inside the vehicle captured by a camera; acquire biometric information of the at least one passenger based on the multiple images; acquire external environmental information of the vehicle; and control the devices inside the vehicle based on the biometric information and the external environmental information.

[0021] The at least one processor may also be configured to change the devices within the controlled vehicle based on values ​​corresponding to the biometric information and values ​​corresponding to the external environment information.

[0022] The at least one passenger may include a driver and a fellow passenger. The at least one processor may also be configured to: acquire multiple first images of the driver and multiple second images of the fellow passenger using the camera; acquire first biometric information of the driver based on the multiple first images; acquire second biometric information of the fellow passenger based on the multiple second images; and control the devices within the vehicle based on the first and second biometric information.

[0023] The at least one processor may also be configured to: change the equipment within the controlled vehicle based on the value corresponding to the first biometric information and the value corresponding to the second biometric information.

[0024] The at least one processor may also be configured to: identify the vehicle mode as being set to autonomous driving mode; and change the seat angle of the at least one passenger based on the biometric information.

[0025] The at least one processor may also be configured to: acquire the biometric information based on the plurality of images at multiple time periods during the driving of the vehicle; store the biometric information and the driving route of the vehicle in an associated form; and provide a predetermined service based on the stored biometric information and the driving route.

[0026] The at least one processor may also be configured to: identify shooting environment conditions; when the identified shooting environment conditions do not meet predetermined data fusion conditions, acquire a predetermined body region of the at least one passenger from each of the plurality of images; generate a plurality of first data associated with an RGB color model of the body region from each of the plurality of images; generate a plurality of second data associated with a YCrCb color model based on the plurality of first data; generate a first time series data associated with a green channel based on the plurality of first data; generate a second time series data associated with a color difference channel based on the plurality of second data; generate a first integrated time series data by combining the first time series data and the second time series data; estimate the heart rate of the at least one passenger by converting the first integrated time series data into the frequency domain; and acquire the biometric information based on the estimated heart rate.

[0027] The at least one processor may also be configured to: when the identified shooting environment conditions meet the predetermined data fusion conditions, acquire multiple images of the at least one passenger using an infrared camera; acquire multiple body regions of the at least one passenger from the multiple images; extract time-series data of each body region from the multiple body regions; generate second integrated time-series data based on the time-series data of each body region in the multiple body regions; generate final time-series data based on the first integrated time-series data and the second integrated time-series data; and acquire the biometric information based on the final time-series data.

[0028] The technical solutions according to various embodiments are not limited to the above-described technical solutions, and based on the detailed description and accompanying drawings, those skilled in the art can clearly understand other technical solutions not mentioned above.

[0029] Beneficial effects According to various embodiments, an electronic device and its operating method may be provided, which uses multiple color models, including RGB color model, YCbCrCg color model, CIELa*b* color model, etc., in a complementary manner to measure rPPG, thereby reducing noise values ​​and improving the accuracy of rPPG measurement of target objects.

[0030] According to various embodiments, an electronic device and its operating method may be provided to improve the measurement accuracy of rPPG on a target object by simultaneously using an RGB camera and an infrared camera, based on color model values ​​and infrared values.

[0031] According to various embodiments, an electronic device and its operating method may be provided to provide highly utilized self-service terminals for vehicle and human resource management based on high-quality rPPG measurable in all environments. Attached Figure Description

[0032] Figure 1 A view illustrating examples of components of a contactless biometric information system according to various embodiments; Figure 2 A view illustrating examples of other components of a contactless biometric information system according to various embodiments; Figure 3 A block diagram illustrating examples of server components according to various embodiments; Figure 4 A block diagram illustrating examples of components of a user equipment according to various embodiments; Figure 5 A flowchart illustrating an example of an operation method for an electronic device based on multiple color models for measuring rPPG and biometric information according to various embodiments; Figure 6 A view illustrating examples of modules for performing operations of measuring rPPG and biometric information using a visible light camera according to various embodiments; Figure 7 A view for illustrating operational examples of preprocessing RGB data according to various embodiments; Figure 8 A view used to illustrate operational examples using multiple color models according to various embodiments; Figure 9 A flowchart illustrating an example of an electronic device for measuring rPPG and biometric information based on infrared channel values ​​in each of multiple regions, according to various embodiments; Figure 10 A view illustrating examples of modules that utilize an infrared (IR) camera to perform operations for measuring rPPG and biometric information according to various embodiments; Figure 11 A view for illustrating operational examples of using infrared channel values ​​for each of multiple regions according to various embodiments; Figure 12A flowchart illustrating an example of an operation method for an electronic device that measures rPPG using a visible light camera and an infrared camera based on shooting environment conditions according to various embodiments; Figure 13 A view for illustrating embodiments of measuring rPPG using a visible light camera and an infrared camera according to various embodiments; Figure 14 A flowchart illustrating an example of an operation method for an electronic device that measures blood pressure based on a blood pressure artificial intelligence model according to various embodiments; Figure 15 A flowchart illustrating an example of an operation method for an electronic device that measures pressure based on a pressure analysis artificial intelligence model according to various embodiments; Figure 16 A view for illustrating examples of modules performing stress analysis operations according to various embodiments; Figure 17 A view for illustrating examples of vehicles providing services based on contactless biometric information according to various embodiments; Figure 18 A flowchart illustrating an example of an operation method of an electronic device for providing mobile travel services based on contactless biometric information and external information, according to various embodiments; Figure 19 A view for illustrating examples of mobile travel services provided based on contactless biometric information and external information according to various embodiments; Figure 20 A flowchart illustrating an example of an operational method of an electronic device for providing a mobility service that further considers passenger information, according to various embodiments; Figure 21 A view used to illustrate examples of the measurement of rPPG / biometric information of passengers according to various embodiments; Figure 22 A flowchart illustrating an example of an operational method of an electronic device for providing mobility services that further take into account vehicle modes, according to various embodiments; Figure 23 A view for illustrating examples of a mobility service with a continuously deformable seat configuration during autonomous driving, according to various embodiments; Figure 24 A flowchart illustrating an example of an operation method of an electronic device that provides a service based on continuous accumulation of biometric information according to various embodiments; Figure 25A A view for illustrating an embodiment of accumulating biometric information measured for each travel path within a preset time in a database, according to various embodiments; Figure 25BA view for illustrating service examples that provide driving recommendation information based on accumulated biometric information according to various embodiments; Figure 26 A view for illustrating examples of self-service terminals that provide services based on contactless biometric information according to various embodiments; Figure 27 A flowchart illustrating an example of an operational method of an electronic device for providing a service based on biometric information of each of a plurality of locations, according to various embodiments; Figure 28 A view for illustrating examples of collecting biometric information of a specific user from a self-service terminal placed in each of multiple locations, according to various embodiments; Figure 29 This is a view used to illustrate service examples based on biometric information from each of multiple locations, according to various embodiments. Detailed Implementation

[0033] It should be understood that the various embodiments of this disclosure and the terminology used herein are not intended to limit the technical features set forth herein to the particular embodiments, but rather to include various modifications, equivalents, or substitutions of the corresponding embodiments. Regarding the description of the drawings, similar reference numerals may be used to refer to similar or related elements. It should be understood that, unless the relevant context clearly indicates otherwise, the singular form of a noun corresponding to an item may include one or more of that item. As used herein, each phrase such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C” may include at least one or all possible combinations of the items listed together in the corresponding phrase. As used herein, terms such as “first” and “second” or “first” and “second” may be used only to simply distinguish one corresponding component from another, without otherwise limiting these components (e.g., in terms of importance or order). It should be understood that if an element (e.g., the first element) is referred to as “coupled,” “coupled to,” “connected,” or “connected to” another element (e.g., the second element), whether or not it carries the terms “operationally” or “communically”, it means that the element can be coupled to the other element directly (e.g., via a wire), wirelessly, or via a third element.

[0034] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and is used interchangeably with other terms such as "logic," "logic block," "part," or "circuit." A module may be a single integrated component adapted to perform one or more functions, or the smallest unit or part of such a single integrated component. For example, according to one embodiment, a module may be implemented as an application-specific integrated circuit (ASIC).

[0035] Various embodiments of this disclosure can be implemented as software (e.g., a program) comprising one or more instructions stored in a machine-readable storage medium (e.g., internal memory). For example, a processor of a machine (e.g., an electronic device) can invoke and execute at least one of the one or more instructions stored in the storage medium. This enables the machine to be operated according to the invoked at least one instruction to perform at least one function. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" means only that the storage medium is a tangible device, excluding signals (e.g., electromagnetic waves), but this term does not distinguish between cases where data is semi-permanently stored in the storage medium and cases where data is temporarily stored in the storage medium.

[0036] According to one embodiment, methods according to various embodiments of this disclosure can be included and provided in a computer program product. The computer program product can be traded as a product between a seller and a buyer. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory, CD-ROM), or distributed online (e.g., downloaded or uploaded) via an app store (e.g., the Play Store™), or directly between two user devices (e.g., smartphones). If distributed online, at least a portion of the computer program product can be temporarily generated or at least temporarily stored in a machine-readable storage medium, such as the memory of a manufacturer's server, an app store's server, or a relay server.

[0037] According to various embodiments, each of the above components (e.g., a module or program) may include a single entity or multiple entities, and some of the multiple entities may be arranged separately in other components. According to various embodiments, one or more of the above components or operations may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, the integrated component can still perform one or more functions of each of the multiple components in the same or similar manner as before integration by the corresponding one of the multiple components. According to various embodiments, operations performed by a module, program, or other component may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more operations may be performed in a different order or omitted, or one or more other operations may be added.

[0038] According to various embodiments, this disclosure provides an operating method for an electronic device, comprising: acquiring multiple images of a user captured by an RGB camera of the electronic device; acquiring a specific region on a specific body part of the user from the multiple images; generating multiple first data associated with an RGB color model in the specific region associated with the multiple images; generating multiple second data associated with a YCrCb color model based on the multiple first data; generating a first time series data associated with a green channel based on the multiple first data; generating a second time series data associated with a color difference channel based on the multiple second data; generating third time series data by combining the first time series data and the second time series data; and estimating the user's heart rate based on converting the third time series data into the frequency domain.

[0039] According to various embodiments, obtaining a specific region on a specific body part of the user from the plurality of images may include: identifying values ​​on a specific color model on the plurality of images; and extracting a region from the identified values ​​that corresponds to a range of preset values.

[0040] According to various embodiments, generating a plurality of first data associated with an RGB color model may include: for each of the plurality of images, obtaining a plurality of values ​​for each of a plurality of color channels of the RGB color model, each of the plurality of values ​​corresponding to a specific time point; identifying a plurality of interval values ​​for a plurality of time windows among the plurality of values ​​for the plurality of color channels; and generating the plurality of first data by adjusting the plurality of values ​​based on the average of the plurality of interval values.

[0041] According to various embodiments, the provided operating method may further include performing at least one of a noise removal operation or a trend correction operation on the plurality of first data using a signal filter.

[0042] According to various embodiments, generating the first time series data associated with the green channel may include: generating first sub-time series data by subtracting the value of the red channel from the value of the green channel, and generating second sub-time series data by subtracting the value of the blue channel from the value of the green channel; and generating the first time series data by combining the first sub-time series data and the second sub-time series data.

[0043] According to various embodiments, the provided operation method may further include: generating a plurality of second data for the YCrCb color model from the plurality of first data; and generating the second time series data by combining a third sub-time series data of the Cr channel and a fourth sub-time series data of the Cb channel based on the plurality of second data.

[0044] According to various embodiments, this disclosure provides an electronic device including at least one processor, the at least one processor being configured to: acquire multiple images of a user captured by an RGB camera; acquire a specific region on a specific body part of the user from the multiple images; generate multiple first data associated with an RGB color model in the specific region associated with the multiple images; generate multiple second data associated with a YCrCb color model based on the multiple first data; generate first time-series data associated with a green channel based on the multiple first data; generate second time-series data associated with a color difference channel based on the multiple second data; generate third time-series data by combining the first time-series data and the second time-series data; and estimate the user's heart rate based on converting the third time-series data into the frequency domain.

[0045] According to various embodiments, as at least part of the operation of obtaining a specific region on a specific body part of the user from the plurality of images, the at least one processor may be configured to: identify values ​​on a specific color model on the plurality of images; and extract a region corresponding to a range of preset values ​​from the identified values.

[0046] According to various embodiments, as at least part of the operation of generating a plurality of first data associated with an RGB color model, the at least one processor may be configured to: for each of the plurality of images, acquire a plurality of values ​​for each of a plurality of color channels of the RGB color model, each of the plurality of values ​​corresponding to a specific time point; identify a plurality of interval values ​​for a plurality of time windows among the plurality of values ​​of the plurality of color channels; and generate the plurality of first data by adjusting the plurality of values ​​based on the average of the plurality of interval values.

[0047] According to various embodiments, the at least one processor may also be configured to perform at least one of the following operations on the plurality of first data: noise removal operation using a moving average filter, high-frequency noise removal operation, or trend correction operation using detrending.

[0048] 1. Non-contact biometric information system The non-contact biometric information system 1 according to various embodiments can be a system implemented to measure the rPPG of a target object by means of an image captured by a camera, and to obtain biometric information (e.g., blood pressure, heart rate, stress index, etc.) based on the rPPG measurement to provide various services. rPPG can refer to the blood volume (PPG) measured from a blood vessel near the skin using a non-contact method (i.e., a remote method). The non-contact biometric information system 1 can improve the quality of the measured rPPG by reducing the influence of noise caused by the shooting environment (e.g., illumination, shadows, jitter) reflected in the image captured by the camera, thus providing highly practical services. This disclosure will be described in detail below.

[0049] 2. Components of the non-contact biometric information system 1 Figure 1 A view illustrating examples of components of a contactless biometric information system 1 according to various embodiments.

[0050] Reference Figure 1 The contactless biometric information system 1 according to various embodiments may include a server 10 and a user device 20. All servers 10 and user devices 20 may be defined as "electronic devices".

[0051] According to various embodiments, server 10 can be implemented to measure rPPG and biometric information attributable to rPPG based on images (e.g., images of body parts) of user U captured by camera C of user device 20, and to provide various types of services. For example, refer to Figure 1 Server 10 can store a first program 30a, which can measure the user U's rPPG and biometric information attributable thereto based on user images received from user equipment 20 and the first program 30a, and can provide various types of services. The first program 30a can be implemented as at least one of a software module, program, various information (parameters), or artificial intelligence (AI) model for analyzing images of user U.

[0052] According to various embodiments, user equipment 20 may be an electronic device of user U. User equipment 20 may include not only electronic devices carried by user U, such as smartphones, tablets, laptops, wearable devices, head-mounted display (HMD) devices, etc., but also installable electronic devices such as self-service terminals, personal computers (PCs), televisions (TVs), etc., but is not limited thereto, and may also include various types of electronic devices including cameras, to include hardware for capturing images of user U. User equipment 20 may display a graphical user interface implemented to provide user U with image capture functionality based on the execution of the second program 30b (e.g., an execution screen providing image capture functionality may be displayed), and may provide at least one of rPPG, biometric information, or information about various types of services received from server 10.

[0053] Figure 2 A view illustrating examples of other components of the contactless biometric information system 1 according to various embodiments. (Refer to...) Figure 2 The contactless biometric information system 1 may include, for example: Figure 1 The server 10 and user equipment shown are only implemented with respect to program 30. Program 30 may include all of the first program 30a and the second program 30b described above. Therefore, the functions of the server 10 can be operated independently (or on the device) within the user equipment 20. In other words, the user equipment 20 can be implemented to capture images on the user U using camera C, measure rPPG and biometric information attributable thereto by analyzing the captured images, and provide various types of services based on the execution of program 30.

[0054] It will be apparent to those skilled in the art that the operation of the electronic device according to the various embodiments described below is to be understood as the operation of server 10, the operation of user equipment 20, or the coordinated operation of server 10 and user equipment 20.

[0055] 2.1 Composition of electronic devices The following will combine Figure 3 and Figure 4 Examples of the composition of an electronic device constituting a contactless biometric information system 1 according to various embodiments will be described.

[0056] 2.1.1 Composition of Server 10 Figure 3 A block diagram illustrating examples of components of server 10 according to various embodiments.

[0057] Reference Figure 3According to various embodiments, server 10 may include a first processor 210, a first communication circuit 220, and a first memory 230. However, this disclosure is not limited to the examples described and / or shown, and server 10 may include more components.

[0058] According to various embodiments, the first processor 210 can control the overall operation of the server 10. For this purpose, the first processor 210 can perform calculations and processing on various types of information and control the operation of components of the server 10 (e.g., the first communication circuit 220). According to one embodiment, as at least part of data processing or calculation, the first processor 210 can store commands or data received from another component into volatile memory, process the commands or data stored in volatile memory, and store the result data into non-volatile memory. According to one embodiment, the first processor 210 may include a main processor (not shown) (e.g., a central processing unit or application processor) or an auxiliary processor (not shown) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor), the auxiliary processor being able to operate independently of the main processor or in conjunction with the main processor. For example, when the server 10 includes a main processor (not shown) and an auxiliary processor (not shown), the auxiliary processor (not shown) may be configured to consume less power than the main processor (not shown) or dedicated to a specific function. The auxiliary processor (not shown) may be implemented independently of the main processor (not shown) or as part of it.

[0059] According to one embodiment of this application, an auxiliary processor (not shown) may replace the main processor (not shown) when the main processor (not shown) is inactive (e.g., in a sleep state), or, when the main processor (not shown) is active (e.g., in an application execution state), control, together with the main processor (not shown), at least some functions or states associated with at least one component of the server 10 (e.g., the first communication circuit 220). According to one embodiment, the auxiliary processor (not shown) (e.g., an image signal processor or a communication processor) may be implemented as part of other components (e.g., the first communication circuit 220) associated with its functions. According to one embodiment, the auxiliary processor (not shown) (e.g., a neural network processing unit) may include hardware structures dedicated to AI model processing. The AI ​​model may be generated through machine learning. Such learning may be performed in the server 10 that performs the AI, or through a separate server (e.g., a learning server). For example, the learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The AI ​​model may include multiple artificial neural network layers. Artificial neural networks may include, but are not limited to, deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), deep Q-networks, or combinations of two or more of the above. AI models may include software structures in addition to hardware structures, or as an alternative.

[0060] In the following description, unless otherwise stated, the operation of server 10 is to be understood as being performed under the control of the first processor 210.

[0061] According to various embodiments, the first communication circuit 220 can communicate with an external device (e.g., user equipment 20). For example, the first communication circuit 220 can be connected to a network via wireless or wired communication to establish communication with the external device (e.g., user equipment 20) and exchange information and / or data through the established communication. Wireless communication may include cellular communication using at least one of LTE, LTE-Advanced (LTE-A), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro), or global system for mobile communications (GSM). According to one embodiment, wireless communication may include at least one of wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), Zigbee, near field communication (NFC), magnetic secure transmission, radio frequency (RF), or body area network (BAN). According to one embodiment, wireless communication may include a Global Navigation Satellite System (GNSS). GNSS may be, for example, a Global Positioning System (GPS), a Global Navigation Satellite System (GLONASS), a BeiDou Navigation Satellite System (hereinafter referred to as "BeiDou"), or Galileo, or the European Global Navigation Satellite System. In this disclosure, "GPS" and "GNSS" are used interchangeably.Wired communication may include at least one of the following: Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), Recommended Standard 232 (RS-232), Low Power Line Communication (LPL) or a plain old telephone service (POTS). Networks may include at least one of the following: telecommunications networks, such as computer networks (e.g., LANs or WANs), the Internet, or telephone networks.

[0062] According to one embodiment of this application, memory 230 can store various types of information. Memory 230 can store data temporarily or semi-permanently. For example, memory 230 can store an authoring module 200 for creating active experience files. Server 10 (e.g., first processor 210) can perform the operation of creating active experience files based on authoring module 200.

[0063] 2.1.2 Composition of User Equipment 20 Figure 4 A block diagram illustrating examples of components of a user equipment 20 according to various embodiments.

[0064] According to various embodiments, refer to Figure 4 User equipment 20 may include a second processor 410, a second communication circuit 420, a camera 430, a touch screen 440, and a second memory 450. The second processor 410 may be implemented in the same manner as the first processor 210 described above, the second communication circuit 420 may be implemented in the same manner as the first communication circuit 220 described above, and the second memory 450 may be implemented in the same manner as the first memory 230, therefore, repeated descriptions are omitted.

[0065] According to various embodiments, camera 430 may be implemented to capture images of a user of user equipment 20. For example, camera 430 may include at least one of a visible light camera (or an RGB camera) or an infrared (IR) camera. A visible light camera may be a camera capable of capturing images in the general visible light region (e.g., the 380 nm to 780 nm band) (i.e., image capture based on visible light), and an infrared camera may be a camera capable of capturing images in the infrared region (e.g., the 700 nm to 1000 nm wavelength range) (i.e., image capture based on infrared light). Depending on the type of user equipment 20, user equipment 20 may be implemented in the form of including a visible light camera, an infrared camera, or both. When the user equipment is implemented in the form of including both a visible light camera and an infrared camera, user equipment 20 may measure rPPG based on images in the visible light region and images in the infrared region, as will be described in detail below.

[0066] According to various embodiments, the touchscreen 440 may be implemented to display a graphical user interface including predetermined information and to acquire user input received on the graphical user interface. For example, user equipment 20 (e.g., second processor 410) may display a graphical user interface (e.g., an execution screen) including at least one of rPPG, biometric information, or multiple services received from server 10. In another example, user equipment 20 (e.g., second processor 410) may display a graphical user interface (e.g., an execution screen) including a menu screen and / or icons for controlling camera 430, and may activate camera 430 to capture user images based on user input on icons received via touchscreen 440.

[0067] 3. rPPG and biometric information measurement methods 3.1 rPPG and biometric information measurement based on multiple color models Figure 5 This is a flowchart illustrating an example of an operational method for an electronic device (e.g., server 10, user device 20) that measures rPPG and biometric information based on multiple color models according to various embodiments. However, operations may be performed in a different order than those described and / or shown, and more or fewer operations may be performed than those described and / or shown. The following will combine... Figures 6 to 8 right Figure 5 Please provide an explanation.

[0068] Figure 6 This is a view illustrating examples of modules that utilize a visible light camera to perform operations for measuring rPPG and biometric information according to various embodiments. Figure 7 This is a view used to illustrate operational examples of preprocessing RGB data according to various embodiments. Figure 8This is a view used to illustrate operational examples of using multiple color models according to various embodiments.

[0069] According to various embodiments, an electronic device (e.g., server 10, user device 20) may acquire multiple images of a user captured by an RGB camera in operation 501, identify specific regions from the multiple images in operation 503, and generate multiple first data associated with a first color model on the specific regions of the multiple images in operation 505. For example, the electronic device (e.g., server 10, user device 20) may use a visible light camera in camera 430 to acquire RGB data of a specific body part (e.g., face) of a user within a preset time period. For example, user device 20 may capture multiple images of a user within a preset time period using a visible light camera upon user request. See also... Figures 6 to 7 Electronic devices (e.g., server 10, user equipment 20) can, based on the execution of program 300 (e.g., second program 30b or program 30), use the facial region extraction module 610 to identify the skin region R2 within each of the multiple images (or image frames) frames #1, ..., #n captured by the user equipment 20, and use the RGB data generation module 620 to obtain the RGB color model values ​​on the skin region R2. This disclosure is not limited to the above example; it can also extract skin regions from other body parts (e.g., thighs, forearms) besides the skin region R2 within the user's facial region R1.

[0070] According to various embodiments, as at least part of the operation of using the facial region extraction module 610 to identify the skin region within the face, an electronic device (e.g., server 10, user device 20) may identify the facial region R1 from multiple captured image frames #1, ..., #n based on an object recognition algorithm, convert the RGB color model value of each pixel in the multiple pixels included in the identified facial region R1 into a YCrCb color model value or an HSV color model value, and then identify at least a portion of the pixels that are greater than or equal to a preset value as the skin region R2 within the facial region R1.

[0071] According to various embodiments, as at least part of the operation of obtaining RGB color model values ​​on skin region R2 using RGB data generation module 620, an electronic device (e.g., server 10, user device 20) may apply Gaussian blur to skin region R2 of multiple image frames #1, ..., frame #n to remove noise, and obtain the average value of the B channel (blue channel), the average value of the R channel (red channel), and the average value of the G channel (green channel) of multiple pixels included in skin region R2 of multiple image frames #1, frame #2, frame #3, frame #4, ..., frame #n after noise removal (i.e., the average value of each channel of the RGB color model on skin region R2). (Refer to...) Figure 7Electronic devices (e.g., server 10, user device 20) can perform mean centering on the average values ​​of each channel of the RGB color model on the skin region R2 of multiple images (frame #1, ..., frame #n). For example, electronic devices (e.g., server 10, user device 20) can adjust (or change, control, correct) the average values ​​of each channel of the RGB color model on the skin region R2 of a subset of images contained within time windows W1, W2, corresponding to a preset number of images (or a preset time) (i.e., mean centering). For example, mean centering on the RGB color model on the skin region R2 can be performed according to the following formula 1: [Formula 1]

[0072] Where V'a is the value of a specific channel (B channel, G channel, R channel) of a specific image frame after mean centering, Va is the value of a specific channel of a specific image frame before mean centering, Vi is the value of a specific channel of each image frame included in a specific time window, i is the identifier of the image frame in the specific time window, n is the number of image frames included in the specific time window, and m is the number of time windows that overlap with the specific image frame.

[0073] That is, according to Formula 1, the average value of a specific channel contained within a specific time window in a specific image can be subtracted from the value of a specific channel on the skin region R2 of a specific image. In this case, refer to... Figure 7 Multiple time windows can be implemented and can be set to overlap each other. Therefore, when multiple time windows are set for a specific image, the average value of the specific channel of the multiple time windows can be subtracted from the value of the specific channel on the skin region R2 of the specific image. The subtracted average value of the specific channel is then divided by the number (m) of the multiple overlapping time windows and then subtracted again.

[0074] Electronic devices (e.g., server 10, user equipment 20) may perform at least one of amplitude correction or trend correction based on the average value, in addition to mean centering.

[0075] According to various embodiments, refer to Figure 6 Electronic devices (e.g., server 10, user equipment 20) can use the first preprocessing module 630 to perform preprocessing operations on the RGB color model values ​​of the skin region R2 of multiple images. The preprocessing operations may include at least one of operations such as removing noise (e.g., high-frequency noise, low-frequency noise) using a signal filter (e.g., a moving average filter) or including a trend correction operation that includes detrending.

[0076] According to various embodiments, in operation 507, an electronic device (e.g., server 10, user device 20) can generate multiple sets of second data associated with a second color model (e.g., YCrCb, CIELa*b*) based on multiple sets of first data (e.g., RGB color model values ​​on skin region R2). For example, referring to... Figure 6 and Figure 8 Electronic devices (e.g., server 10, user equipment 20) can use time series data generation modules 640 (e.g., first to third time series data generation modules 640a, 640b, 640c) to convert the RGB color model values ​​of skin regions R2 on multiple images into values ​​of other color models. Other color models may include at least one of the YCrCb color model or the CIE La*b* color model, but are not limited to the examples above.

[0077] According to various embodiments, electronic devices (e.g., server 10, user device 20) can generate first time-series data associated with the green channel based on multiple first data (e.g., RGB color model values) in operation 509, generate at least one second time-series data associated with the color difference channel based on multiple second data (e.g., YCrCb color model values) in operation 511, and generate integrated time-series data based on the combination of the first and second time-series data in operation 513. For example, electronic devices (e.g., server 10, user device 20) can generate final time-series data (e.g., integrated time-series data) by combining values ​​of multiple color models (e.g., RGB color model, YCrCb color model, or CIE La*b* color model) on the skin region R2 obtained as a result. By combining these values ​​to compensate for the shortcomings of different color models, noise caused by the shooting environment (e.g., illumination, shadows, camera shake) during user photography can be reduced, and higher quality rPPG can be obtained.

[0078] In one embodiment, reference is made to Figure 8 Electronic devices (such as server 10 and user equipment 20) can use the first time-series data generation module 640a to generate first time-series data 810 by adding a first value (green channel minus red channel) and a second value (green channel minus blue channel) to each of multiple images using the RGB color model values ​​on the skin region R2. The green channel best reflects blood volume, while the red and blue channels contained in the ambient light during shooting may introduce errors. Therefore, the first time-series data value after subtracting the red and blue channels from the green channel can better reflect blood volume and reduce the errors introduced by the red and blue channels of the external light source.

[0079] In one embodiment, reference is made to Figure 8Electronic devices (such as server 10, user equipment 20) can use the second time series data generation module 640b to add all color difference channels (Cr channel, Cb channel) to each of multiple images using the YCrCb color model values ​​on the skin region R2, thereby generating second time series data 820.

[0080] In one embodiment, reference is made to Figure 8 Electronic devices (such as server 10, user equipment 20) can use the third time series data generation module 640c to generate third time series data 830 containing only the a value by using the CIE La*b* color model value on the skin region R2.

[0081] Electronic devices (e.g., server 10, user device 20) can use the rPPG generation module 650 to combine at least a portion of the generated time-series data 810, 820, and 830 to generate a third time-series data. For example, electronic devices (e.g., server 10, user device 20) can combine the first time-series data 810 with the second time-series data 820 to generate integrated time-series data. Thus, the values ​​of the first time-series data 810, after removing the red and blue channels, can be supplemented by the second time-series data 820, which includes the color difference channels. Electronic devices (e.g., server 10, user device 20) can add the first time-series data 810, the second time-series data 820, and the third time-series data 830 to generate integrated time-series data. Electronic devices (e.g., server 10, user device 20) can combine the second time-series data 820 with the third time-series data 830 to generate integrated time-series data. The generated integrated time-series data represents blood flow velocity and can be defined as an rPPG signal.

[0082] According to various embodiments, electronic devices (e.g., server 10, user equipment 20) may perform preprocessing operations on multiple time series data 810, 820, 830 before combining them. For example, the preprocessing operations may include Z-score operations.

[0083] According to various embodiments, in operation 515, electronic devices (e.g., server 10, user device 20) can acquire user biometric information based on integrated time-series data. For example, electronic devices (e.g., server 10, user device 20) can use biometric information acquisition module 660 to analyze the integrated time-series data to acquire biometric information. Biometric information may include heart rate, blood pressure, stress index, etc. For example, in operation 515, electronic devices (e.g., server 10, user device 20) can convert the integrated time-series data into the frequency domain, extract frequency values ​​corresponding to the heart rate range (40 bpm to 240 bpm), use a bandpass filter to extract a preset frequency band containing the highest frequency (e.g., the frequency band corresponding to 20 bpm) from the extracted frequency values, and extract the heart rate based on the peak-to-peak distance within the extracted frequency band. Electronic devices (e.g., server 10, user device 20) can estimate other biometric information such as blood pressure based on the extracted heart rate.

[0084] 3.2 Measurement of rPPG and biometric information based on infrared channel values ​​of each region in multiple regions Figure 9 This is a flowchart illustrating an example of an operational method for an electronic device (e.g., server 10, user equipment 20) that measures rPPG and biometric information based on infrared channel values ​​in each of multiple regions according to various embodiments. However, the operations may be performed in a different order than those described and / or shown, and more or fewer operations may be performed than those described and / or shown. The following will combine... Figures 10 to 11 right Figure 9 Please provide a detailed explanation.

[0085] Figure 10 This is a view illustrating examples of modules that utilize an infrared camera to perform operations for measuring rPPG and biometric information according to various embodiments. Figure 11 This is a view used to illustrate operational examples of using infrared channel values ​​for each of multiple regions according to various embodiments.

[0086] According to various embodiments, in operation 901, electronic devices (e.g., server 10, user device 20) can acquire multiple images of the user captured by an infrared camera. For example, the electronic devices (e.g., server 10, user device 20) can use the infrared camera in camera 430 to acquire infrared data of specific body parts (e.g., face) of the user within a preset time period to measure the user's rPPG. For example, user device 20 can use the infrared camera to capture multiple images of the user within a preset time period upon user request.

[0087] According to various embodiments, in operation 903, an electronic device (e.g., server 10, user equipment 20) can identify multiple regions of a face based on facial feature information extracted from multiple images. For example, referring to... Figure 10 and Figure 11 Electronic devices (e.g., server 10, user device 20) can use feature point extraction module 1010 to extract feature points representing a user's face from multiple images, and use landmark recognition module 1020 to identify the regions where the extracted feature points are defined as landmarks as multiple landmark regions L1, L2, L3, L4. For example, the multiple landmark regions L1, L2, L3, L4 may include the upper left cheek L1, the lower left cheek L3, the upper right cheek L2, and the lower right cheek L4. Landmark recognition module 1020 can be an AI model trained to recognize feature points that represent the parts (e.g., specific bones) defining the multiple landmark regions L1, L2, L3, L4, and can recognize and output feature points based on the information of the extracted feature points (e.g., coordinates, vectors) input, which respectively represent the parts (e.g., specific bones) defining the landmark regions L1, L2, L3, L4. Electronic devices (such as server 10, user equipment 20) can define the area where the output feature points are located as landmark areas L1, L2, L3, L4.

[0088] According to various embodiments, electronic devices (e.g., server 10, user equipment 20) can acquire multiple time-series data on infrared channels from multiple regions L1, L2, L3, L4 in operation 905, and acquire integrated time-series data based on at least one of the average or variance of the multiple time-series data in operation 907. For example, referring to... Figure 10 Electronic devices (e.g., server 10, user equipment 20) can use the landmark area time series data extraction module 1030 (e.g., first to fourth area extraction modules 1030a, 1030b, 1030c, 1030d) to extract the average time series signals of the infrared channels of multiple areas L1, L2, L3, and L4. The extracted average time series signals of the infrared channels of multiple areas L1, L2, L3, and L4 are preprocessed, and the preprocessed infrared channel signals of multiple areas L1, L2, L3, and L4 are combined to obtain integrated time series data.

[0089] For example, as at least part of the preprocessing operation of the average time-series signal of the infrared channels, an electronic device (e.g., server 10, user device 20) may use multiple region extraction modules 1030a, 1030b, 1030c, 1030d. For example, the electronic device (e.g., server 10, user device 20) may use multiple region extraction modules 1030a, 1030b, 1030c, 1030d to generate average time-series signals of multiple regions L1, L2, L3, L4 of the infrared channels on multiple sequentially captured image frames using a double-ended queue, and perform at least one of mean centering or Z-score on the generated average time-series signals of the multiple regions L1, L2, L3, L4 of the infrared channels to generate time-series signals of multiple regions L1, L2, L3, L4 of the infrared channels. The generated time-series signal may be defined as an rPPG signal. Subsequently, electronic devices (e.g., server 10, user equipment 20) can use preprocessing module 1040 to perform at least one of detrending based on moving average filter / moving time window or filtering based on Bézier curve on the rPPG signals of multiple regions L1, L2, L3, L4 for preprocessing.

[0090] Therefore, electronic devices (e.g., server 10, user equipment 20) can combine a portion of the pre-processed rPPG signals from the infrared channels of multiple regions L1, L2, L3, and L4 to generate integrated time-series data. For example, combining may refer to performing arithmetic operations on the pre-processed rPPG signals from the infrared channels of multiple regions L1, L2, L3, and L4 to generate integrated time-series data, and combining may be determined according to the shooting environment (e.g., illumination, shadow, jitter). For example, under the first shooting environment conditions (e.g., shadow conditions), the first result signal obtained by subtracting the rPPG of the second region L2 from the rPPG of the first region L1 can be added to the second result signal obtained by subtracting the rPPG of the fourth region L4 from the rPPG of the third region L3 to generate integrated time-series data; under the second shooting environment conditions (e.g., shadow conditions), the third result signal obtained by subtracting the rPPG of the third region L3 from the rPPG of the first region L1 can be added to the fourth result signal obtained by subtracting the rPPG of the fourth region L4 from the rPPG of the second region L2 to generate integrated time-series data. This disclosure is not limited to the above examples; the preprocessed rPPG signals on the infrared channels of multiple regions L1, L2, L3, and L4 can be combined in various ways.

[0091] According to various embodiments, in operation 909, electronic devices (e.g., server 10, user device 20) can acquire the user's biometric information based on integrated time series data. For example, electronic devices (e.g., server 10, user device 20) can use biometric information acquisition module 1050 to perform frequency analysis on the integrated time series data to estimate heart rate in the manner described in operation 515 above, which will not be repeated here.

[0092] For example, electronic devices (e.g., server 10, user equipment 20) can use the visible light camera in camera 430 to acquire RGB data of a specific body part (e.g., face) of the user within a preset time period to measure the user's rPPG. For example, user equipment 20 can, upon user request, use the visible light camera to capture multiple images of the user within a preset time period. (See reference...) Figures 6 to 7 Electronic devices (e.g., server 10, user equipment 20) can, based on the execution of program 300 (e.g., second program 30b or program 30), use the face region extraction module 610 to identify the skin region R2 within each of the multiple images (or image frames) frames #1, ..., #n captured by the user equipment 20, and use the RGB data generation module 620 to obtain the RGB color model values ​​on the skin region R2. This disclosure is not limited to the above example; in addition to the skin region R2 within the user's face region R1, skin regions can also be extracted from other body parts (e.g., thighs, forearms).

[0093] 3.3 Hybrid rPPG and Biometric Information Measurement Figure 12 This is a flowchart illustrating an example of an operation method for an electronic device (e.g., server 10, user equipment 20) that measures rPPG using a visible light camera and an infrared camera based on shooting environment conditions according to various embodiments. However, the operations may be performed in a different order than those described and / or shown, and more or fewer operations may be performed than those described and / or shown. The following will combine... Figure 13 right Figure 12 Please provide a detailed explanation.

[0094] Figure 13 This is a view used to illustrate an embodiment of measuring rPPG using a visible light camera and an infrared camera according to various embodiments.

[0095] According to various embodiments, electronic devices (e.g., server 10, user device 20) can acquire shooting environment conditions in operation 1201 and determine whether fusion conditions are met in operation 1203. For example, non-contact rPPG / biometric information measurement and services can be performed under various shooting conditions. Therefore, electronic devices (e.g., server 10, user device 20) can acquire information that affects camera shooting, such as current illumination information and jitter information, as shooting environment conditions, and determine whether fusion conditions are met based on whether the acquired information exceeds (or falls below) a preset value. For example, when the illumination value exceeds the preset value, it can be determined that the fusion conditions are met; when the illumination value is lower than the preset value, it can be determined that the fusion conditions are not met. For example, when the jitter value exceeds the preset value, it can be determined that the fusion conditions are met; when the jitter value is lower than the preset value, it can be determined that the fusion conditions are not met.

[0096] For example, refer to Figure 13 This allows for contactless rPPG / biometric information measurement and services for passengers (e.g., drivers or fellow passengers) in vehicles. (See reference...) Figure 13 (a) can perform contactless rPPG / biometric information measurement and services based on images captured by the vehicle's built-in camera; refer to Figure 13 (b) Non-contact rPPG / biometric information measurement and services can be performed based on images captured by a camera on a user device 20 (e.g., a mobile device) inside the vehicle. Therefore, electronic devices (e.g., server 10, user device 20) can acquire information affecting camera capture, such as illuminance and jitter information, obtained through sensors built into the vehicle (e.g., illuminance sensors, motion sensors) or sensors on user device 20 (e.g., illuminance sensors, motion sensors), as shooting environment condition information, and determine whether fusion conditions are met based on this information. (Refer to...) Figure 13 (b) The means of transport may be driverless autonomous vehicles (e.g., autonomous vehicles, autonomous ships).

[0097] Although not shown in the example, cameras used for non-contact rPPG / biometric information measurement and services can be installed in various locations (or spaces) equipped with cameras, in addition to vehicles, such as houses, buildings, offices, and corridors.

[0098] According to various embodiments, when the fusion condition is met (1203-Yes), the electronic device (e.g., server 10, user device 20) can acquire first time-series data based on an RGB camera in operation 1205, acquire second time-series data based on an infrared camera in operation 1207, and acquire biometric information based on the first and second time-series data in operation 1209. For example, when the fusion condition is met, the shooting environment can be identified as an environment where rPPG accuracy is reduced, therefore the electronic device (e.g., server 10, user device 20) can proceed as described above. Figure 5 The operation method generates first integrated time series data based on images captured by a visible light camera, and follows the above... Figure 9 The operation method generates second integrated time-series data based on images captured by an infrared camera. Electronic devices (e.g., server 10, user equipment 20) can generate time-series data by reflecting the first and second integrated time-series data together (e.g., summing and averaging or directly summing), and measure biometric information (e.g., heart rate) based on the generated time-series data, or measure the final biometric information by complementing the biometric information (e.g., heart rate) measured based on the first and second integrated time-series data.

[0099] According to various embodiments, when the fusion condition is not met (1203-No), the electronic device (e.g., server 10, user device 20) can acquire time-series data based on an RGB camera in operation 1211, and acquire biometric information based on the time-series data in operation 1213. For example, the electronic device (e.g., server 10, user device 20) can generate integrated time-series data based on images captured by a visible light camera, and proceed as described above. Figure 5 The operational method estimates biometric information. For example, unlike the above, electronic devices (e.g., server 10, user equipment 20) can generate integrated time-series data based on images captured by an infrared camera, and follow the above-described operational method. Figure 9 The operational method is used to estimate biometric information.

[0100] According to various embodiments, when the shooting condition information is within a first range, operation based on an RGB camera can be performed; when the shooting condition information is within a second range below the first range, operation based on an infrared camera can be performed; and when the shooting condition information is within a third range between the first and second ranges, a hybrid operation based on both an RGB camera and an infrared camera can be performed. However, this should not be considered a limitation. For example, when the illuminance is within a first range, operation based on an RGB camera can be performed; when the illuminance is within a second range below the first range, operation based on an infrared camera can be performed; and when the illuminance is within a third range between the first and second ranges, a hybrid operation can be performed.

[0101] 3.4 AI Model-Based rPPG / Biometric Information Measurement Operation 3.4.1 Blood Pressure Measurement Based on Blood Pressure AI Model Figure 14 This is a flowchart illustrating an example of an operation method for an electronic device (e.g., server 10, user device 20) that measures blood pressure based on a blood pressure AI model according to various embodiments. However, the operations may be performed in a different order than those described and / or shown, and more or fewer operations may be performed than those described and / or shown.

[0102] According to various embodiments, an electronic device (e.g., server 10, user device 20) may acquire multiple images of a user captured by an RGB camera in operation 1401, identify a specific region from the multiple images in operation 1403, and generate multiple first data associated with a first color model on the specific region of the multiple images in operation 1405. For example, the electronic device (e.g., server 10, user device 20) may acquire the RGB color model value of a skin region R2 on the multiple captured images within a preset time period, as described in operations 501 to 505.

[0103] According to various embodiments, an electronic device (e.g., server 10, user device 20) may generate multiple second data associated with a second color model based on multiple first data in operation 1407, generate first time-series data associated with the green channel based on multiple first data in operation 1409, and generate at least one second time-series data associated with the color difference channel based on multiple second data in operation 1411. For example, the electronic device (e.g., server 10, user device 20) may perform the operation of generating time-series data (rPPG) of each color model in multiple color models (e.g., RGB, YCgCr, CIE La*b*) according to the above operations 1407 to 1411.

[0104] According to various embodiments, an electronic device (e.g., server 10, user device 20) can, in operation 1413, acquire blood pressure based on inputting first time-series data and second time-series data into a blood pressure measurement AI model. For example, the blood pressure measurement AI model can be an AI model trained to output blood pressure values ​​based on information from at least a portion of the time-series data in multiple input color models (e.g., RGB, YCgCr, CIE La*b*). Training can be performed based on various learning algorithms such as supervised learning, unsupervised learning, and machine learning; therefore, detailed descriptions are omitted.

[0105] In one embodiment, the blood pressure AI model can be an AI model trained to take at least a portion of time series data from multiple color models (e.g., RGB, YCgCr, CIE La*b*) within a specific time period as input data and blood pressure measured at a specific time as output data.

[0106] In one embodiment, the blood pressure measurement AI model can be an AI model trained to take at least a portion of the frequency feature values ​​of multiple color models (e.g., RGB, YCrCb, CIE La*b*) within a specific time period as input data, and take the blood pressure value measured at that specific time as output data. The frequency feature values ​​may include information about the interpeak distance.

[0107] 3.4.2 Pressure Measurement Based on Pressure Analysis AI Model Figure 15 This is a flowchart illustrating an example of an operation method for an electronic device (e.g., server 10, user equipment 20) that measures pressure based on a pressure analysis AI model according to various embodiments. However, operations may be performed in a different order than those described and / or shown, and more or fewer operations may be performed than those described and / or shown. The following will combine... Figure 16 right Figure 15 Please provide a detailed explanation.

[0108] Figure 16 This is a view used to illustrate examples of modules performing stress analysis operations according to various embodiments.

[0109] According to various embodiments, electronic devices (e.g., server 10, user equipment 20) can acquire multiple images of a user captured by an RGB camera in operation 1501, and generate an image based on the multiple images and a specific region (e.g., ...) in operation 1503. Figure 5 The first time-series signal (e.g., RGB color model value) associated with the RGB color model on the skin region R2). Operations 1501 to 1503 of the electronic device (e.g., server 10, user device 20) can be performed based on the RGB data generation module 1610, in accordance with the operations 501 to 503 of the electronic device (e.g., server 10, user device 20) described above, and will not be repeated here.

[0110] According to various embodiments, an electronic device (e.g., server 10, user equipment 20) may generate a second time-series signal associated with a CIE La*b* color model over a specific region based on multiple images in operation 1505, and correct a first time-series signal based on the second time-series signal in operation 1507. For example, refer to Figure 16Electronic devices (e.g., server 10, user equipment 20) can use correction module 1620 to interpolate the first time-series signal associated with the RGB color model on the skin region R2, and use the second time-series signal on the L channel measured from multiple images converted to the CIE La*b* color model to correct (or preprocess) the brightness of the first time-series signal.

[0111] According to various embodiments, electronic devices (e.g., server 10, user device 20) can generate a third time-series signal associated with the YCrCgCb color model based on a corrected first time-series signal in operation 1509. The electronic devices (e.g., server 10, user device 20) can use a YCrCgCb time-series signal generation module 1630 to generate a YCrCgCb time-series signal based on the first time-series signal on a luminance-preprocessed RGB model, and use a preprocessing module 1640 to preprocess the YCrCgCb time-series signal. Preprocessing operations may include at least one of the following: noise removal based on a moving average filter, detrending based on a sliding window, Butterworth bandpass filtering, amplitude correction, or extracting (or truncating) the signal for a preset time (e.g., 8 seconds) based on a downward peak value after detecting the signal peak value of each channel (Y, Cr, Cg, Cb).

[0112] According to various embodiments, electronic devices (e.g., server 10, user device 20) can acquire a user's stress index based on a third time-series signal and additional information during operation 1511. For example, electronic devices (e.g., server 10, user device 20) can input blood pressure feature values ​​(e.g., SDNN, SDSD, RMSSD) extracted based on a preprocessed third time-series signal (YCrCgCb signal) into a heart rate variability (HRV) analysis AI model 1650. Electronic devices (e.g., server 10, user device 20) can input rPPG obtained based on an rPPG extraction module and biometric information (e.g., heart rate, oxygen saturation) measured by rPPG based on a biometric information acquisition module 1680, along with the acquired blood pressure feature values, into a stress index analysis algorithm (formula) to obtain a stress index. The measured rPPG and biometric information can be processed as described above. Figure 5 , Figure 9 and Figure 12 The measurement process described above will not be repeated here. Electronic devices (such as server 10 and user equipment 20) can post-process the acquired stress index based on a post-processing algorithm and provide the stress index to the user.

[0113] 4. Mobility services Figure 17A view for illustrating an example of a vehicle V that provides services based on contactless biometric information according to various embodiments.

[0114] According to various embodiments, electronic devices (e.g., server 10, user equipment 20) may be implemented to measure rPPG and biometric information based on the analysis of images taken of passengers (e.g., drivers or fellow passengers) riding in a vehicle V, and to provide predetermined services according to the rPPG and biometric information measurement methods described in "Part 3" above.

[0115] According to various embodiments, refer to Figure 17 The vehicle V may include communication circuitry 1710, processor 1720, camera 1730, and configuration device 1740, but is not limited to the example shown, and may be implemented to include more devices. The camera 1730 may not be located within the vehicle V, and may provide services based on images captured by the camera 430 of the user equipment 20 held within the vehicle V.

[0116] According to various embodiments, electronic devices (e.g., server 10, user equipment 20) can receive a variety of information from the communication circuitry 1710 of the vehicle V. For example, the various information may include images of passengers (e.g., drivers or fellow passengers) riding in the vehicle V captured by camera 1730, as well as information about the environmental conditions at which the images were taken.

[0117] According to various embodiments, electronic devices (e.g., server 10, user equipment 20) can send control signals to the vehicle V to enable the processor 1720 of the vehicle V to control the configuration device 1740 based on rPPG and biometric information as a result analysis. The processor 1720 of the vehicle V can control the configuration device 1740 based on the received control signals. For example, the configuration device 1740 may include not only devices that directly affect driving, such as steering devices 1740a and engine 1740b, but also devices that do not directly affect driving but provide convenience, such as seats 1740c and lights (not shown).

[0118] 4.1 Providing mobile travel services based on contactless biometric information and external information Figure 18 This is a flowchart illustrating an example of an operation method for an electronic device (e.g., server 10, user equipment 20) providing mobile travel services based on contactless biometric information and external information according to various embodiments. However, the operations may be performed in a different order than those described and / or shown, and more or fewer operations may be performed than those described and / or shown. The following will combine... Figure 19 right Figure 18 Please provide a detailed explanation.

[0119] Figure 19 This is a view used to illustrate examples of mobile travel services provided based on contactless biometric information and external information according to various embodiments.

[0120] According to various embodiments, electronic devices (e.g., server 10, user device 20) can identify a vehicle start-up event in operation 1801, capture multiple images of the user via a camera in operation 1803, and acquire biometric information based on the captured multiple images in operation 1805. For example, electronic devices (e.g., server 10, user device 20) can use a camera 1730 installed in a vehicle (e.g., a vehicle) or a camera 430 installed in the user device 20 held in the vehicle (e.g., a vehicle) to acquire multiple images and acquire biometric information. Cameras 1730 and 430 may include at least one of a visible light camera or an infrared camera. The operations of measuring rPPG and biometric information can be performed as described in "Part 3" above, and will not be repeated here.

[0121] According to various embodiments, electronic devices (e.g., server 10, user device 20) can acquire external environmental information of the vehicle in operation 1807, and control in-vehicle devices based on biometric information and external environmental information in operation 1809. For example, electronic devices (e.g., server 10, user device 20) can acquire external environmental information related to the location of the currently activated vehicle. External environmental information may include climate information such as temperature, wind direction, wind speed, and weather at the vehicle's location, as well as various other information that can be collected from an external server. Thus, electronic devices (e.g., server 10, user device 20) can provide control signals to the vehicle for controlling vehicle configuration devices 1740 based on biometric information measured non-contactly and the collected external environmental information. For example, refer to... Figure 19 Electronic devices (e.g., server 10, user equipment 20) can provide control signals for controlling different types of configured devices based on blood pressure measured non-contactly and external temperature measured as external environmental information. (Refer to...) Figure 19 In (a) and (b), when the blood pressure value is a specific value (e.g., 130), electronic devices (e.g., server 10, user device 20) can determine that an event has occurred for controlling the configuration device 1740 of the vehicle. When the event occurs, the electronic devices (e.g., server 10, user device 20) can acquire external temperature information as external environmental information, and as... Figure 19 As shown in (a), when the external temperature is higher than a preset value, electronic devices (e.g., server 10, user equipment 20) can generate a control signal to activate the air conditioner in configuration device 1740; Figure 19As shown in (b), when the external temperature is lower than a preset value, electronic devices (e.g., server 10, user equipment 20) can generate a control signal for opening the window in the configuration device 1740 and provide the control signal to the vehicle's processor 1720. The vehicle's processor 1720 can control the vehicle's configuration device 1740 based on the control signal. This disclosure is not limited to the examples described and / or shown; various operations can be performed to control the vehicle's configuration device 1740, and operations to control other configuration devices 1740 can be performed in a similar manner, detailed description omitted.

[0122] According to various embodiments, as part of the operation of generating control signals described above, electronic devices (e.g., server 10, user equipment 20) may use at least one AI model that stores information about the type of the configuration device 1740 to be controlled and the control method based on biometric information, as well as information about the external environment, in the form of a lookup table, or pre-learns information about the type of the configuration device 1740 to be controlled and the control method based on biometric information, as well as information about the external environment.

[0123] 4.1.1 Provide mobility services that further consider passenger information Figure 20 This is a flowchart illustrating an example of an operation method for an electronic device (e.g., server 10, user equipment 20) that provides a mobility service that further considers passenger information according to various embodiments. However, operations may be performed in a different order than those described and / or shown, and more or fewer operations may be performed than those described and / or shown. The following will combine... Figure 21 right Figure 20 Please provide a detailed explanation.

[0124] Figure 21 A view used to illustrate examples of measuring rPPG / biometric information of passengers according to various embodiments.

[0125] According to various embodiments, electronic devices (such as server 10, user equipment 20) may provide mobility services that further take into account the biometric information of passengers in the vehicle when providing mobility services as described in “Part 4.1” above.

[0126] According to various embodiments, electronic devices (e.g., server 10, user equipment 20) may use a camera to capture multiple images of multiple passengers in operation 2001, acquire biometric information of multiple passengers based on the captured images in operation 2003, and ultimately provide mobility services by performing operations 1807 to 1809. For example, refer to Figure 21In addition to the driver, passengers can also ride in the vehicle, and in the case of an autonomous vehicle, multiple passengers can ride. In this case, if only the driver's biometric information is considered when providing mobility services, it may cause inconvenience to other passengers. Therefore, electronic devices (such as server 10 and user equipment 20) can further consider the biometric information of other passengers when providing mobility services.

[0127] For example, electronic devices (e.g., server 10, user equipment 20) can use a camera 1730 installed in a vehicle (e.g., a vehicle) or a camera 430 installed in a user equipment 20 held within a vehicle (e.g., a vehicle) to acquire multiple images of multiple passengers within a preset time period. In one embodiment, camera 1730 may be designed to have a field of view (FOV) capable of capturing all passengers, or may be positioned at a location capable of capturing all passengers (e.g., a vehicle interior mirror, roof). Alternatively, multiple cameras 1730 may be positioned near the seats of multiple passengers. In one embodiment, camera 430 may be designed to have an FOV capable of capturing all passengers, or the user equipment 20 held by multiple passengers may be placed near their respective seats, thereby enabling the capture of passenger images.

[0128] Therefore, electronic devices (e.g., server 10, user equipment 20) can generate different control signals by further considering biometric images based on passenger image measurements. For example, when the biometric information of multiple passengers is within a similar range, the electronic devices (e.g., server 10, user equipment 20) can directly generate a specific control signal for configuring device 1740. For example, as... Figure 21 As shown, when the biometric information of multiple passengers is not in a similar range (i.e. there are differences), electronic devices (such as server 10 and user equipment 20) can generate control signals for opening the windows. Specifically, control signals can be generated to open the windows near the driver with high blood pressure only, while keeping the windows near the passengers with normal blood pressure unchanged.

[0129] 4.1.2 Provide mobility services that further consider vehicle modes Figure 22 This is a flowchart illustrating an example of an operation method for an electronic device (e.g., server 10, user equipment 20) that provides mobility services further considering vehicle modes according to various embodiments. However, operations may be performed in a different order than those described and / or shown, and more or fewer operations may be performed than those described and / or shown. The following will combine... Figure 23 right Figure 22 Please provide a detailed explanation.

[0130] Figure 23 This is a view used to illustrate examples of mobility services with continuously deformable seat configurations during autonomous driving, according to various embodiments.

[0131] According to various embodiments, electronic devices (e.g., server 10, user equipment 20) can identify the vehicle's driving mode as autonomous driving mode in operation 2201, and control the configuration of the vehicle seats based on biometric information in operation 2203. For example, the vehicle can be set to autonomous driving mode under the driver's control after startup. Figure 23 As shown, electronic devices (e.g., server 10, user device 20) can continuously (or periodically) measure the user's biometric information (e.g., blood pressure) based on captured images in autonomous driving mode, and control the configuration of the vehicle seat based on the biometric information measured during autonomous driving mode, changing it to a configuration that is comfortable for the user, thereby further improving user convenience. For example, as Figure 23 As shown, when a non-contact blood pressure value is detected to exceed a preset value E, an electronic device (e.g., server 10, user device 20) can generate a control signal to adjust the seat angle from the current angle to another angle and provide the control signal to the vehicle's processor 1820.

[0132] According to various embodiments, electronic devices (e.g., server 10, user device 20) can control other devices in the vehicle based on biometric information and external environment information in operation 2205. Operation 2205 of the electronic devices (e.g., server 10, user device 20) can be performed in the same manner as operation 1908 of the electronic devices (e.g., server 10, user device 20) described above, and repeated descriptions are omitted here.

[0133] 5. Provide services based on continuous accumulation of biometric information Figure 24 This is a flowchart illustrating an example of an operation method for an electronic device (e.g., server 10, user equipment 20) providing services based on the continuous accumulation of biometric information according to various embodiments. However, operations may be performed in a different order than those described and / or shown, and more or fewer operations may be performed than those described and / or shown. The following will combine... Figure 25A and Figure 25B right Figure 24 Please provide a detailed explanation.

[0134] Figure 25A This is a view used to illustrate an embodiment of accumulating biometric information measured for each travel path within a preset time in a database 2500, according to various embodiments. Figure 25B A view for illustrating service examples that provide driving recommendation information based on accumulated biometric information according to various embodiments.

[0135] According to various embodiments, electronic devices (e.g., server 10, user device 20) can identify specific types of life events in operation 2401 and capture multiple images of the user using a camera during the duration of the specific type of life event. For example, a specific type of life event may refer to various activities that the user can perform, such as driving, exercising, studying, working, etc. In this case, the electronic devices (e.g., server 10, user device 20) can receive input from the user by providing a graphical user interface for receiving information input indicating what kind of life event has occurred, and can predict the type of life event the user is currently performing based on pattern information from sensor values ​​collected using various types of sensors (e.g., motion sensors, angular velocity sensors, illuminance sensors, etc.). When a specific type of life event occurs, the electronic devices (e.g., server 10, user device 20) can periodically capture images of the user at multiple times based on the user device 20's camera 430 or a camera positioned in the space where the specific type of life event occurs (e.g., a camera 1730 inside a vehicle), and measure the user's rPPG and biometric information at multiple times based on the captured images, as described in "Part 3" above. For example, as Figure 25A As shown, when a user travels along a specific route each day for multiple days within a preset period, biometric information (e.g., blood pressure, stress) acquired during the journey can be stored based on images captured at different times by a camera 1730 installed in the vehicle or a camera 430 on a user device held in the vehicle. The stored biometric information may include at least one of biometric information for each time period of the journey or average biometric information. This disclosure is not limited to the examples described and / or shown, and it will be apparent to those skilled in the art that it can also be applied to other types of life events (e.g., exercise, study, work, etc.) and / or multiple types of biometric information (e.g., heart rate, blood oxygen saturation, etc.).

[0136] According to various embodiments, electronic devices (e.g., server 10, user equipment 20) may, in operation 2405, store biometric information in a form associated with attribute information related to life events based on multiple captured images. For example, as Figure 25A As shown, electronic devices (such as server 10 and user device 20) can accumulate blood pressure information according to different driving routes under the life event category called "User Driving". That is, attribute information can refer to elements that define the same life event category. In the case of driving, attribute information can be information about the driving route (such as starting point, destination, path) and driving time. In the case of exercise, information such as exercise type and exercise time can be attribute information of the exercise life event.

[0137] According to various embodiments, electronic devices (e.g., server 10, user device 20) can provide services based on stored information during operation 2407. For example, electronic devices (e.g., server 10, user device 20) can provide accumulated information on various lifestyle categories of a user stored over a preset period to another external server, or provide services based on preset accumulated information itself. The accumulated information can be personalized information specific to the user and can be used as an objective basis for providing optimized services to the user based on lifestyle categories. For example, when recommending a route from a first location to a second location, such as... Figure 25B As shown, electronic devices (e.g., server 10, user device 20) can provide information about the stress index of each driving route from the first location to the second location based on biometric information (e.g., stress index) accumulated in database 2500 for each driving route from the first location to the second location. Therefore, users can select a driving route based on driving time and distance, or they can select a driving route considering the stress index (or changes in biometric information) that the user may experience.

[0138] According to various embodiments, electronic devices (e.g., server 10, user device 20) can determine whether to provide services based on stored information based on the user's current biometric information. For example, when the value of rPPG and / or biometric information measured non-contactly meets preset conditions, the electronic device (e.g., server 10, user device 20) can initiate the operation of providing services based on stored information. For example, when at least one of blood pressure or stress index is greater than or equal to a preset value, the reference demand for biometric information during the user's life may increase, so the electronic device (e.g., server 10, user device 20) can perform the operation of providing services based on biometric information stored according to various lifestyle categories. In other words, when the reference demand for biometric information during the user's life is low, conventional services can be performed.

[0139] 6. Self-service terminal service Figure 26 A view illustrating an example of a self-service terminal 2600 that provides services based on contactless biometric information according to various embodiments.

[0140] According to various embodiments, electronic devices (e.g., server 10, user equipment 20) can be implemented to measure rPPG and biometric information based on the analysis of user images captured by self-service terminals 2600 placed in various locations, in the same manner as the rPPG and biometric measurement methods described in "Part 3" above, and to provide reservation services. Self-service terminals 2600 can be placed in locations where staff move around, or in various locations such as doorways and passageways.

[0141] According to various embodiments, refer to Figure 26 The self-service terminal 2600 may include a communication circuit 2610, a processor 2620, a camera 2630, and a display 2640, and is not limited to the example shown, and may be implemented to include more devices.

[0142] According to various embodiments, electronic devices (such as server 10, user equipment 20) can receive various types of information from the communication circuitry 2610 of the self-service terminal 2600. For example, the various types of information may include images of staff taken by camera 2630 and information about the shooting environment.

[0143] According to various embodiments, electronic devices (e.g., server 10, user equipment 20) can send control signals based on the analyzed rPPG signals and biometric information, causing the processor 2620 of the self-service terminal 2600 to output predetermined information (e.g., construction site related information I) through the display 2640. In this case, the construction site related information I displayed on the display 2640 of the self-service terminal 2600 may include not only information received from electronic devices (e.g., server 10, user equipment 20), but also information received from other external servers (e.g., weather information).

[0144] 6.1 Provide services based on biometric information from multiple locations. Figure 27 This is a flowchart illustrating an example of an operation method for an electronic device (e.g., server 10, user equipment 20) that provides services based on biometric information from multiple locations according to various embodiments. However, operations may be performed in a different order than those described and / or shown, and more or fewer operations may be performed than those described and / or shown. The following will combine... Figures 28 to 29 right Figure 27 Please provide a detailed explanation.

[0145] Figure 28 This is a view used to illustrate examples of collecting biometric information of a specific user from self-service terminals placed in each of multiple locations, according to various embodiments. Figure 29 This is a view used to illustrate service examples based on biometric information from multiple locations according to various embodiments.

[0146] According to various embodiments, an electronic device (e.g., server 10, user equipment 20) may acquire identification information of a specific user in operation 2701, identify multiple locations of the specific user in operation 2703, and store biometric information of the specific user based on multiple images of the specific user taken using self-service terminals placed at multiple locations in operation 2705. For example, refer to Figure 29The system can acquire biometric information of specific users (or staff) based on self-service terminals 2600 set up at multiple construction sites. For example, when a specific user enters the first construction site at a specific time, the first self-service terminal 2600a can provide the captured image of the specific user along with reference information (e.g., time, area) to electronic devices (e.g., server 10, user device 20). For example, when a specific user enters another construction site (e.g., the second construction site, the third construction site) at a later time, the electronic devices (e.g., server 10, user device 20) can receive the image of the specific user and reference information (e.g., time, area) captured by self-service terminals 2600b, 2600c placed at the other construction site (e.g., the second construction site, the third construction site). The electronic devices (e.g., server 10, user device 20) can identify the specific user based on the facial image contained in the captured image and a pre-registered facial image, and store the biometric information measured based on the captured image based on the reference information and the location of the identified specific user. The operation of measuring biometric information based on the captured image can be performed in the same manner as described in "Part 3" above, and repeated descriptions are omitted here.

[0147] According to various embodiments, electronic devices (e.g., server 10, user equipment 20) can provide services based on biometric information from multiple locations during operation 2707. For example, such as Figure 29 As shown in (a), electronic devices (e.g., server 10, user device 20) can sequentially provide trend information on biometric information (e.g., blood pressure) at various construction sites where a specific user is located via self-service terminal 2600. Based on this trend information, the electronic devices (e.g., server 10, user device 20) can estimate the increase in biometric information at other construction sites where the specific user will next work. When the estimated biometric information exceeds a preset value, the electronic devices (e.g., server 10, user device 20) can prevent the specific user from entering other construction sites, thereby preventing accidents. For example, as... Figure 29 As shown in (b), electronic devices (e.g., server 10, user device 20) can select suitable locations (e.g., workplaces) for users based on biometric information that grows with location and provide relevant information, thereby assigning users to workplaces where they can work more smoothly. For example, the workplace where the user has the lowest blood pressure can be identified as the most suitable workplace for the user. When there are multiple workers and the user's blood pressure is lower than the average blood pressure of the multiple workers, the appropriate workplace can be determined.

Claims

1. A method for operating an electronic device that controls equipment inside a vehicle, the method comprising: Acquire multiple images of at least one passenger inside the vehicle captured by a camera; Based on the multiple images, obtain the biometric information of at least one passenger; Obtain the external environment information of the vehicle; as well as The device inside the vehicle is controlled based on the biometric information and the external environmental information.

2. The operating method according to claim 1, wherein, The biometric information includes at least one of the following: heart rate, blood oxygen saturation, blood pressure, and stress index of the at least one passenger.

3. The operating method according to claim 1, wherein, The external environmental information includes at least one of the following: temperature, wind direction, wind speed, and weather conditions corresponding to the vehicle's location.

4. The operating method according to claim 1, wherein, The equipment within the vehicle includes at least one of the following: steering equipment, engine, seats, lighting equipment, air conditioning and heating system, speakers, windows, and doors.

5. The operating method according to claim 1, wherein, When controlling the devices inside the vehicle based on the biometric information and the external environment information, the controlled devices inside the vehicle change according to the values ​​corresponding to the biometric information and the values ​​corresponding to the external environment information.

6. The operating method according to claim 1, wherein, The at least one passenger includes the driver and fellow passengers, and the operating method further includes: The camera is used to acquire multiple first images of the driver and multiple second images of the passenger. The driver's first biometric information is obtained based on the plurality of first images; Based on the multiple second images, the second biometric information of the passenger is obtained; and The device inside the vehicle is controlled based on the first biometric information and the second biometric information.

7. The operating method according to claim 6, wherein, When controlling the device inside the vehicle based on the biometric information and the external environment information, the controlled device inside the vehicle changes according to the value corresponding to the first biometric information and the value corresponding to the second biometric information.

8. The operating method according to claim 1, wherein, The operation method further includes: The vehicle identification mode is set to autonomous driving mode; and The seat angle of at least one passenger is changed based on the biometric information.

9. The operating method according to claim 1, wherein, The operation method further includes: During the vehicle's operation, the biometric information is acquired at multiple time periods based on the multiple images; The biometric information and the vehicle's travel route are stored in an associated format; and The service is provided based on the stored biometric information and the travel route.

10. The operating method according to claim 1, wherein, The acquisition of the biometric information includes: Identify shooting environment conditions; When the identified shooting environment conditions do not meet the predetermined data fusion conditions, the predetermined body area of ​​the at least one passenger is obtained from each of the plurality of images; Generate multiple first data sets associated with the RGB color model of the body region from each of the multiple images; Based on the aforementioned first data, generate multiple second data associated with the YCrCb color model; First time-series data associated with the green channel is generated based on the multiple first data; Generate second time-series data associated with the color difference channel based on the plurality of second data; First integrated time series data is generated by combining the first time series data and the second time series data; The heart rate of the at least one passenger is estimated by converting the first integrated time series data into the frequency domain; and The biometric information is obtained based on the estimated heart rate.

11. The operating method according to claim 10, wherein, The acquisition of the biometric information includes: When the identified shooting environment conditions meet the predetermined data fusion conditions, multiple images of the at least one passenger are acquired using an infrared camera; Obtain multiple body regions of the at least one passenger from the multiple images; Extract the time-series data for each of the multiple body regions; A second integrated time series data is generated based on the time series data of each of the plurality of body regions; Generate final time series data based on the first integrated time series data and the second integrated time series data; and The biometric information is obtained based on the final time series data.

12. An electronic device for controlling equipment inside a vehicle, the electronic device comprising: At least one processor; The at least one processor is configured to: Acquire multiple images of at least one passenger inside the vehicle captured by a camera; Based on the multiple images, obtain the biometric information of at least one passenger; Obtain the external environment information of the vehicle; as well as The device inside the vehicle is controlled based on the biometric information and the external environmental information.

13. The electronic device according to claim 12, wherein, The at least one processor is further configured to: The controlled equipment inside the vehicle is modified based on the values ​​corresponding to the biometric information and the values ​​corresponding to the external environment information.

14. The electronic device according to claim 12, wherein, The at least one passenger includes the driver and a fellow passenger, and the at least one processor is further configured to: The camera is used to acquire multiple first images of the driver and multiple second images of the passenger. The driver's first biometric information is obtained based on the plurality of first images; The second biometric information of the passengers is obtained based on the multiple second images; as well as The device inside the vehicle is controlled based on the first biometric information and the second biometric information.

15. The electronic device according to claim 14, wherein, The at least one processor is further configured to: The equipment inside the controlled vehicle is modified based on the values ​​corresponding to the first biometric information and the second biometric information.

16. The electronic device according to claim 12, wherein, The at least one processor is further configured to: The vehicle identification mode is set to autonomous driving mode; and The seat angle of at least one passenger is changed based on the biometric information.

17. The electronic device according to claim 12, wherein, The at least one processor is further configured to: During the vehicle's operation, the biometric information is acquired at multiple time periods based on the multiple images; The biometric information and the vehicle's driving route are stored in an associated format; as well as The service is provided based on the stored biometric information and the travel route.

18. The electronic device according to claim 12, wherein, The at least one processor is further configured to: Identify shooting environment conditions; When the identified shooting environment conditions do not meet the predetermined data fusion conditions, the predetermined body area of ​​the at least one passenger is obtained from each of the plurality of images; Generate multiple first data sets associated with the RGB color model of the body region from each of the multiple images; Based on the aforementioned first data, generate multiple second data associated with the YCrCb color model; First time-series data associated with the green channel is generated based on the multiple first data; Generate second time-series data associated with the color difference channel based on the plurality of second data; First integrated time series data is generated by combining the first time series data and the second time series data; The heart rate of the at least one passenger was estimated by converting the first integrated time series data into the frequency domain. as well as The biometric information is obtained based on the estimated heart rate.

19. The electronic device according to claim 18, wherein, The at least one processor is further configured to: When the identified shooting environment conditions meet the predetermined data fusion conditions, multiple images of the at least one passenger are acquired using an infrared camera; Obtain multiple body regions of the at least one passenger from the multiple images; Extract the time-series data for each of the multiple body regions; A second integrated time series data is generated based on the time series data of each of the plurality of body regions; The final time series data is generated based on the first integrated time series data and the second integrated time series data; as well as The biometric information is obtained based on the final time series data.

Images