Vehicle electronic device for providing guide for protecting driver by detecting respiration of driver and operating method thereof

Abstract

Provided are an electronic device for monitoring respiration of a vehicle driver and providing a respiration guide for driver protection when abnormal respiration is detected, and an operating method thereof. The electronic device may obtain a respiration signal of a driver by detecting the movement of at least one of the chest and abdomen by the respiration of the driver from an image obtained through a camera, detect abnormal respiration including at least one of hyperventilation, hypoventilation, transient apnea, and dyspnea by monitoring a respiration signal in real time, and display a respiration guide user interface (UI) which provides a respiration exercise guide to realign respiration of the driver and relieve abnormal respiration on the basis of the detection of abnormal respiration.

Description

A vehicle electronic device that detects the driver's breathing and provides guidance for driver protection, and an operating method thereof

[0001] The present disclosure relates to a vehicle electronic device that detects the respiration of a vehicle driver and, if abnormal respiration is detected, provides breathing guidance information for driver protection, and a method of operating the same. Specifically, the present disclosure discloses an electronic device that monitors the respiration signal of a driver (or passenger) through an occupant monitoring system (OMS) including a camera positioned within a vehicle, and provides breathing exercise guidance information to the driver, and a method of operating the same.

[0002] An occupant monitoring system (OMS) uses a camera to capture images of not only the driver but also passengers, and recognizes the driver's face from the acquired images to perform actions such as drowsy driving warnings and driver status warnings, or is used in intelligent speed assist devices, reverse assist devices, event data recorders (EDR), or emergency braking devices.

[0003] Recently, legislation is being implemented worldwide to mandate the installation of occupant monitoring systems in vehicles for driver protection and the prevention of road traffic accidents. In particular, in the European Union, the use of occupant monitoring systems has significantly reduced the number of traffic fatalities. Consequently, the European Commission has set a goal of halving the number of traffic fatalities by 2030 and is actively exploring ways to utilize occupant monitoring systems. Similar legislation is also planned in the United States, China, Australia, and other countries, and Korea is also considering legislation to mandate the installation of occupant monitoring systems in vehicles.

[0004] Meanwhile, automakers are attempting to integrate driver state understanding technology into their vehicles, not only to detect drowsy or inattentive driving, but also to advance autonomous driving and safety technologies. As part of this driver state understanding technology, occupant monitoring systems are needed to monitor the driver's (or passenger's) breathing and detect abnormal breathing patterns outside the normal range. This technology can protect the driver (or passenger) from abnormal breathing and prevent traffic accidents.

[0005] One aspect of the present disclosure provides a method for an electronic device mounted in a vehicle to detect the respiration of a driver of the vehicle and provide a breathing guide for driver protection. The method of operating the electronic device according to one embodiment of the present disclosure may include a step of detecting movement of at least one of the chest and abdomen caused by the driver's respiration from an image acquired through a camera to acquire a breathing signal of the driver. The method of operating the electronic device according to one embodiment of the present disclosure may include a step of monitoring the acquired breathing signal in real time to detect abnormal respiration including at least one of hyperventilation, hypopnea, transient apnea, and dyspnea. The method of operating the electronic device according to one embodiment of the present disclosure may include a step of displaying a breathing guide user interface (UI) that provides a breathing exercise guide to alleviate the abnormal respiration by reorganizing the driver's respiration when abnormal respiration is detected.

[0006] One aspect of the present disclosure provides an in-vehicle electronic device that detects a driver's breathing and provides guidance for driver protection. The electronic device according to one embodiment of the present disclosure may include a display unit, a camera that captures an image of the driver, a memory that stores at least one instruction, and at least one processor that executes at least one instruction. The at least one processor may detect movement of at least one of the chest and abdomen caused by the driver's breathing from an image acquired through the camera to obtain a breathing signal of the driver. The at least one processor may monitor the breathing signal in real time to detect abnormal breathing, including at least one of hyperventilation, hypopnea, transient apnea, and dyspnea. When abnormal breathing is detected, the at least one processor may display a breathing guide UI (user interface) on the display unit that provides a breathing exercise guide to alleviate the abnormal breathing by reorganizing the driver's breathing.

[0007] One aspect of the present disclosure provides a computer program product including a computer-readable storage medium. The storage medium may include instructions readable by an electronic device, which cause an electronic device to perform the following operations: detecting movement of at least one of the chest and abdomen caused by the driver's breathing from an image acquired through a camera to acquire a driver's breathing signal; monitoring the acquired breathing signal in real time to detect abnormal breathing including at least one of hyperventilation, hypopnea, transient apnea, and dyspnea; and displaying a breathing guide UI (user interface) that provides a breathing exercise guide to alleviate the abnormal breathing by reorganizing the driver's breathing when abnormal breathing is detected.

[0008] The present disclosure can be readily understood by the following detailed description and its accompanying drawings, wherein reference numerals refer to structural elements.

[0009] FIG. 1 is a diagram illustrating an operation of an electronic device according to one embodiment of the present disclosure to detect a driver's breathing and provide a breathing guide UI (user interface).

[0010] FIG. 2 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to detect a driver's breathing and provide a breathing guide UI (user interface).

[0011] FIG. 3 is a block diagram illustrating components of an electronic device according to one embodiment of the present disclosure.

[0012] FIG. 4 is a flowchart illustrating a method for an electronic device to obtain a driver's breathing signal according to one embodiment of the present disclosure.

[0013] FIG. 5 is a diagram illustrating an operation of an electronic device according to one embodiment of the present disclosure to obtain a breathing signal from at least one region of the chest and abdomen identified based on the driver's shoulder feature points.

[0014] FIG. 6 is a flowchart illustrating a method for an electronic device to obtain a driver's breathing signal according to one embodiment of the present disclosure.

[0015] FIG. 7 is a diagram illustrating an operation of an electronic device according to one embodiment of the present disclosure to obtain a breathing signal from at least one region of the chest and abdomen identified based on the driver's shoulder feature points and pelvic feature points.

[0016] FIG. 8 is a flowchart illustrating a method for an electronic device to obtain a driver's breathing signal according to an embodiment of the present disclosure.

[0017] FIG. 9 is a diagram for explaining an operation of an electronic device according to one embodiment of the present disclosure to acquire a breathing signal from a valid area excluding a seat belt among the entire area of ​​an image captured of a driver.

[0018] FIG. 10 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to obtain a breathing signal by selecting a specific region among a plurality of regions included in a driver's body part.

[0019] FIG. 11 is a diagram illustrating an operation of an electronic device according to one embodiment of the present disclosure to select a specific area among a plurality of areas included in a driver's body part and acquire a breathing signal.

[0020] FIG. 12 is a flowchart illustrating a method for an electronic device according to one embodiment of the present disclosure to correct a breathing signal based on motion information resulting from vibration of a vehicle or body movement of a driver.

[0021] FIG. 13 is a graph illustrating an operation of an electronic device according to one embodiment of the present disclosure to correct a breathing signal based on motion information resulting from vibration of a vehicle or body movement of a driver.

[0022] FIG. 14 is a flowchart illustrating a method for an electronic device to detect abnormal respiration based on rest respiration according to one embodiment of the present disclosure.

[0023] FIG. 15A is a diagram illustrating an operation of an electronic device displaying a breathing guide UI according to one embodiment of the present disclosure.

[0024] FIG. 15b is a diagram illustrating an operation of an electronic device displaying a breathing guide UI according to one embodiment of the present disclosure.

[0025] FIG. 16 is a diagram illustrating an operation of an electronic device according to one embodiment of the present disclosure to output a warning message upon detecting abnormal breathing.

[0026] FIG. 17 is a diagram illustrating an operation of an electronic device according to one embodiment of the present disclosure to display a breathing waveform representing a driver's thoracic breathing and abdominal breathing and a breathing guide UI together.

[0027] FIG. 18 is a diagram illustrating an operation of an electronic device according to one embodiment of the present disclosure to display the amount of change in each of a driver's thoracic breathing and abdominal breathing.

[0028] FIG. 19 is a flowchart illustrating a method for an electronic device to protect a driver when hyperventilation is detected according to one embodiment of the present disclosure.

[0029] FIG. 20 is a drawing for explaining an operation of an electronic device according to one embodiment of the present disclosure to protect a driver when hyperventilation is detected.

[0030] FIG. 21 is a block diagram illustrating an operation of an electronic device according to one embodiment of the present disclosure to control devices included in a vehicle when abnormal breathing of a driver is detected.

[0031] FIG. 22 is a drawing for explaining an operation performed by an electronic device of the present disclosure using artificial intelligence technology.

[0032] FIG. 23 is a diagram illustrating an embodiment of the present disclosure in which an electronic device operates in conjunction with a server.

[0033] Figure 24 is a drawing for explaining Figure 23 in detail.

[0034] The terms used in the embodiments of this specification have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0035] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0036] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0037] As used herein, the expression "configured to" can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in terms of hardware. Instead, in some contexts, the expression "a system configured to" can mean that the system is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured to perform A, B, and C" can mean a dedicated processor for performing the operations (e.g., an embedded processor), or a general-purpose processor (e.g., a CPU or an application processor) that can perform the operations by executing one or more software programs stored in memory.

[0038] Additionally, when a component is referred to as being "connected" or "connected" to another component in the present disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0039] In the present disclosure, a "vehicle" refers to a means of transportation that runs on roads or tracks. A "vehicle" may include an internal combustion engine vehicle equipped with an engine as a power source, a hybrid vehicle equipped with an engine and an electric motor as a power source, an electric vehicle equipped with an electric motor as a power source, and the like. In one embodiment of the present disclosure, a "vehicle" may include at least one of an automobile, a train, and a motorcycle.

[0040] In the present disclosure, an 'occupant monitoring system (OMS)' refers to a device that captures an image of at least one occupant among a driver, a front passenger, and a rear-row passenger using a camera, and recognizes the face of the occupant from the image or recognizes the occupant's actions (e.g., drowsiness, sleep, conversation, mobile device operation, etc.) using an artificial intelligence model. The occupant monitoring system may include a driver monitoring system (DMS). In one embodiment of the present disclosure, the occupant monitoring system includes a camera that captures an occupant, and may acquire a respiration signal from an image of the driver (or occupant) captured through the camera.

[0041] In this disclosure, "abnormal respiration" refers to breathing in which the breath per minute, breathing pattern, or breathing waveform deviates from the range of normal breathing. For example, abnormal respiration may include hyperventilation, acute hyperventilation, hypopnea, transient apnea, and dyspnea.

[0042] In the present disclosure, a "breathing guide UI (user interface)" is an interface that provides guidance information for self-regulation of breathing by the driver (or passenger), thereby alleviating abnormal breathing and protecting the driver. In the present disclosure, the breathing guide UI may include guidance information that monitors breathing conditions such as inhalation and exhalation patterns and respiratory rate per minute, and induces self-breathing by the driver. In one embodiment of the present disclosure, the breathing guide UI may be displayed on an instrument cluster within a vehicle. However, the present disclosure is not limited thereto, and the breathing guide UI may be displayed through a Head-Up Display (HUD) projected on the windshield of the vehicle.

[0043] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0044] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0045] FIG. 1 is a drawing illustrating an operation of an electronic device (100) according to one embodiment of the present disclosure to detect the breathing of a driver (10) and provide a breathing exercise guide UI (user interface) (40).

[0046] The electronic device (100) is placed in a vehicle and may include a camera (110) and a display unit (140). Although not shown in FIG. 1, the electronic device (100) may further include other components in addition to the camera (110) and the display unit (140). The components of the electronic device (100) will be described in detail in FIG. 3.

[0047] The electronic device (100) may be implemented as an occupant monitoring system that monitors occupants, including the driver (10), within a vehicle. In the present disclosure, an 'occupant monitoring system (OMS)' refers to a device that captures an image of at least one occupant among the driver (10), the front passenger, and the rear row occupants using a camera (110), and recognizes the occupant's face from the image or recognizes the occupant's actions (e.g., drowsiness, sleep, conversation, mobile device operation, etc.) using an artificial intelligence model. The occupant monitoring system may include a driver monitoring system (DMS). In one embodiment of the present disclosure, the occupant monitoring system may obtain a respiration signal (22, 24) from an image of the driver (10) or occupant obtained through the camera (110).

[0048] In the entire embodiment of the present disclosure, all functions and / or operations of the electronic device (100), such as “capturing a breathing signal by photographing the driver (10)” and “displaying a breathing guide UI (40) for protecting the driver (10)”, may be equally applied to at least one passenger among the front passenger and the rear row passenger, as well as the driver (10), even if described as ‘the driver (10)’.

[0049] Referring to FIG. 1, a camera (110) is mounted within a vehicle and can capture images of passengers within the vehicle. In one embodiment of the present disclosure, an electronic device (100) can capture images of a driver (10) using the camera (110) to obtain an image including at least one of the driver's (10) body parts, such as the chest (12) and the abdomen (14) (Operation ①).

[0050] The electronic device (100) can obtain the driver's (10) breathing signal (22, 24) from the image (Operation ②). In one embodiment of the present disclosure, the electronic device (100) can obtain the driver's (10) breathing signal (22, 24) by detecting the movement of at least one body part among the chest (12) and the abdomen (14).

[0051] The electronic device (100) detects abnormal breathing of the driver (10) (Operation ③). In one embodiment of the present disclosure, abnormal breathing can be detected by monitoring the breathing signal (22, 24) of the driver (10) in real time. In one embodiment of the present disclosure, the electronic device (100) can detect abnormal breathing through a breathing waveform graph (30), such as when the breathing intensity exceeds a threshold or the number of breaths per minute increases rapidly. Abnormal breathing can include, for example, at least one of hyperventilation, acute hyperventilation, hypopnea, transient apnea, and dyspnea.

[0052] The electronic device (100) may display a breathing guide user interface (UI) (40) when abnormal breathing is detected (operation ④). In the present disclosure, the breathing guide UI (40) is an interface for alleviating abnormal breathing and protecting the driver (10) or passenger by providing guide information for self-regulation of breathing of the driver (10) or passenger. In the embodiment illustrated in FIG. 1, the breathing guide UI (40) may include a breathing waveform graph (42) indicating breathing patterns of inhalation and exhalation, number of breaths per minute, etc., and a breathing exercise guide (44) for inducing self-breathing of the driver (10).

[0053] The breathing guide UI (40) may be displayed through the display unit (140). The display unit (140) may be an instrument cluster display, but is not limited thereto. In one embodiment of the present disclosure, the display unit (140) may be configured as at least one of a Center Information Display (CID), a navigation device, a Head Up Display (HUD), or a passenger seat display.

[0054] Hereinafter, with reference to FIGS. 1 and 2 together, a detailed description will be given of the function and / or operation of an electronic device (100) detecting the breathing of a vehicle driver (10) and displaying a breathing guide UI (40) for protecting the driver (10).

[0055] FIG. 2 is a flowchart illustrating a method in which an electronic device (100) according to one embodiment of the present disclosure detects a driver's breathing and provides a breathing guide UI (user interface).

[0056] In step S210, the electronic device (100) detects movement of at least one of the chest and abdomen caused by the driver's breathing from an image acquired through a camera to acquire a breathing signal of the driver. Referring also to operation ① of FIG. 1, the electronic device (100) can capture a photo of the driver (10, see FIG. 1) using a camera (110, see FIG. 1) to acquire an image including at least one of the chest (12) and abdomen (14) among the body parts of the driver (10). The electronic device (100) can acquire a breathing signal (22, 24) of the driver (10) from the acquired image. Referring to operation ② of FIG. 1 together, the electronic device (100) recognizes at least one of the driver's (10) chest (12) and abdomen (14) from an image, detects and tracks minute movements (motion) of at least one of the recognized chest (12) and abdomen (14) during changes in breathing between inhalation and exhalation, and thereby obtains a breathing signal (22, 24).

[0057] In one embodiment of the present disclosure, the electronic device (100) can obtain the driver's (10) breathing signal (22, 24) by using an optical reflectance analysis technique that analyzes the brightness change of a body part (e.g., chest (12), abdomen (14)) from an image to track the micro-movement of the body part due to breathing. In one embodiment of the present disclosure, the electronic device (100) can obtain the driver's (10) breathing signal (22, 24) by using an optical flow analysis technique that analyzes the pixel value change of an image to track the micro-movement of a body part (e.g., chest (12), abdomen (14)) that changes during breathing. However, the present invention is not limited thereto, and the electronic device (100) may estimate the heart rate of the driver (10) by analyzing the skin absorption color amount from an image acquired through an RGB camera, and may obtain a respiration signal (22, 24) through a skin light absorption analysis technology (remote Photoplethysmography, rPPG) that measures respiration. In addition, in one embodiment of the present disclosure, the electronic device (100) may further include a thermal imaging camera or a radar sensor, and may obtain a respiration signal (22, 24) of the driver (10) by using either the thermal imaging camera or the radar sensor.

[0058] In step S220, the electronic device (100) monitors the breathing signal in real time to detect abnormal breathing of the driver. In one embodiment of the present disclosure, the electronic device (100) can acquire a breathing signal in real time and monitor the acquired breathing signal to detect abnormal breathing including at least one of hyperventilation, acute hyperventilation, hypopnea, transient apnea, and dyspnea. Referring also to operation ③ of FIG. 1, the electronic device (100) can detect abnormal breathing through a breathing waveform graph (30, see FIG. 1), such as when the breathing intensity exceeds a threshold value or the number of breaths per minute increases rapidly.

[0059] In one embodiment of the present disclosure, the electronic device (100) may acquire a respiration signal related to the driver's rest respiration when the vehicle is stopped or when driving at a low speed below a critical speed, and may detect abnormal respiration by comparing the driver's respiration signal monitored in real time with the rest respiration signal. For example, the electronic device (100) may determine that abnormal respiration has been detected if the difference between the respiration signal of rest respiration and the driver's respiration signal monitored in real time exceeds a preset threshold.

[0060] In step S230, the electronic device (100) displays a breathing guide UI (user interface) that provides a breathing exercise guide to alleviate abnormal breathing by realigning the driver's breathing. In the present disclosure, the 'breathing guide UI' is guide information that allows the driver or passenger to control breathing through self-breathing, and is an interface for alleviating abnormal breathing and protecting the driver or passenger. Referring to operation ④ of FIG. 1 together, the breathing guide UI (40, see FIG. 1) may include a breathing waveform graph (42, see FIG. 1) indicating a breathing pattern of inhalation and exhalation, the number of breaths per minute, etc., and a breathing exercise guide (44, see FIG. 1) that induces self-breathing of the driver (10). In one embodiment of the present disclosure, the electronic device (100) may display the breathing guide UI (40) that includes both breathing waveforms of a thoracic breathing signal obtained from the chest movement of the driver (10) and an abdominal breathing signal obtained from the abdominal movement of the driver among the breathing signals. In this case, the breathing guide UI (40) may include information on real-time changes in each of the thoracic breathing signal and the abdominal breathing signal.

[0061] In one embodiment of the present disclosure, the electronic device (100) may display the breathing guide UI (40) on the instrument cluster display of the vehicle. However, the present invention is not limited thereto, and the electronic device (100) may display the breathing guide UI (40) through a HUD (Head Up Display) projected on the windshield of the vehicle.

[0062] In one embodiment of the present disclosure, the electronic device (100) may determine the timing for displaying the breathing guide UI (40) based on at least one of whether the vehicle is executing autonomous driving and the driver's (10) intervention status in driving the vehicle. The electronic device (100) may recognize the autonomous driving level of the vehicle and determine whether to display the breathing guide UI (40) by determining the correlation between the driver's (10) breathing and vehicle driving control. For example, in a driving stage or autonomous driving level (e.g., level 0, level 1, level 2) that requires the driver's (10) intervention in driving the vehicle, the electronic device (100) may display the breathing guide UI (40) only when the vehicle is stopped or in a low-speed driving section. For example, in a fully autonomous driving mode (e.g., autonomous driving level 3 or higher), the electronic device (100) may display the breathing guide UI (40) regardless of the vehicle speed.

[0063] Although FIGS. 1 and 2 illustrate and describe a breathing guide UI (40) as being provided to provide visual information, the present disclosure is not limited thereto. In one embodiment of the present disclosure, the electronic device (100) further includes a speaker that outputs an acoustic signal, and may output a voice message guiding breathing exercises through the speaker. The voice message may include, for example, "Take deep, slow breaths," "Breath in," "Breath out," "Great job," etc. In one embodiment of the present disclosure, the electronic device (100) may also output a warning sound through the speaker when abnormal breathing of the driver (10) is detected.

[0064] In one embodiment of the present disclosure, when abnormal breathing of the driver (10) is detected, the electronic device (100) may control the steering wheel through an Electronic Control Unit (ECU) to generate a vibration effect in the steering wheel.

[0065] Recently, legislation is being implemented worldwide to mandate the installation of occupant monitoring systems in vehicles to protect drivers (10) and prevent road traffic accidents. In particular, the European Union is actively exploring the use of occupant monitoring systems in line with its goal of reducing traffic fatalities by half. Similarly, countries such as the United States, China, and Australia are planning to implement similar legislation, and Korea is also considering legislation requiring the installation of occupant monitoring systems in vehicles.

[0066] Meanwhile, vehicle manufacturers are attempting to equip their vehicles with driver state understanding technology to not only detect drowsy or inattentive driving, but also to advance autonomous driving and safety technologies. As part of this driver state understanding technology, a technology is needed that monitors the breathing of the driver (10) or passengers using an occupant monitoring system, detects abnormal breathing beyond the normal range, such as hyperventilation, hypopnea, temporary apnea, and dyspnea, thereby protecting the driver (or passengers) from abnormal breathing and preventing traffic accidents in advance.

[0067] The present disclosure provides an electronic device and an operating method thereof that obtains a breathing signal of a driver (10) or a passenger from an image obtained by photographing the driver (10) or a passenger through an occupant monitoring system (OMS) including a camera (110) and a display unit (140), and, when abnormal breathing outside the range of normal breathing is detected, provides breathing exercise guide information for the driver's (10) self-breathing to protect the driver (10) and prevent traffic accidents in advance.

[0068] The electronic device (100) according to the embodiment illustrated in FIGS. 1 and 2 captures an image of a driver (10) through a camera (110), analyzes the movement of at least one of the chest (12) and abdomen (14) of the driver (10) from the image to obtain a breathing signal of the driver (10), and monitors the breathing signal in real time to display a breathing guide UI (40) when abnormal breathing is detected, thereby assisting the driver's (10) self-breathing and alleviating the abnormal breathing. Through this, the electronic device (100) according to one embodiment of the present disclosure provides a technical effect of preventing traffic accidents caused by respiratory instability such as hyperventilation, acute hyperventilation, hypopnea, temporary apnea, and dyspnea in advance, and protecting the driver (10). In addition, the electronic device (100) according to one embodiment of the present disclosure provides a breathing exercise guide UI for abdominal breathing by the abdomen (14) as well as thoracic breathing by the chest (12), thereby controlling the balance of thoracic breathing and abdominal breathing, thereby providing technical effects such as stress relief and reduction of high blood pressure in the driver (10).

[0069] FIG. 3 is a block diagram illustrating components of an electronic device (100) according to one embodiment of the present disclosure.

[0070] The electronic device (100) may be implemented as an occupant monitoring system (OMS). Referring to FIG. 3, the electronic device (100) may include a camera (110), a processor (120), a memory (130), a display unit (140), and a speaker (150). The camera (110), the processor (120), the memory (130), the display unit (140), and the speaker (150) may be electrically and / or physically connected to each other. In FIG. 3, only essential components for explaining the operation of the electronic device (100) are illustrated, and the components included in the electronic device (100) are not limited to those illustrated in FIG. 3. In one embodiment of the present disclosure, the electronic device (100) may further include a communication interface (160, see FIG. 23) configured to transmit and receive data with a server (200, see FIGS. 22 and 23) or an external device.

[0071] In one embodiment of the present disclosure, the electronic device (100) may not include some of the components illustrated in FIG. 3. For example, the electronic device (100) may not include a speaker (150).

[0072] The camera (110) is configured to capture images of passengers, such as the driver and / or passengers, riding in the vehicle. The camera (110) may be configured as one or more. The camera (110) may include a lens module, an image sensor, and an image processing module. The camera (110) may capture still images or video of an object (e.g., a driver) by an image sensor (e.g., a CMOS or CCD). The video may include a plurality of image frames continuously captured by capturing an object through the camera. The image processing module may encode still images composed of a single image frame captured by the image sensor or video data composed of a plurality of image frames and transmit the encoded data to the processor (120).

[0073] In one embodiment of the present disclosure, the camera (110) may be implemented as an RGB camera. However, the present disclosure is not limited thereto, and the camera (110) may also be implemented as an IR (Infrared) camera including an infrared light source and an image sensor. In this case, the infrared illumination irradiates light in the infrared band toward a driver (or passenger) in the vehicle, and may be composed of, for example, a laser light source, a light-emitting diode (LED) light source, or a vertical-cavity surface-emission laser (VCSEL) light source. The image sensor may acquire an image by receiving light reflected from the infrared light source and converting it into a digital signal. In one embodiment of the present disclosure, the image sensor may acquire a driver image by collecting reflected light reflected from a passenger through a lens, receiving the reflected light through a light-receiving sensor, and converting an analog light signal into a digital signal through an analog-to-digital converter. The image sensor may be comprised of, for example, one of a complementary metal oxide semi-conductor (CMOS), a charge coupled device (CCD), or a charge priming device (CPD).

[0074] In one embodiment of the present disclosure, the camera (110) is configured as a combination of an RGB camera and an IR camera, and can acquire an image by photographing at least one of the driver's chest and abdomen.

[0075] The processor (120) can execute one or more instructions of a program stored in the memory (130). The processor (120) may be configured with hardware components that perform arithmetic, logic, and input / output operations and image processing. Although the processor (120) is illustrated as a single element in FIG. 3 , it is not limited thereto. In one embodiment of the present disclosure, the processor (120) may be configured with one or more multiple elements. One or more processors included in the processor (120) may be circuitry such as a system on chip (SoC), an integrated circuit (IC), or the like. For example, the processor (120) may be a general-purpose processor such as a central processing unit (CPU), an application processor (AP), a digital signal processor (DSP), a graphics-only processor such as a graphics processing unit (GPU), a vision processing unit (VPU), or an artificial intelligence-only processor such as a neural processing unit (NPU). If the processor (120) is an artificial intelligence-only processor, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0076] The processor (120) may include various processing circuits and / or multiple processors. For example, the term "processor" as used in this disclosure, including the claims, may include various processing circuits, including at least one processor. One or more processors in at least one processor may be configured to perform various functions described in this disclosure individually and / or collectively in a distributed manner. As used herein, "processor," "at least one processor," and "one or more processors" may be configured to perform various functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, at least one processor may include a combination of processors that perform various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0077] The memory (130) may be configured as at least one type of storage medium among, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), or an optical disk.

[0078] The memory (130) may store instructions related to functions and / or operations of the electronic device (100) acquiring a breathing signal of a vehicle driver, monitoring the breathing signal to detect abnormal breathing, and displaying a breathing guide UI when abnormal breathing is detected. In one embodiment of the present disclosure, the memory (130) may store at least one of instructions, an algorithm, a data structure, a program code, and an application program that can be read by the processor (120). The instructions, algorithms, data structures, and program codes stored in the memory (130) may be implemented in a programming or scripting language such as, for example, C, C++, Java, or an assembler.

[0079] The processor (120) can perform functions and / or operations of acquiring a vehicle driver's breathing signal, monitoring the breathing signal to detect abnormal breathing, and displaying a breathing guide UI when abnormal breathing is detected by executing instructions or program codes stored in the memory (130). Hereinafter, the functions and / or operations performed by the processor (120) by executing instructions or program codes stored in the memory (130) will be described.

[0080] The processor (120) can capture an image of the driver through the camera (110) and obtain the driver's breathing signal from the obtained image. In one embodiment of the present disclosure, the processor (120) can recognize a breathing detection area including at least one of the chest and abdomen among the driver's body parts from the image obtained through the camera (110). For example, the processor (120) can extract shoulder feature points of the driver from the image and recognize at least one breathing detection area of ​​the driver's chest area and abdomen area based on the extracted shoulder feature points. For example, the processor (120) can extract shoulder feature points and pelvic feature points of the driver from the image and recognize at least one breathing detection area of ​​the driver's chest area and abdomen area based on the extracted shoulder feature points and pelvic feature points. If the driver is wearing a seat belt and thus obscures the breathing detection area, the processor (120) can identify the seat belt area from the image and recognize an effective area for detecting breathing by removing the seat belt area. A specific embodiment in which the processor (120) recognizes the breathing detection area will be described in detail in FIGS. 4 to 9.

[0081] The processor (120) can obtain the driver's breathing signal by detecting micro-movements of a breathing detection area including at least one of the driver's chest and abdomen. In one embodiment of the present disclosure, the processor (120) can obtain the driver's breathing signal by using an optical reflectance analysis technique that irradiates a light source (e.g., an infrared light source) coupled to the camera (110) to the driver's body parts (e.g., the chest and abdomen) and analyzes the change in light reflectance of the infrared light source reflected from the body parts to determine the driver's breathing pattern and breathing rate of inhalation and exhalation. The optical reflectance analysis technique is a method of identifying a body part from an image and analyzing the change in brightness of the body part to track the micro-movement of the body part due to breathing. In one embodiment of the present disclosure, the processor (120) can also obtain the driver's breathing signal by using an optical flow analysis technique that analyzes the change in pixel values ​​of an image acquired through the camera (110) to track the micro-movements of a body part (e.g., the chest and abdomen) that change during breathing.

[0082] However, the method by which the processor (120) obtains the driver's breathing signal from the image is not limited as described above. In one embodiment of the present disclosure, the processor (120) may estimate the driver's heart rate by analyzing the skin absorption color amount from the image obtained through the RGB camera, and may obtain the driver's breathing signal through a skin light absorption analysis technology (remote Photoplethysmography, rPPG) that measures breathing. In addition, in one embodiment of the present disclosure, the camera (110) may include at least one of a thermal imaging camera or a radar sensor as a component, and the processor (120) may further include a thermal imaging camera or a radar sensor, and may obtain the driver's breathing signal using either the thermal imaging camera or the radar sensor.

[0083] In one embodiment of the present disclosure, the processor (120) may recognize a body part of the driver from an image acquired through the camera (110) and segment the recognized body part into a plurality of regions. For example, the processor (120) may recognize body parts such as the driver's face, neck, chest, and abdomen from the image, and segment the recognized body parts into the face, neck, chest, and abdomen, respectively. The processor (120) may detect movement of the segmented plurality of regions and acquire a breathing signal including at least one of a breathing waveform and a breathing intensity from each of the plurality of regions. The processor (120) may select an optimal region for monitoring the driver's breathing signal from among the plurality of regions based on at least one of the pattern consistency of the breathing waveform and the amount of change in the breathing intensity, and acquire the breathing signal of the selected region in real time. A specific embodiment of the processor (120) segmenting the driver's body part from the image into a plurality of regions and selecting an optimal region for monitoring the breathing signal from among the segmented plurality of regions will be described in detail with reference to FIGS. 10 and 11 .

[0084] However, the present invention is not limited thereto, and in one embodiment of the present disclosure, the processor (120) may merge breathing signals obtained from each of a plurality of regions and monitor the merged breathing signal in real time.

[0085] When acquiring a driver's breathing signal, if the vehicle vibrates or the driver moves his or her body, the accuracy of the breathing signal may be reduced due to motion artifacts caused by the vibration and body movement. In one embodiment of the present disclosure, the processor (120) acquires motion information caused by the vehicle's vibration or the driver's body movement, identifies a time period in which motion information is detected among the breathing waveforms according to the passage of time of the breathing signal, and corrects the breathing signal in the identified period. A specific embodiment in which the processor (120) acquires motion information caused by the vehicle's vibration and the driver's body movement, and corrects the breathing signal based on the motion information will be described in detail with reference to FIGS. 12 and 13.

[0086] The processor (120) can monitor the acquired breathing signal in real time to detect abnormal breathing, including at least one of hyperventilation, hypoventilation, transient apnea, and dyspnea. In one embodiment of the present disclosure, the processor (120) can monitor the breathing signal in real time to detect a situation in which abnormal breathing occurs, such as when the breathing intensity exceeds a threshold or when the number of breaths per minute increases rapidly.

[0087] In one embodiment of the present disclosure, the processor (120) acquires a respiration signal related to the driver's rest respiration when the vehicle is stopped or driving at a low speed below a critical speed, and compares the driver's respiration signal monitored in real time with the rest respiration signal to detect abnormal respiration. A specific embodiment in which the processor (120) detects abnormal respiration by comparing the respiration signal of rest respiration with the respiration signal acquired in real time will be described in detail in FIG. 14.

[0088] The processor (120) may display a breathing guide UI (user interface) that provides breathing exercise guidance on the display unit (140) when abnormal breathing is detected. In the present disclosure, the 'breathing guide UI' is guide information that allows the driver or passenger to control breathing through self-breathing, and is an interface for alleviating abnormal breathing and protecting the driver or passenger. In one embodiment of the present disclosure, the processor (120) may display a breathing guide UI that includes a breathing waveform graph indicating a breathing pattern of inhalation / exhalation, number of breaths per minute, etc., and a breathing exercise guide graphic that induces self-breathing of the driver through the display unit (140).

[0089] In one embodiment of the present disclosure, the processor (120) may display, through the display unit (140), a breathing guide UI including both a thoracic breathing signal obtained from the driver's chest movement among breathing signals and a respiratory waveform of an abdominal breathing signal obtained from the driver's abdominal movement. In this case, the breathing guide UI may include information regarding real-time changes in each of the thoracic breathing signal and the abdominal breathing signal.

[0090] In one embodiment of the present disclosure, the processor (120) may recognize at least one of whether the vehicle is performing autonomous driving and the driver's intervention in driving the vehicle, and determine the timing for displaying a breathing guide UI based on the recognition result. The processor (120) may recognize the autonomous driving level of the vehicle, determine the correlation between the driver's breathing and vehicle driving control, and determine whether to display the breathing guide UI. For example, in a driving stage or autonomous driving level (e.g., level 0, level 1, level 2) that requires driver intervention in driving the vehicle, the processor (120) may display the breathing guide UI only when the vehicle is stopped or in a low-speed driving section. For example, in a fully autonomous driving mode (e.g., autonomous driving level 3 or higher), the processor (120) may display the breathing guide UI regardless of the vehicle's speed.

[0091] Specific examples of the breathing guide UI displayed through the display unit (140) will be described in detail in FIGS. 15a, 15b, 16, 17, and 18.

[0092] In one embodiment of the present disclosure, when the processor (120) recognizes that a driving stage requires driver intervention, the processor (120) may generate a simplified breathing guide UI so as not to interfere with vehicle driving, and display the simplified breathing guide UI through the display unit (140). In a situation where the vehicle is driving in a fully autonomous driving mode, the processor (120) may control the display unit (140) to display the breathing guide UI in a full display state.

[0093] In one embodiment of the present disclosure, the processor (120) may determine whether to execute a breathing guide UI based on the characteristics of the driving route in conjunction with the vehicle's navigation device. For example, the processor (120) may receive information about the driving route from the navigation device, identify complex driving routes requiring concentration, high-occurrence locations of vehicle accidents, or locations requiring attention, and display the breathing guide UI when the vehicle deviates from the identified route or location.

[0094] In one embodiment of the present disclosure, if hyperventilation is detected during abnormal respiration, the processor (120) can control the driving of the vehicle to output a warning message indicating an emergency situation and, if there is no driver interaction with the output warning message, move the vehicle to the shoulder of the road and stop. The processor (120) can send an emergency call for rescue. A specific embodiment of the processor (120) stopping the vehicle to the shoulder of the road and sending an emergency call when hyperventilation is detected will be described in detail with reference to FIGS. 19 and 20 .

[0095] The display unit (140) is configured to display a breathing guide UI under the control of the processor (120). In one embodiment of the present disclosure, the display unit (140) may be implemented as an instrument cluster display. However, the present disclosure is not limited thereto, and the display unit (140) may be configured as at least one of, for example, a Center Information Display (CID), a Head Up Display (HUD), a navigation device, or a passenger seat display.

[0096] The display unit (140) may include a screen composed of at least one of, for example, a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode display, a flexible display, a 3D display, or an electrophoretic display.

[0097] The speaker (150) is a device configured to output an acoustic signal. The speaker (150) can output a voice message guiding breathing exercises under the control of the processor (120). The voice message may include, for example, "Take deep, slow breaths," "Breath in," "Breath out," "Great job," etc. In one embodiment of the present disclosure, the speaker (150) can also output a warning sound under the control of the processor (120) when abnormal breathing of the driver is detected.

[0098] In one embodiment of the present disclosure, the speaker (150) may output sound effects that guide breathing exercises. For example, the speaker (150) may output sound effects such as a countdown sound effect for the start of breathing exercises, an inhalation / exhalation breathing exercise sound effect, or a natural sound background sound.

[0099] Although not illustrated in FIG. 3, the electronic device (100) can control various devices of the vehicle (e.g., seat belts, steering wheel, ambient lights, seats, etc.) through an Electronic Control Unit (ECU). In one embodiment of the present disclosure, when abnormal breathing of the driver is detected, the processor (120) of the electronic device (100) can control various devices of the vehicle through the ECU to alleviate the driver's abnormal breathing and perform functions and / or operations that can protect the driver by lowering stress and blood pressure. A specific embodiment in which the electronic device (100) controls the devices of the vehicle through the ECU when abnormal breathing of the driver is detected will be described in detail in FIG. 21.

[0100] FIG. 4 is a flowchart illustrating a method for an electronic device (100) to obtain a driver's breathing signal according to one embodiment of the present disclosure.

[0101] FIG. 5 is a drawing for explaining an operation of an electronic device (100) according to one embodiment of the present disclosure to obtain a breathing signal from at least one region of the chest (510) and abdomen (520) identified based on the driver's shoulder feature points (P1, P2).

[0102] Hereinafter, with reference to FIGS. 4 and 5 together, a detailed description will be given of a function and / or operation of an electronic device (100) according to one embodiment of the present disclosure for identifying at least one region among the chest (510) and the abdomen (520) and obtaining a respiration signal from the identified region.

[0103] Steps S410 to S450 illustrated in FIG. 4 are steps that embody step S210 of FIG. 2. Referring to FIG. 4, in step S410, the electronic device (100) captures an image of the driver using a camera. Referring also to FIG. 5, the electronic device (100) can capture an image (500) of the driver's upper body, including the face, neck, chest, and abdomen, using a camera.

[0104] In step S420 of FIG. 4, the electronic device (100) extracts feature points of both shoulders of the driver from the image and calculates the center point of respiration. In one embodiment of the present disclosure, the electronic device (100) can recognize key feature points of body parts from the image of the driver by performing vision recognition using an artificial intelligence model. In the present disclosure, 'vision recognition' means image signal processing that inputs an image to an artificial intelligence model and, through inference using the artificial intelligence model, detects an object from the input image, classifies the object into a specific category, or segments the object. In one embodiment of the present disclosure, the 'artificial intelligence model' can be implemented as a deep neural network model that is trained to recognize an object (e.g., a face, a neck, a chest, and an abdomen, etc.) from input image data and recognize feature points of the object (e.g., a facial feature point, a neck feature point, a shoulder feature point, a pelvic feature point, etc.). In the present disclosure, the "deep neural network model" may be an end-to-end model trained using a supervised learning method that applies tens of thousands or hundreds of millions of multiple images as input data and applies the feature points of body parts included in the input data as ground truth. The deep neural network model may be implemented as, for example, a convolutional neural network (CNN) model, but is not limited thereto.Deep neural network models can be implemented as, for example, Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), or Deep Q-Networks.

[0105] However, the present invention is not limited thereto, and in one embodiment of the present disclosure, the processor (120) may extract the driver's body features from the image using a known image processing technique.

[0106] In the embodiment illustrated in FIG. 5, the processor (120, see FIG. 3) of the electronic device (100) performs vision recognition by inputting an image (500) into an artificial intelligence model, thereby extracting a right shoulder feature point (P1) and a left shoulder feature point (P2) from the image (500). The processor (120) calculates a median point centered on the extracted right shoulder feature point (P1) and left shoulder feature point (P2), and thereby calculates a respiration center point (P m ) can identify the location. In one embodiment of the present disclosure, the processor (120) calculates the midpoint of the shoulder feature points (P1, P2) on both sides through a center point extraction calculation method using two points and a ring, and moves the central axis around the calculated midpoint to identify the breathing center point (P m ) can obtain location information.

[0107] Referring again to FIG. 4, in step S430, the electronic device (100) determines at least one region among the thoracic region and the abdominal region based on the breathing center point. Referring also to FIG. 5, the processor (120) of the electronic device (100) determines the breathing center point (P m) can be determined as the chest (510) area. In one embodiment of the present disclosure, the processor (120) determines the breathing center point (P m ) can be determined as the abdominal (520) region, with a predetermined size centered on a point spaced a preset distance (d) from the abdominal region.

[0108] In the embodiment illustrated in FIG. 5, the electronic device (100) is illustrated as determining both the chest (510) and abdomen (520) regions from the image (500), but the embodiment of the present disclosure is not limited to the illustrated embodiment. In one embodiment of the present disclosure, the processor (120) of the electronic device (100) may determine only the chest (510) region or only the abdomen (520) region.

[0109] Referring back to FIG. 4, in step S440, the electronic device (100) detects movement of at least one of the chest region and the abdomen region to obtain a respiration signal. In one embodiment of the present disclosure, the processor (120) of the electronic device (100) can obtain a respiration signal by recognizing and tracking micro-movement of at least one of the chest (510) and abdomen (520) regions using any one of optical reflectance analysis technology, optical flow analysis technology, and skin optical absorbance analysis technology. The method of obtaining a respiration signal using optical reflectance analysis, optical flow analysis, skin optical absorbance analysis technology, etc. is the same as that described with reference to FIGS. 2 and 3, and therefore, redundant descriptions are omitted. However, the method of obtaining a respiration signal is not limited to the above-described techniques, and the electronic device (100) of the present disclosure can obtain a respiration signal by recognizing and tracking micro-movement of at least one of the chest (510) and abdomen (520) regions using any known technique.

[0110] In step S450 of FIG. 4, the electronic device (100) filters the respiratory signal to obtain a respiratory waveform. The respiratory waveform may include information about the breathing pattern of inhalation / exhalation, respiratory volume, respiratory rate per minute, etc. based on the respiratory signal obtained over time.

[0111] After step S450 is performed, step S220 illustrated in FIG. 2 may be performed.

[0112] FIG. 6 is a flowchart illustrating a method for an electronic device (100) according to one embodiment of the present disclosure to obtain a driver's breathing signal.

[0113] FIG. 7 is a drawing for explaining an operation of an electronic device (100) according to one embodiment of the present disclosure to obtain a breathing signal from at least one region of the chest (710) and abdomen (720) identified based on the driver's shoulder feature points (P1, P2) and pelvic feature points (P3, P4).

[0114] Hereinafter, with reference to FIGS. 6 and 7 together, a detailed description will be given of a function and / or operation of an electronic device (100) according to one embodiment of the present disclosure for identifying at least one region among the chest (710) and the abdomen (720) and obtaining a respiration signal from the identified region.

[0115] Steps S610 to S650 illustrated in FIG. 6 are steps that embody step S210 of FIG. 2. Referring to FIG. 6, in step S610, the electronic device (100) captures an image of the driver using a camera. Referring also to FIG. 7, the electronic device (100) can capture an image (700) of the driver's upper body, including the face, neck, chest, and abdomen, using a camera.

[0116] In step S620 of FIG. 6, the electronic device (100) extracts the driver's shoulder feature points and pelvic feature points from the image to calculate the center of gravity. In one embodiment of the present disclosure, the electronic device (100) may perform vision recognition using an artificial intelligence model to recognize key feature points of body parts from the driver's image. Since a specific description of the vision recognition using the artificial intelligence model is the same as step S420 of FIG. 4, redundant descriptions are omitted. Referring also to the embodiment illustrated in FIG. 7, the processor (120, see FIG. 3) of the electronic device (100) may perform vision recognition to input an image (700) into the artificial intelligence model to extract shoulder feature points (P1, P2) on both sides and pelvic feature points (P3, P4) on both sides from the image (700). However, it is not limited thereto, and the processor (120) may also extract shoulder feature points (P1, P2) and pelvic feature points (P3, P4) on both sides from the image (700) using a known image processing technique.

[0117] The processor (120) constructs a virtual polygon based on the extracted shoulder feature points (P1, P2) and pelvic feature points (P3, P4) on both sides, and the center of gravity point (P) of the virtual polygon G ) can be calculated. The processor (120) calculates the calculated center of gravity point (P G ) can be used to identify the location of the breathing center by moving the central axis around it.

[0118] Referring again to FIG. 6, in step S630, the electronic device (100) determines at least one region among the chest region and the abdomen region based on the center of gravity point. Referring also to FIG. 7, the processor (120) of the electronic device (100) determines the center of gravity point (P G ) can be determined as the chest (710) area with a predetermined size centered on the position of the center of gravity point (P). In one embodiment of the present disclosure, the processor (120) determines the center of gravity point (P G) can be determined as the abdomen (720) region with a predetermined size centered on a point spaced a preset distance (d).

[0119] In the embodiment illustrated in FIG. 7, the electronic device (100) is illustrated as determining both the thorax (710) and abdomen (720) regions from the image (700), but the embodiment of the present disclosure is not limited to the illustrated embodiment. In one embodiment of the present disclosure, the processor (120) of the electronic device (100) may determine only the thorax (710) region or only the abdomen (720) region.

[0120] In step S640 of FIG. 6, the electronic device (100) detects movement of at least one region among the thoracic region and the abdominal region to obtain a respiration signal. In step S650, the electronic device (100) filters the respiration signal to obtain a respiration waveform. Steps S640 and S650 of FIG. 6 are identical to steps S440 and S450 illustrated in FIG. 4, and therefore, redundant descriptions are omitted.

[0121] After step S650 is performed, step S220 illustrated in FIG. 2 may be performed.

[0122] FIG. 8 is a flowchart illustrating a method for an electronic device (100) to obtain a driver's breathing signal according to one embodiment of the present disclosure.

[0123] FIG. 9 is a drawing for explaining an operation of an electronic device (100) according to one embodiment of the present disclosure to obtain a breathing signal from a valid area excluding a seat belt among the entire area of ​​an image captured of a driver.

[0124] Hereinafter, with reference to FIGS. 8 and 9 together, the electronic device (100) according to one embodiment of the present disclosure will be described in detail with respect to the function and / or operation of a breathing signal from an effective area excluding a seat belt (930) area.

[0125] Steps S810 to S860 illustrated in FIG. 8 are steps that embody step S210 of FIG. 2. Referring to FIG. 8, in step S810, the electronic device (100) captures an image of the driver using a camera. Referring also to FIG. 9, the electronic device (100) can capture an image (900) of the driver's upper body, including the face, neck, chest, and abdomen, using a camera.

[0126] In step S820 of FIG. 8, the electronic device (100) extracts the driver's shoulder feature points, pelvic feature points, and seat belt area from the image. In one embodiment of the present disclosure, the electronic device (100) may perform vision recognition using an artificial intelligence model to recognize key feature points of body parts and the seat belt area from the driver's image. Since a specific description of the vision recognition using the artificial intelligence model is the same as step S420 of FIG. 4, redundant description will be omitted. Referring also to the embodiment illustrated in FIG. 9, the processor (120, see FIG. 3) of the electronic device (100) may perform vision recognition to input an image (900) into the artificial intelligence model to extract shoulder feature points (P1, P2), pelvic feature points (P3, P4), and seat belts (930) from the image (900). However, it is not limited thereto, and the processor (120) may also extract shoulder feature points (P1, P2), pelvic feature points (P3, P4), and seat belt (930) from the image (900) using a known image processing technique.

[0127] Referring again to FIG. 8, in step S830, the electronic device (100) determines at least one region among the chest region and the abdomen region based on the shoulder feature points and the pelvic feature points. Referring also to the embodiment illustrated in FIG. 9, the processor (120) constructs a virtual polygon based on the extracted shoulder feature points (P1, P2) and pelvic feature points (P3, P4) on both sides, and determines the center of gravity point (P) of the virtual polygon. G ) can be calculated. The processor (120) calculates the calculated center of gravity point (P G ) can be determined as the chest (910) area. In one embodiment of the present disclosure, the processor (120) determines the center of gravity point (P G ) can be determined as the abdominal (920) region, with a predetermined size centered on a point spaced a preset distance (d) from the abdominal region.

[0128] In the embodiment illustrated in FIG. 9, the electronic device (100) is illustrated as determining both the chest (910) and abdomen (920) regions from the image (900), but the embodiment of the present disclosure is not limited to the illustrated embodiment. In one embodiment of the present disclosure, the processor (120) of the electronic device (100) may determine only the chest (910) region or only the abdomen (920) region.

[0129] In step S840 of FIG. 8, the electronic device (100) detects a valid area excluding the seat belt area among the chest area and the abdomen area. Referring also to FIG. 9, the processor (120) may segment an area covered by the seat belt (930) among the chest (910) and abdomen (920) areas, thereby removing the seat belt (930) from the entire area of ​​the image (900). The processor (120) may determine at least one area among the chest (910) and abdomen (920) from which the seat belt (930) has been removed as a valid area.

[0130] In step S850 of FIG. 8, the electronic device (100) detects movement in the valid area to obtain a respiration signal. The specific method for obtaining the respiration signal is the same as step S440 illustrated in FIG. 4, so a redundant description is omitted.

[0131] In step S860 of FIG. 8, the electronic device (100) filters the respiration signal to obtain a respiration waveform. Step S860 of FIG. 8 is identical to step S450 illustrated in FIG. 4, and therefore, a duplicate description will be omitted.

[0132] After step S860 is performed, step S220 illustrated in FIG. 2 may be performed.

[0133] In the case of optical reflectance analysis technology and optical flow analysis technology, they are effective in a static measurement environment where the subject (e.g., the driver) does not move, but the accuracy of the respiratory signal measurement may decrease in a vehicle environment where there is frequent movement. In the embodiment illustrated in FIGS. 4 to 9, when the driver's body movement occurs, the electronic device (100) measures depth information according to the distance from the camera for the shoulder length and seat position measured through the shoulder feature points (P1, P2) on both sides, and compensates for the amount of light in the driver's breathing detection area (e.g., the chest or abdomen) as the depth value increases or decreases. Through this, the electronic device (100) according to one embodiment of the present disclosure can continuously and consistently measure the driver's respiratory signal.

[0134] In particular, in the embodiments illustrated in FIGS. 8 and 9, the electronic device (100) can prevent the occurrence of artifacts in the respiratory signal due to the difference in light reflectivity caused by the driver's body part being covered by the seat belt (930, see FIG. 9), thereby detecting movement due to breathing more precisely and improving the accuracy of the respiratory signal.

[0135] FIG. 10 is a flowchart illustrating a method for an electronic device (100) according to one embodiment of the present disclosure to select a specific area among a plurality of areas included in a driver's body part and obtain a breathing signal.

[0136] FIG. 11 is a drawing for explaining an operation of an electronic device (100) according to one embodiment of the present disclosure to select a specific area among a plurality of areas (1110 to 1140) included in a body part from an image (1100) of a driver and obtain a breathing signal.

[0137] Hereinafter, with reference to FIGS. 10 and 11, a detailed description will be given of a function and / or operation of an electronic device (100) according to one embodiment of the present disclosure for selecting a specific area among a plurality of areas (1110 to 1140) included in a body part from a driver's image (1100) and obtaining a breathing signal.

[0138] Steps S1010 to S1040 illustrated in FIG. 10 are steps that specify step S210 of FIG. 2. Referring to FIG. 10, in step S1010, the electronic device (100) segments a driver's body part included in an image acquired through a camera into a plurality of regions. In one embodiment of the present disclosure, the electronic device (100) can perform vision recognition using an artificial intelligence model to recognize and segment major body parts from the driver's image. In one embodiment of the present disclosure, the 'artificial intelligence model' can be implemented as a deep neural network model trained to recognize and classify objects (e.g., face, neck, chest, abdomen, etc.) from input image data, and to segment the objects. In the present disclosure, a 'deep neural network model' may be an end-to-end model trained in a supervised learning manner that applies images of tens of thousands or hundreds of millions of body parts as input data, and applies labels representing classification information of the body parts, for example, labels mathematically embedded with face, neck, chest, abdomen, etc., as ground truth values. The deep neural network model may be implemented as, for example, a convolutional neural network (CNN) model, but is not limited thereto. The deep neural network model may also be implemented as, for example, a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network.

[0139] Referring to the embodiment illustrated in FIG. 11, the processor (120, see FIG. 3) of the electronic device (100) can input an image (1100) obtained by photographing a driver through a camera into an artificial intelligence model, and perform vision recognition using the artificial intelligence model to recognize body parts of the driver, for example, a face (1110), a neck (1120), a chest (1130), and an abdomen (1140), from the image (1100). The processor (120) can segment the recognized face (1110), neck (1120), chest (1130), and abdomen (1140), respectively.

[0140] However, the present invention is not limited thereto, and in one embodiment of the present disclosure, the processor (120) may recognize a body part of a driver from an image (1100) using a known image processing technique, and segment the recognized body part into a plurality of regions, for example, a face (1110), a neck (1120), a chest (1130), and an abdomen (1140).

[0141] Referring back to FIG. 10, in step S1020, the electronic device (100) detects movement of the divided plurality of regions and obtains a respiration signal including at least one of a respiration waveform and a respiration intensity from each of the plurality of regions. In one embodiment of the present disclosure, the processor (120) of the electronic device (100) may recognize and track micro-movements from each of the plurality of regions using any one of optical reflectance analysis technology, optical flow analysis technology, and skin optical absorbance analysis technology to obtain a respiration signal. The method of obtaining a respiration signal using optical reflectance analysis, optical flow analysis, skin optical absorbance analysis technology, etc. is the same as that described with reference to FIGS. 2 and 3, and therefore, redundant descriptions are omitted. Referring also to FIG. 11, the processor (120) may recognize and track micro-movements of the face (1110) to obtain a first respiration signal (1112) regarding the face (1110). Similarly, the processor (120) can recognize and track micro-movements of the neck (1120) to obtain a second respiration signal (1122), recognize and track micro-movements of the chest (1130) to obtain a third respiration signal (1132), and recognize and track micro-movements of the abdomen (1140) to obtain a fourth respiration signal (1142).

[0142] Referring back to FIG. 10, in step S1030, the electronic device (100) selects an optimal region for monitoring the driver's breathing signal from among a plurality of regions based on at least one of the pattern consistency of the breathing waveform and the amount of change in the breathing intensity. The electronic device (100) may determine the degree of stability of the breathing waveform and the breathing intensity based on vibration of the vehicle or body movement, and adaptively select an optimal region for best monitoring breathing from among a plurality of regions of the driver's body. In one embodiment of the present disclosure, the criterion for determining the 'degree of stability of the breathing waveform and the breathing intensity' may include at least one of the pattern consistency of the breathing waveform and the amount of change in the breathing intensity. For example, if the breathing waveform changes irregularly or the amount of change in the breathing intensity changes to a value exceeding a preset range, the electronic device (100) may determine that the breathing waveform and the breathing intensity are not stable.

[0143] Referring to the embodiment illustrated in FIG. 11, the processor (120) of the electronic device (100) may analyze the pattern of the respiratory waveform and the amount of change in the respiratory intensity of the first respiratory signal (1112) to the fourth respiratory signal (1142), select the third respiratory signal (1132) having the most stable respiratory signal among the first respiratory signal (1112) to the fourth respiratory signal (1142), and select the chest (1130), which is the body region where the selected third respiratory signal (1132) is measured, as the optimal region for monitoring the respiratory signal. The selection of the chest (1130) in FIG. 11 is merely an example for the convenience of explanation, and the electronic device (100) according to the embodiment of the present disclosure is not limited to selecting the chest (1130) as the optimal region for monitoring the respiratory signal among the plurality of regions (1110 to 1140).

[0144] Referring again to FIG. 10, in step S1040, the electronic device (100) acquires a respiratory signal of a selected area in real time. Referring also to FIG. 11, the processor (120) of the electronic device (100) can recognize and track the micro-movement of the thorax (1130) caused by the driver's breathing, thereby acquiring a third respiratory signal (1132) caused by thoracic breathing in real time.

[0145] After step S1040 is performed, step S220 illustrated in FIG. 2 may be performed.

[0146] In general, a relatively accurate breathing signal can be obtained from the chest or abdomen of a vehicle driver, but if the driver wears thick outerwear (e.g., padding, down jacket, jumper, etc.), it may be difficult to measure and obtain a fine breathing signal. Conversely, if the driver wears accessories such as sunglasses, a mask, hijab, or muffler, the accuracy of the breathing signal from the face, neck, etc. may be low. The electronic device (100) according to the embodiment illustrated in FIGS. 10 and 11 divides the driver's body part into a plurality of regions (1110 to 1140) from an image (1100) obtained through a camera, dynamically selects an optimal region for measuring and monitoring a breathing signal among the plurality of regions (1110 to 1140), and obtains a breathing signal from the selected region, thereby improving the accuracy of the breathing signal.

[0147] In one embodiment of the present disclosure, the processor (120) of the electronic device (100) may merge the first respiration signal (1112) to the fourth respiration signal (1142) acquired from each of the plurality of regions (1110 to 1140) and monitor the merged respiration signal in real time to detect abnormal respiration. As the respiration waveforms of the first respiration signal (1112) to the fourth respiration signal (1142) acquired from each of the plurality of regions (1110 to 1140) are merged, even if movement occurs in some regions among the plurality of regions (1110 to 1140) or the respiration signal is distorted due to vibration of the vehicle, the accuracy of the respiration signal may be improved because it is compensated for and canceled by the respiration signals of other regions.

[0148] FIG. 12 is a flowchart illustrating a method for an electronic device (100) according to one embodiment of the present disclosure to correct a breathing signal based on motion information resulting from vibration of a vehicle or body movement of a driver.

[0149] FIG. 13 is a graph (1300, 1310) for explaining an operation of an electronic device (100) according to one embodiment of the present disclosure to correct a breathing signal based on motion information caused by vibration of a vehicle or body movement of a driver.

[0150] Hereinafter, with reference to FIGS. 12 and 13, a detailed description will be given of a function and / or operation of an electronic device (100) according to one embodiment of the present disclosure for correcting a breathing signal based on motion information resulting from vibration of a vehicle or body movement of a driver.

[0151] Steps S1210 to S1230 illustrated in FIG. 12 are steps that specify step S210 of FIG. 2. Referring to FIG. 12, in step S1210, the electronic device (100) obtains motion information due to vibration of the vehicle or body movement of the driver. Referring also to FIG. 13, the processor (120, see FIG. 3) of the electronic device (100) can obtain a respiration waveform graph (1300) representing a respiration signal over time based on an image obtained through a camera. The processor (120) can obtain a motion signal graph (1310) representing a motion measurement value by measuring vibration of the vehicle or body movement of the driver over time.

[0152] Referring back to FIG. 12, in step S1220, the electronic device (100) identifies a time period in which motion information is detected among the respiratory waveform. In one embodiment of the present disclosure, the processor (120) of the electronic device (100) can identify a time period in which a motion signal exceeds a preset threshold. Referring to the graphs (1300, 1310) illustrated in FIG. 13, the processor (120) can identify a plurality of time periods (1312, 1314, 1316) in which a motion signal value exceeds a preset threshold from the motion signal graph (1310). The processor (120) can measure the respiratory signal value of the respiratory waveform graph (1300) in the time periods corresponding to the identified time periods (1312, 1314, 1316), thereby identifying a time period (1302, 1304, 1306) in which a distortion of the respiratory signal occurs. In the identified time intervals (1302, 1304, 1306), the breathing pattern of inhalation / exhalation is not constant, and the breaths per minute is also distorted, resulting in a value exceeding the driver's original breaths per minute.

[0153] Referring back to FIG. 12, in step S1230, the electronic device (100) corrects the respiratory signal of the respiratory waveform in the identified section. In one embodiment of the present disclosure, the processor (120) of the electronic device (100) can correct the respiratory signal by removing an abnormal signal exceeding a preset threshold value among the respiratory signal values ​​in a time section where the respiratory signal is distorted due to a motion artifact caused by vibration of the vehicle or a driver's body movement.

[0154] In one embodiment of the present disclosure, the processor (120) can compensate for the respiratory signal by smoothing the respiratory signal in a time interval where the respiratory signal is distorted using a filter. The processor (120) can smooth the respiratory signal distorted by motion artifacts using, for example, a median filter or a Savgol filter.

[0155] In one embodiment of the present disclosure, the processor (120) can correct the respiratory signal by adjusting the peak distance at which the peak value of the respiratory signal is observed in a time interval where the respiratory signal is distorted. For example, when the respiratory signal is distorted by motion artifacts and the peak value is frequently measured compared to the respiratory waveform of normal breathing, the processor (120) can correct the distortion that shortens the inhalation / exhalation pattern by increasing the peak distance.

[0156] After step S1230 of FIG. 12 is performed, step S220 illustrated in FIG. 2 may be performed.

[0157] When vibration of a vehicle or arbitrary or random body movement of a driver occurs, distortion may occur in the light reflectivity, which may lower the accuracy of the respiratory signal. The electronic device (100) according to the embodiment illustrated in FIGS. 12 and 13 obtains a motion signal caused by vibration of a vehicle or body movement of a driver, identifies a time section in which distortion of the respiratory signal occurs due to the motion signal, and corrects the respiratory signal in the identified time section, thereby reducing errors such as respiration rate per minute and respiration intensity, and improving the accuracy of the respiratory signal. As a result of the experiment, the Mean Absolute Error (MAE) error of the respiratory signal was reduced by 2.35 bpm (breaths per minute) and the accuracy was increased by 50.5% compared to the case where distortion of the respiratory signal was not corrected.

[0158] FIG. 14 is a flowchart illustrating a method for an electronic device (100) to detect abnormal respiration based on rest respiration according to one embodiment of the present disclosure.

[0159] Steps S1410 and S1420 are steps that are specific examples of step S210 illustrated in FIG. 2. In step S1410, the electronic device (100) obtains a respiration signal related to the driver's rest respiration when the vehicle is stopped or driving at low speeds. In one embodiment of the present disclosure, the processor (120, see FIG. 3) of the electronic device (100) receives a signal from the vehicle's speedometer, brake pedal, auto hold, etc. through an electronic control unit (ECU), and can identify whether the vehicle is stopped or driving at a low speed below a preset threshold speed based on the received signal. The processor (120) can perform respiration detection when the vehicle is stopped or driving at low speeds, thereby obtaining a respiration signal in the driver's physical and / or mental rest state. For example, the processor (120) may obtain a breathing signal including at least one of a breathing pattern, a breathing rate, and a breathing expansion pattern for each stopping time of the vehicle, and may calculate breaths per minute by dividing the number of peaks of breathing according to the measured time during the stopping time of the vehicle (e.g., 20 seconds to several minutes) by 1 minute. The processor (120) may obtain the calculated breathing rate per minute as a breathing signal of stable breathing.

[0160] In step S1420, the electronic device (100) monitors the driver's breathing signal in real time while the vehicle is driving. The specific method by which the electronic device (100) acquires the driver's breathing signal and monitors the breathing signal in real time has been described in detail in FIGS. 2 and 3, and thus, a duplicate description will be omitted.

[0161] Steps S1430 to S1460 are steps that specify step S220 illustrated in FIG. 2. In step S1430, the electronic device (100) calculates a difference between a breathing signal of stable breathing and a breathing signal monitored in real time. In one embodiment of the present disclosure, the processor (120) of the electronic device (100) may calculate a difference between a breathing signal monitored in real time and a reference value, using the number of breaths per minute of stable breathing as a reference value.

[0162] In step S1440, the electronic device (100) compares the calculated difference value with a threshold value (α). In one embodiment of the present disclosure, the threshold value (α) may be preset by the user. For example, the threshold value (α) may be, but is not limited to, the number of breaths per minute.

[0163] If the calculated difference value exceeds the threshold value (α) (step S1450), the electronic device (100) detects abnormal breathing of the driver. In one embodiment of the present disclosure, the processor (120) of the electronic device (100) can detect abnormal breathing such as hyperventilation, acute hyperventilation, hypopnea, temporary apnea, or dyspnea if the calculated difference value exceeds the threshold value (α).

[0164] If the calculated difference value exceeds the threshold value (α) (step S1460), the electronic device (100) detects the driver's normal breathing. If normal breathing is detected, the electronic device (100) returns to step S1420 and monitors the driver's breathing signal in real time while the vehicle is driving.

[0165] The electronic device (100) according to the embodiment illustrated in Fig. 14 monitors the driver's breathing signal acquired in real time, using the driver's stable breathing as a reference value when the vehicle is stopped or driving at low speed, thereby determining whether the driver's breathing rhythm is disturbed compared to the stable breathing, thereby detecting abnormal breathing, and thus can efficiently and accurately detect abnormal breathing.

[0166] FIG. 15a is a diagram illustrating an operation of an electronic device (100) according to one embodiment of the present disclosure to display a breathing guide UI (1500a).

[0167] Referring to FIG. 15a, the electronic device (100) can display a breathing guide UI (1500a) on the instrument panel display (140a) of the vehicle. The breathing guide UI (1500a) is an interface for alleviating abnormal breathing and protecting the driver by providing guide information for self-regulation of the driver's breathing. In the embodiment illustrated in FIG. 15a, the breathing guide UI (1500a) can include a breathing waveform graph (1510) indicating the driver's inhalation / exhalation breathing pattern and the number of breaths per minute, and a graphical user interface (UI) (1520) for guiding the driver's self-breathing.

[0168] In one embodiment of the present disclosure, the graphic UI (1520) may be configured with graphics that provide visual effects for self-breathing, such as inhalation / exhalation, by comparing the driver's breathing signal monitored in real time. For example, the graphic UI (1520) may be configured with a snowman-shaped graphic element that includes a circular graphic element, the diameter of which increases when guiding inhalation, and decreases when guiding exhalation. However, the present invention is not limited thereto.

[0169] FIG. 15b is a diagram illustrating an operation of an electronic device (100) according to one embodiment of the present disclosure to display a breathing guide UI (1500b).

[0170] Referring to FIG. 15b, the electronic device (100) can display a breathing guide UI (1500b) through a Head Up Display (HUD) (140b) projected on a windshield (1500) of a vehicle. In the embodiment illustrated in FIG. 15b, the electronic device (100) may further include a projector (1502) in addition to the components illustrated in FIG. 3. The projector (1502) is an optical engine configured to project an image on the windshield (1500), which is a front window of the vehicle. The projector (1502) is configured to generate light of the image and may be an optical engine including an image panel, an illumination optical system, a projection optical system, and the like. In one embodiment of the present disclosure, a projector (1502) may obtain graphic data (or image data) constituting a breathing guide UI (1500b) from a processor (120, see FIG. 3), generate a virtual image based on the obtained graphic data (or image data), and project light constituting the virtual image output from a light source onto a windshield (1500) through an emission surface. The image projected by the projector (1502) may be displayed through a HUD (140b) on the windshield (1500).

[0171] The breathing guide UI (1500b) is an interface designed to alleviate abnormal breathing and protect the driver by providing guidance information for the driver's self-regulation of breathing. The breathing guide UI (1500b) illustrated in Fig. 15b is identical to the breathing guide UI (1500a) illustrated in Fig. 15a, except that it is displayed via the HUD (140b). Therefore, a duplicate description will be omitted.

[0172] When driving a vehicle, a driver receives approximately 80% to 90% of situational awareness information through his or her eyes. The electronic device (100) according to the embodiment illustrated in FIGS. 15a and 15b displays a breathing guide UI (1500a, 1500b) on the vehicle's instrument panel display (140a) or HUD (140b) so that it does not interfere with driving and can be easily viewed, thereby enabling a driver with limited hands to immediately relax their breathing while driving. Thus, the electronic device (100) according to one embodiment of the present disclosure provides a technical effect that can alleviate abnormal breathing and stress in the driver and prevent the occurrence of road traffic accidents in advance.

[0173] Although not shown in FIGS. 15a and 15b, the electronic device (100) can adjust the brightness and color temperature of the vehicle's instrument panel display (140a) or HUD (140b) based on the breathing signal. In general, the higher the color temperature and the brighter the lighting environment, the higher the physical and mental tension, and the more the arousal effect can be generated. For example, a lighting environment with a color temperature of 3000K and a brightness of 300 lux (lx) is the lighting environment that can most relieve both the low-stress group and the high-stress group, and a lighting environment with a color temperature of 6000K and a brightness of 100 lux, 300 lux, and 600 lux is the lighting environment that relieves the low-stress group. In particular, a lighting environment with a brightness of 100 lux can most relieve stress. In one embodiment of the present disclosure, the memory (130, see FIG. 3) of the electronic device (100) may have information about optimal brightness and color temperature for alleviating driver stress and abnormal breathing stored in advance. When abnormal breathing of the driver is detected, the processor (120, see FIG. 3) of the electronic device (100) according to one embodiment of the present disclosure may load the information about brightness and color temperature stored in the memory (130) to adjust the brightness and color temperature of the vehicle's instrument panel display (140a) or HUD (140b).

[0174] FIG. 16 is a diagram illustrating an operation of an electronic device (100) according to one embodiment of the present disclosure to output a warning UI (1600) when detecting abnormal breathing.

[0175] Referring to FIG. 16, the electronic device (100) can monitor the driver's breathing signal in real time to detect hypopnea or temporary apnea. If hypopnea or temporary apnea is detected, the electronic device (100) can output a warning UI (1600) through the display unit (140). The warning UI (1600) can include, for example, a warning message (1610) for warning the driver of hypopnea and a graphic UI (1620) for executing a breathing guide UI for guiding respiration readjustment to alleviate hypopnea.

[0176] FIG. 17 is a diagram illustrating an operation of an electronic device (100) according to one embodiment of the present disclosure to display a respiratory waveform graph (1710, 1720) representing a driver's thoracic breathing and abdominal breathing and a respiratory guide graphic UI (1730) together.

[0177] Referring to FIG. 17, the electronic device (100) can acquire a driver's breathing signal in real time and display breathing waveform graphs (1710, 1720) on the display unit (140). In one embodiment of the present disclosure, the electronic device (100) can acquire images by photographing both the driver's chest and abdomen through a wide-angle camera, and can acquire thoracic breathing signals and abdominal breathing signals by tracking micro-movements of the chest and abdomen from the acquired images. The electronic device (100) can display a first breathing waveform graph (1710) of the thoracic breathing signal acquired over time and a second breathing waveform graph (1720) of the abdominal breathing signal together with a breathing guide UI (1730).

[0178] The breathing guide graphic UI (1730) may be configured with graphics that provide visual effects for self-breathing, such as inhalation / exhalation, based on thoracic breathing signals and abdominal breathing signals acquired in real time. In the embodiment illustrated in FIG. 17, the breathing guide graphic UI (1730) may include a character (e.g., 'HOLD') that guides to temporarily stop thoracic breathing by the chest and a graphic UI that guides abdominal breathing by the abdomen. The 'graphic UI' is a graphic element that guides inhalation / exhalation, and may be configured with, for example, a snowman-shaped graphic in which the diameter of a circle increases when guiding inhalation and decreases when guiding exhalation. However, the present invention is not limited thereto.

[0179] FIG. 18 is a diagram illustrating an operation of an electronic device (100) according to one embodiment of the present disclosure to display the amount of change in each of a driver's thoracic breathing and abdominal breathing.

[0180] Referring to FIG. 18, the electronic device (100) can display information on the change in thoracic respiration (1810) and abdominal respiration (1820) over time on the display unit (140). The change in thoracic respiration (1810) is a graphic UI that represents the change in thoracic respiration signal obtained by the movement of the driver's chest over time, and may include, for example, information on the change in breaths per minute of thoracic respiration. The change in abdominal respiration (1820) is a graphic UI that represents the change in diaphragmatic respiration signal obtained by the movement of the driver's abdomen over time, and may include, for example, information on the change in breaths per minute of abdominal respiration. In the embodiment illustrated in FIG. 18, the thoracic breathing change amount (1810) may output information indicating that the driver's thoracic breathing has decreased from 20 bpm to 15 bpm, and the abdominal breathing change amount (1820) may output information indicating that the driver's abdominal breathing has increased from 13 bpm to 18 bpm.

[0181] In one embodiment of the present disclosure, the electronic device (100) may display changes in thoracic breathing (1810) and abdominal breathing (1820) along with a breathing guide graphic UI (1830). The breathing guide graphic UI (1830) may include information regarding changes in thoracic breathing and abdominal breathing. In the embodiment illustrated in FIG. 18, the breathing guide graphic UI (1830) may output information indicating that thoracic breathing has decreased by 25% and abdominal breathing has increased by 38%.

[0182] Diaphragmatic breathing by the abdomen has the effect of relieving stress and high blood pressure by strengthening the parasympathetic nerve compared to thoracic breathing. It is also known that research results have shown that if diaphragmatic breathing is performed for more than 80 minutes every day, systolic blood pressure can be reduced from 174 mmHg to 146 mmHg and diastolic blood pressure can be reduced from 97 mmHg to 81 mmHg. The electronic device (100) according to the embodiment shown in FIGS. 17 and 18 displays respiratory waveform graphs (1710, 1720) of not only thoracic breathing by the chest but also abdominal breathing by the abdomen, and provides information on the amount of change in each of thoracic breathing and abdominal breathing over time, thereby controlling the balance between thoracic breathing and abdominal breathing, thereby providing technical effects such as relieving the driver's stress and reducing high blood pressure.

[0183] FIG. 19 is a flowchart illustrating a method for protecting a driver when hyperventilation is detected by an electronic device (100) according to one embodiment of the present disclosure.

[0184] FIG. 20 is a drawing for explaining an operation of an electronic device (100) according to one embodiment of the present disclosure to protect a driver when hyperventilation is detected.

[0185] Hereinafter, with reference to FIGS. 19 and 20, the function and / or operation of the electronic device (100) to protect the driver when hyperventilation is detected will be described in detail.

[0186] Referring to step S1910 of FIG. 19, the electronic device (100) monitors a breathing signal to detect hyperventilation. Referring also to operation ① of FIG. 20, the electronic device (100) captures a photograph of the driver to acquire an image, acquires a breathing signal (2000) from the acquired image, and monitors the breathing signal (2000) in real time to detect abnormal breathing such as hyperventilation. The specific method by which the electronic device (100) detects abnormal breathing is the same as that described in FIGS. 2 and 3, and therefore, redundant descriptions are omitted.

[0187] In step S1920 of FIG. 19, the electronic device (100) outputs a warning message indicating an emergency situation. Referring also to operation ② of FIG. 20, the electronic device (100) may output a warning message (2010) on the display unit (140) that warns of abnormal breathing and guides automatic stop.

[0188] In step S1930 of FIG. 19, if there is no driver interaction with the warning message, the electronic device (100) controls the driving of the vehicle to move to the shoulder and stop the vehicle. Referring also to FIG. 20, the warning message (2010) may include a cancel button UI that releases the warning message output and terminates the warning situation. In one embodiment of the present disclosure, if a driver input to the cancel button UI is detected, the electronic device (100) may terminate the warning situation and not display the warning message (2010). However, if no driver input is received to the cancel button UI or no interaction is detected, the electronic device (100) may control the driving of the vehicle to stop the vehicle on the shoulder. Referring to operation ③ of FIG. 20, the electronic device (100) can capture the shoulder parking situation of a vehicle using an external camera and display the acquired external environment image (2020) through the display unit (140).

[0189] Referring again to FIG. 19, in step S1940, the electronic device (100) transmits an emergency call for a rescue request. In one embodiment of the present disclosure, the electronic device (100) further includes a communication interface (160, see FIG. 24) and can transmit an emergency call requesting rescue to an emergency rescue organization (e.g., a fire department, a medical institution, etc.) through the communication interface (160). Referring also to operation ④ of FIG. 20, the electronic device (100) can display an emergency call UI (2030) on the display unit (140).

[0190] In the embodiment illustrated in FIGS. 19 and 20, the electronic device (100) provides a technical effect that can protect the driver and prevent traffic accidents in advance by immediately moving the vehicle to the shoulder and stopping it when the driver's hyperventilation or temporary hyperventilation is detected and the driver loses control of the vehicle.

[0191] FIG. 21 is a block diagram illustrating an operation of controlling devices included in a vehicle (1000) when an electronic device (100) according to one embodiment of the present disclosure detects abnormal breathing of a driver.

[0192] Referring to FIG. 21, the vehicle (1000) may include an ECU (1010), a seat belt (1020), a steering wheel (1030), interior lighting (1040), ambient lighting (1050), and a seat (1060). FIG. 21 only illustrates configurations for explaining an operation of controlling devices of the vehicle (1000) when the electronic device (100) detects abnormal breathing of the driver, and the devices included in the vehicle (1000) are not limited to those illustrated in FIG. 21.

[0193] An ECU (electronic control unit) (1010) is an electronic device that controls the functions and / or operations of not only the engine, automatic transmission, and other drivetrains of a vehicle, but also devices within the vehicle, such as the braking system and steering system. A connector (C) is a hardware device that connects the electronic device (100) and devices within the vehicle (1000). In one embodiment of the present disclosure, the electronic device (100) may be physically and / or electrically interconnected with the ECU (1010) through the connector (C). The electronic device (100) may transmit a control signal to the ECU (1010) through the connector (C).

[0194] When the processor (120, see FIG. 3) of the electronic device (100) detects abnormal breathing of the driver, it transmits a control signal to the ECU (1010), and controls various devices of the vehicle through the ECU (1010) to alleviate the driver's abnormal breathing and reduce stress and blood pressure, thereby protecting the driver. For example, when the driver's abnormal breathing is detected, the processor (120) can relax the pressure of the seat belt (1020) through the ECU (1010). For example, when the processor (120) detects abnormal breathing of the driver, it can also provide a warning notification to the driver by applying a pressure signal to the seat belt (1020).

[0195] In one embodiment of the present disclosure, the processor (120) may control the steering wheel (1030) to vibrate when abnormal breathing of the driver is detected, thereby providing steering wheel vibration feedback. For example, the processor (120) may control the steering wheel (1030) to vibrate according to the timing of the driver's inhalation / exhalation, thereby providing breathing guidance to the driver.

[0196] In one embodiment of the present disclosure, the processor (120) can adjust the color temperature and brightness of the interior lighting (1040) and the ambient light (1050) when abnormal breathing of the driver is detected. Generally, a lighting environment with a color temperature of 3000K and a brightness of 300 lux (lx) is a lighting environment that can most alleviate both the low-stress group and the high-stress group, and a lighting environment with a color temperature of 6000K and a brightness of 100 lux, 300 lux, and 600 lux is a lighting environment that alleviates the low-stress group. For example, the processor (120) can adjust the color temperature of the interior lighting (1040) and the ambient light (1050) to 3000K and the brightness to 300 lux to alleviate the driver's abnormal breathing and lower blood pressure.

[0197] In one embodiment of the present disclosure, the processor (120) can control the heating or cooling of the seat (1060) based on the driver's breathing signal. For example, if the driver's breathing signal is monitored in real time and hypoventilation symptoms are detected, the processor (120) can control the heating wire of the seat (1060) to increase the temperature of the seat (1060). For example, if the driver's hyperventilation symptoms are detected, the processor (120) can control the fan included in the seat (1060) to lower the temperature of the seat (1060) to provide a cool seating environment.

[0198] When the body temperature drops below normal body temperature (e.g., 36.5°C), the respiratory and circulatory nervous system functions may slow down, and driving alertness may decrease. In the embodiment illustrated in FIG. 21, the electronic device (100) controls the heating wire of the seat (1060) to increase the temperature of the seat (1060) when the body temperature drops due to hypopnea symptoms, thereby increasing the driver's body temperature and improving hypopnea symptoms.

[0199] When experiencing symptoms of hyperventilation, the driver may experience shortness of breath and a rise in body temperature. In the embodiment illustrated in FIG. 21, the electronic device (100) controls the fan of the seat (1060) to lower the temperature of the seat (1060) when the body temperature rises due to hyperventilation, thereby lowering the driver's body temperature and stabilizing breathing.

[0200] FIG. 22 is a drawing for explaining an operation performed by an electronic device (100) using artificial intelligence technology according to one embodiment of the present disclosure.

[0201] Specifically, at least one of the following operations performed by the electronic device (100): i) detecting movement of at least one of the chest and abdomen caused by the driver's breathing from an image acquired through a camera (110, see FIGS. 1 and 3) to acquire a driver's breathing signal, ii) monitoring the breathing signal in real time to detect abnormal breathing including at least one of hyperventilation, hypopnea, temporary apnea, and dyspnea, and iii) displaying a breathing guide UI (user interface) that provides a breathing exercise guide to alleviate the abnormal breathing by reorganizing the driver's breathing when abnormal breathing is detected, may be performed using artificial intelligence (AI) technology that performs calculations through a neural network.

[0202] Artificial intelligence technology (hereinafter referred to as “AI technology”) is a technology that obtains the desired result by analyzing and / or processing input data, such as classification, based on operations through a neural network.

[0203] These AI technologies can be implemented using algorithms. Here, an algorithm or set of algorithms for implementing AI technologies is called a neural network. Here, a neural network can receive input data, perform operations for the aforementioned analysis and / or classification, and output result data. In order for a neural network to accurately output result data corresponding to the input data, the neural network needs to be trained. Here, "training" can mean training the neural network to discover or acquire methods for analyzing input data, classifying input data, and / or extracting features necessary for generating result data from input data. Specifically, through the learning process, the neural network can learn training data (e.g., multiple different images) and optimize the weight values ​​within the neural network. Then, by processing the input data through the neural network with the optimized weight values, the desired result is output.

[0204] A neural network can be classified as a deep neural network when the number of hidden layers, which are internal layers that perform calculations, is multiple, that is, when the depth of the neural network that performs calculations increases. Neural networks include, but are not limited to, convolutional neural networks, recurrent neural networks, restricted Boltzmann machines, deep belief networks, bidirectional recurrent deep neural networks, and deep Q-networks. In addition, neural networks can be subdivided. For example, a convolutional neural network can be subdivided into a deep convolutional neural network (D-CNN) or a Capsnet neural network (not shown).

[0205] An "AI model" may refer to a neural network comprising at least one layer that receives input data and operates to output a desired result. Furthermore, an "AI model" may refer to an algorithm that performs operations through a neural network to output a desired result, a set of multiple algorithms, a processor for executing an algorithm (or a set of algorithms), software for executing an algorithm (or a set of algorithms), or hardware for executing an algorithm (or a set of algorithms).

[0206] At least one of the following operations may be performed based on an AI model: i) an operation of detecting movement of at least one of the chest and abdomen due to the driver's breathing from an image acquired through the camera (110) to acquire a driver's breathing signal; ii) an operation of monitoring the breathing signal in real time to detect abnormal breathing including at least one of hyperventilation, hypoventilation, temporary apnea, and dyspnea; and iii) an operation of displaying a breathing guide UI (user interface) that provides a breathing exercise guide to alleviate the abnormal breathing by reorganizing the driver's breathing when abnormal breathing is detected.

[0207] Referring to FIG. 22, a neural network (2200) can be trained by receiving training data. Then, the trained neural network (2200) receives input data (2210) as an input terminal (2220), and the input terminal (2220), a hidden layer (2230), and an output terminal (2240) can perform operations to analyze the input data (2210) and data transmitted from the previous layer to output output data (2250). In FIG. 22, the hidden layer (2230) is illustrated as being a single layer, but this is merely an example, and the hidden layer (2230) may be composed of multiple layers.

[0208] In the disclosed embodiment, the neural network (2200) can be trained to i) divide a body part of a driver included in an image acquired through a camera (110) into a plurality of regions, ii) detect movement of the divided plurality of regions to obtain a respiratory signal including at least one of a respiratory waveform and a respiratory intensity from each of the plurality of regions, iii) select an optimal region for monitoring the respiratory signal of the driver among the plurality of regions based on at least one of a pattern consistency of a respiratory waveform and a change amount of a respiratory intensity, and iv) obtain a respiratory signal of the selected region in real time.

[0209] In the disclosed embodiment, the neural network (2200) can be trained to i) merge respiratory signals obtained from each of a plurality of regions, and ii) monitor the merged respiratory signals in real time to detect abnormal breathing.

[0210] In the disclosed embodiment, the neural network (2200) can be trained to detect abnormal respiration by i) acquiring a respiration signal regarding a driver's rest respiration when the vehicle is stopped or when driving at a low speed below a preset threshold speed, and ii) determining whether a difference between the respiration signal of rest respiration and the respiration signal of the driver monitored in real time exceeds a preset threshold.

[0211] In the disclosed embodiment, the neural network (2200) can be trained to display a respiratory waveform of a respiratory signal acquired in real time along with a respiratory movement guide.

[0212] In the disclosed embodiment, the neural network (2200) can be trained to i) display together the respiratory waveforms of the thoracic respiratory signal obtained from the driver's chest movement among the respiratory signals and the abdominal respiratory signal obtained from the driver's abdominal movement, and ii) display information regarding the real-time change amount of each of the thoracic respiratory signal and the abdominal respiratory signal.

[0213] In the disclosed embodiment, the neural network (2200) can be trained to i) determine when to display a breathing guide UI based on at least one of whether the vehicle is performing autonomous driving and whether the driver is involved in driving the vehicle, and ii) display the breathing guide UI at the determined time.

[0214] In the disclosed embodiment, the neural network (2200) can be trained to i) output a warning message indicating an emergency situation when hyperventilation is detected during abnormal breathing, ii) control the driving of the vehicle to move the vehicle to the shoulder and stop if there is no driver interaction with the output warning message, and iii) transmit an emergency call for rescue.

[0215] In the disclosed embodiment, data or program code related to a neural network (2200) that performs at least one of the following operations: i) detecting movement of at least one of the chest and abdomen caused by the driver's breathing from an image acquired through the camera (110) to acquire a driver's breathing signal, ii) monitoring the breathing signal in real time to detect abnormal breathing including at least one of hyperventilation, hypopnea, temporary apnea, and dyspnea, and iii) displaying a breathing guide UI (user interface) that provides a breathing exercise guide to alleviate the abnormal breathing by reorganizing the driver's breathing when abnormal breathing is detected, is stored in a memory (130, see FIG. 3), and learning using the neural network (2200) can be performed by a processor (120, see FIG. 3). In this case, the processor (120) may include an artificial intelligence-only processor such as an NPU (Neural Processing Unit).

[0216] In the disclosed embodiment, data or program code related to a neural network (2200) that performs at least one of the following operations: i) detecting movement of at least one of the chest and abdomen caused by the driver's breathing from an image acquired through the camera (110) to acquire a driver's breathing signal, ii) monitoring the breathing signal in real time to detect abnormal breathing including at least one of hyperventilation, hypopnea, temporary apnea, and dyspnea, and iii) displaying a breathing guide UI (user interface) that provides a breathing exercise guide to alleviate the abnormal breathing by reorganizing the driver's breathing when abnormal breathing is detected, is stored in the memory (130), and learning using the neural network (2200) can be performed by the processor (120).

[0217] However, it is not limited thereto, and the neural network (2200) that performs at least one of the following operations: i) detecting movement of at least one of the chest and abdomen caused by the driver's breathing from an image acquired through the camera (110) to acquire the driver's breathing signal, ii) monitoring the breathing signal in real time to detect abnormal breathing including at least one of hyperventilation, hypopnea, temporary apnea, and dyspnea, and iii) displaying a breathing guide UI (user interface) that provides a breathing exercise guide to alleviate the abnormal breathing by reorganizing the driver's breathing when abnormal breathing is detected, may be implemented in a separate device (not shown) or processor (not shown) distinct from the electronic device (100).

[0218] The computation via the aforementioned neural network (2200) may also be performed by a server (200, see FIGS. 23 and 24) capable of communicating with an electronic device (100) via a wireless communication network according to one embodiment. Communication between the electronic device (100) and the server (200) is described with reference to FIGS. 23 and 24.

[0219] FIG. 23 is a drawing illustrating an electronic device (100) according to the disclosed embodiment that operates in conjunction with a server (200).

[0220] Figure 24 is a drawing for explaining Figure 23 in detail.

[0221] The electronic device (100) may include a camera (110), a processor (120), a memory (130), a display unit (140), and a speaker (150). Referring to FIG. 24, the electronic device (100) may further include a communication interface (160). The camera (110), the processor (120), the memory (130), the display unit (140), and the speaker (150) illustrated in FIG. 24 are the same as the camera (110, see FIG. 3), the processor (120, see FIG. 3), the memory (130, see FIG. 3), the display unit (140, see FIG. 3), and the speaker (150, see FIG. 3) illustrated and described in FIG. 3, and therefore, a duplicate description will be omitted.

[0222] The server (200) can transmit and receive data with the electronic device (100) through a communication network (300) and process the data.

[0223] Referring to FIGS. 23 and 24 together, the server (200) may include a communication interface (210) for communicating with an electronic device (100), a processor (220) for performing at least one instruction, and a memory (230).

[0224] The server (200) may train an AI model and store the trained AI model. Then, the server (200) may perform at least one of the following operations using the trained AI model: i) detecting movement of at least one of the chest and abdomen caused by the driver's breathing from an image acquired through the camera (110) to acquire the driver's breathing signal; ii) monitoring the breathing signal in real time to detect abnormal breathing including at least one of hyperventilation, hypopnea, temporary apnea, and dyspnea; and iii) displaying a breathing guide UI (user interface) that provides a breathing exercise guide to alleviate the abnormal breathing by reorganizing the driver's breathing when abnormal breathing is detected.

[0225] In general, the electronic device (100) may be limited in the storage capacity of the memory (130, see FIG. 3), the computational processing speed of the processor (120, see FIG. 3), the ability to collect learning data sets, etc., compared to the server (200). Therefore, operations requiring storage of large amounts of data and large amounts of computation may be performed in the server (200), and then the necessary data and / or AI models may be transmitted to the electronic device (100) via a communication network. The electronic device (100) may receive and use the necessary data and / or AI models via the server (200), thereby performing the necessary operations quickly and easily, even without a processor having large amounts of memory and fast computational capabilities.

[0226] In the disclosed embodiment, the server (200) may include a neural network (2200) as described in FIG. 22.

[0227] In Fig. 24, the components of the server (200) will be described in detail. Referring to Fig. 24, the server (200) may include a communication interface (210), a processor (220), and a memory (230).

[0228] The communication interface (210) communicates with an external device via a wireless communication network. Here, the external device (not shown) may include a server capable of performing at least one operation required by the electronic device (100) or transmitting data required by the electronic device (100).

[0229] The communication interface (210) includes at least one communication module, such as a short-range communication module, a wired communication module, a mobile communication module, a broadcast reception module, etc. Here, at least one communication module means a communication module capable of transmitting and receiving data through a network that follows a communication standard, such as a tuner that performs broadcast reception, Bluetooth, Wi-Fi, Wibro (Wireless broadband), WiMAX (World Interoperability for Microwave Access), CDMA, WCDMA, the Internet, 3G, 4G, 5G, and / or a communication method using millimeter waves (mmWave).

[0230] The mobile communication module included in the communication interface (210) can communicate with another device (e.g., electronic device (100)) located remotely through a communication network according to communication standards such as 3G, 4G (LTE), and / or 5G. Here, the communication module that communicates with another device located remotely may be referred to as a 'remote communication module'. In one embodiment of the present disclosure, the electronic device (100) further includes a communication interface (160), and the communication interface (210) of the server (200) can transmit and receive data with the communication interface (160) of the electronic device (100) wired or wirelessly.

[0231] The processor (220) controls the overall operation of the server (200). For example, the processor (220) can perform required operations by executing at least one instruction and at least one program of the server (200).

[0232] The memory (230) can store at least one instruction, program, or data required for the server (200) to perform a predetermined operation. In addition, the memory (230) can store data required for the server (200) to perform operations according to a neural network.

[0233] In the disclosed embodiment, the server (200) may store the neural network (2200) described in FIG. 22. The neural network (2200) may be stored in at least one of the processor (220) and the memory (230). The neural network (2200) included in the server (200) may be a trained neural network.

[0234] Additionally, the server (200) can transmit a neural network (2200, see FIG. 22) for which learning has been completed to the communication interface (160) of the electronic device (100) through the communication interface (210). The electronic device (100) can obtain and store the neural network (2200) for which learning has been completed, and obtain desired output data through the neural network (2200).

[0235] The present disclosure provides a method for an electronic device (100) mounted in a vehicle (1000) to detect the respiration of a driver of the vehicle (1000) and provide a breathing guide for driver protection. An operating method of an electronic device (100) according to an embodiment of the present disclosure may include a step (S210) of detecting movement of at least one of the chest and abdomen caused by the driver's respiration from an image acquired through a camera (110) to acquire a driver's respiration signal. An operating method of an electronic device (100) according to an embodiment of the present disclosure may include a step (S220) of monitoring the acquired respiration signal in real time to detect abnormal respiration including at least one of hyperventilation, hypopnea, temporary apnea, and dyspnea. An operating method of an electronic device (100) according to one embodiment of the present disclosure may include a step (S230) of displaying a breathing guide UI (user interface) that provides a breathing exercise guide to alleviate abnormal breathing by reorganizing the driver's breathing when abnormal breathing is detected.

[0236] In one embodiment of the present disclosure, the step of obtaining the breathing signal (S210) may include a step of dividing a body part of a driver included in an image obtained through a camera (110) into a plurality of regions (S1010); and a step of detecting movement of the divided plurality of regions to obtain a breathing signal including at least one of a breathing waveform and a breathing intensity from each of the plurality of regions (S1020). The step of obtaining the breathing signal (S210) may include a step of selecting an optimal region for monitoring the breathing signal of the driver among the plurality of regions based on at least one of a pattern consistency of a breathing waveform and a change amount of a breathing intensity (S1030); and a step of obtaining the breathing signal of the selected region in real time (S1040).

[0237] In one embodiment of the present disclosure, the step of acquiring the respiration signal (S210) may include a step of merging respiration signals acquired from each of a plurality of regions. The step of detecting abnormal respiration (S220) may include a step of monitoring the merged respiration signal in real time to detect abnormal respiration.

[0238] In one embodiment of the present disclosure, the step of obtaining the breathing signal (S210) may include a step of obtaining motion information due to vibration of the vehicle (1000) or body movement of the driver (S1210). The step of obtaining the breathing signal (S210) may further include a step of identifying a time period in which motion information is detected among a breathing waveform according to the passage of time of the breathing signal (S1220); and a step of correcting the breathing signal of the breathing waveform in the identified period (S1230).

[0239] In one embodiment of the present disclosure, the step of obtaining the respiration signal (S210) may include a step of obtaining a respiration signal related to the driver's rest respiration when the vehicle (1000) is stopped or when driving at a low speed below a preset threshold speed (S1410). The step of detecting the abnormal respiration (S220) may include a step of calculating a difference value between the respiration signal of the rest respiration and the respiration signal of the driver monitored in real time (S1430); and a step of detecting the abnormal respiration by determining whether the calculated difference value exceeds a preset threshold value (S1440, S1450).

[0240] In one embodiment of the present disclosure, in the step of displaying the breathing guide UI (S230), the electronic device (100) can display the breathing guide UI on the instrument cluster of the vehicle (1000).

[0241] In one embodiment of the present disclosure, in the step of displaying the breathing guide UI (S230), the electronic device (100) can display a breathing waveform of a breathing signal acquired in real time together with a breathing exercise guide.

[0242] In one embodiment of the present disclosure, the step (S230) of displaying the breathing guide UI may include a step of displaying respiratory waveforms of a thoracic breathing signal obtained from the driver's chest movement among the breathing signals and an abdominal breathing signal obtained from the driver's abdominal movement. The step (S230) of displaying the breathing guide UI may include a step of displaying information regarding real-time changes in each of the thoracic breathing signal and the abdominal breathing signal.

[0243] In one embodiment of the present disclosure, the operating method of the electronic device (100) may further include a step (S1910) of outputting a warning message indicating an emergency situation when hyperventilation is detected during abnormal breathing. The operating method of the electronic device (100) may further include a step (S1920) of controlling the driving of the vehicle (1000) to move the vehicle (1000) to the shoulder of the road and stop it if there is no driver interaction corresponding to the outputted warning message. The operating method of the electronic device (100) may further include a step (S1930) of sending an emergency call for a rescue request.

[0244] The present disclosure provides an electronic device (100) in a vehicle (1000) that detects a driver's breathing and provides guidance for driver protection. The electronic device (100) according to one embodiment of the present disclosure may include a display unit (140), a camera (110) that captures a driver's breathing to obtain an image, at least one processor (120) including a processing circuit, and a memory (130) that stores at least one instruction. The one or more instructions are individually or collectively executed by at least one processor (120), so that the electronic device (100) can detect movement of at least one of the chest and abdomen caused by the driver's breathing from an image obtained through the camera (110), thereby obtaining a driver's breathing signal. As the one or more commands are individually or collectively executed by at least one processor (120), the electronic device (100) can monitor a breathing signal in real time to detect abnormal breathing including at least one of hyperventilation, hypoventilation, temporary apnea, and dyspnea. As the one or more commands are individually or collectively executed by at least one processor (120), the electronic device (100) can display a breathing guide UI (user interface) on the display unit (140) that provides a breathing exercise guide to alleviate the abnormal breathing by reorganizing the driver's breathing when abnormal breathing is detected.

[0245] In one embodiment of the present disclosure, the one or more commands are individually or collectively executed by at least one processor (120), so that the electronic device (100) can divide a body part of a driver included in an image acquired through a camera (110) into a plurality of regions, detect movement of the divided plurality of regions, and obtain a respiration signal including at least one of a respiration waveform and a respiration intensity from each of the plurality of regions. The one or more commands are individually or collectively executed by at least one processor (120), so that the electronic device (100) can select an optimal region for monitoring a respiration signal of the driver among the plurality of regions based on at least one of a pattern consistency of a respiration waveform and a change amount of a respiration intensity, and obtain a respiration signal of the selected region in real time.

[0246] In one embodiment of the present disclosure, the one or more commands are individually or collectively executed by at least one processor (120), so that the electronic device (100) can merge breathing signals obtained from each of a plurality of regions and monitor the merged breathing signal in real time to detect abnormal breathing.

[0247] In one embodiment of the present disclosure, the one or more commands are individually or collectively executed by at least one processor (120), so that the electronic device (100) can obtain motion information due to vibration of the vehicle (1000) or body movement of the driver, and identify a time period in which motion information is detected among the respiratory waveform according to the passage of time of the respiratory signal. The one or more commands are individually or collectively executed by at least one processor (120), so that the electronic device (100) can correct the respiratory signal of the respiratory waveform in the identified period.

[0248] In one embodiment of the present disclosure, the one or more commands are individually or collectively executed by at least one processor (120), so that the electronic device (100) can obtain a respiration signal related to the driver's rest respiration when the vehicle (1000) is stopped or driving at a low speed below a preset threshold speed. The one or more commands are individually or collectively executed by at least one processor (120), so that the electronic device (100) can detect abnormal respiration by calculating a difference value between the respiration signal of the rest respiration and the respiration signal of the driver monitored in real time, and determining whether the calculated difference value exceeds a preset threshold value.

[0249] In one embodiment of the present disclosure, the display unit (140) may be configured as an instrument cluster display within a vehicle (1000). By individually or collectively executing one or more of the above commands by at least one processor (120), the electronic device (100) may display a breathing guide UI on the instrument cluster display.

[0250] In one embodiment of the present disclosure, the display unit (140) can display a respiratory waveform of a respiratory signal acquired in real time together with a respiratory movement guide.

[0251] In one embodiment of the present disclosure, the display unit (140) can display respiratory waveforms of a thoracic respiratory signal obtained from the driver's chest movement and an abdominal respiratory signal obtained from the driver's abdominal movement together among the respiratory signals. The display unit (140) can display information regarding real-time changes in each of the thoracic respiratory signal and the abdominal respiratory signal.

[0252] In one embodiment of the present disclosure, the one or more commands are individually or collectively executed by at least one processor (120), so that the electronic device (100) can determine a time point for displaying the breathing guide UI based on at least one of whether the vehicle (1000) is performing autonomous driving and whether the driver is involved in driving the vehicle. The one or more commands are individually or collectively executed by at least one processor (120), so that the electronic device (100) can control the display unit (140) to display the breathing guide UI at the determined time point.

[0253] In one embodiment of the present disclosure, when the one or more commands are individually or collectively executed by at least one processor (120), the electronic device (100) can output a warning message indicating an emergency situation through the display unit (140) when hyperventilation is detected during abnormal breathing. When the one or more commands are individually or collectively executed by at least one processor (120), the electronic device (100) can control the driving of the vehicle (1000) to move the vehicle (1000) to the shoulder and stop the vehicle if there is no driver interaction corresponding to the output warning message. The electronic device (100) may further include a communication interface (160). By individually or collectively executing one or more of the above commands by at least one processor (120), the electronic device (100) can control the communication interface (160) to send an emergency call for a rescue request.

[0254] The present disclosure provides a computer program product including a computer-readable storage medium. The storage medium may include instructions readable by an electronic device (100) so that the electronic device (100) performs the following operations: detecting movement of at least one of the chest and abdomen due to breathing of a driver of a vehicle (1000) from an image acquired through a camera (110) to acquire a breathing signal of the driver; monitoring the acquired breathing signal in real time to detect abnormal breathing including at least one of hyperventilation, hypopnea, temporary apnea, and dyspnea; and displaying a breathing guide UI (user interface) that provides a breathing exercise guide to alleviate abnormal breathing by reorganizing the driver's breathing when abnormal breathing is detected.

[0255] The program executed by the electronic device (100) described herein may be implemented as hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by any system capable of executing computer-readable instructions.

[0256] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to do a desired thing or may independently or collectively command a processing device to do a desired thing.

[0257] Software may be implemented as a computer program containing instructions stored on a computer-readable storage medium. Examples of computer-readable storage media include magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disks, hard disks, etc.) and optical readable media (e.g., CD-ROMs, DVDs (Digital Versatile Discs)). The computer-readable storage media may be distributed across network-connected computer systems, so that computer-readable code may be stored and executed in a distributed manner. The media may be readable by a computer, stored in a memory, and executed by a processor.

[0258] A computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium does not contain signals and is tangible, but does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0259] Additionally, programs according to the embodiments disclosed herein may be provided as part of a computer program product. The computer program product may be traded as a commodity between sellers and buyers.

[0260] A computer program product may include a software program, a computer-readable storage medium having the software program stored thereon. For example, a computer program product may be sold by a manufacturer of an electronic device or an electronic market (e.g., Samsung Galaxy Store). TM , Google Play Store TM) may include a product in the form of a software program (e.g., a downloadable application) that is distributed electronically. For electronic distribution, at least a portion of the software program may be stored in a storage medium or temporarily created. In this case, the storage medium may be a cloud server of a manufacturer of a vehicle (1000), a server of a manufacturer of an electronic device (100), a server of an electronic market, or a storage medium of a relay server that temporarily stores the software program.

[0261] In a system comprising an electronic device (100), a server (200, see FIGS. 23 and 24), and another electronic device, the computer program product may include a storage medium of the server (200) or a storage medium of the other electronic device. Alternatively, if there is a third device (e.g., a mobile device such as a smart phone) that is communicatively connected to the electronic device (100), the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the electronic device (100) to another electronic device or the third device, or from the third device to the electronic device (100).

[0262] In this case, one of the electronic device (100), another electronic device, and a third device (e.g., a mobile device such as a smartphone) may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the electronic device (100), another electronic device, and a third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0263] For example, the electronic device (100) may execute a computer program product stored in a memory (130, see FIG. 3) to control another electronic device that is in communication with the electronic device (100) to perform a method according to the disclosed embodiments.

[0264] As another example, a third device may execute a computer program product to control an electronic device in communication with the third device to perform a method according to the disclosed embodiment.

[0265] When a third device executes a computer program product, the third device may download the computer program product from the electronic device (100) and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a pre-loaded state to perform the method according to the disclosed embodiments.

[0266] Although the embodiments described above have been described with limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components such as the described computer system or modules are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

Claims

1. A method for providing a breathing guide for driver protection by detecting the respiration of a driver of a vehicle (1000) by an electronic device (100) mounted in the vehicle (1000), A step (S210) of obtaining a breathing signal of the driver by detecting movement of at least one of the chest and abdomen caused by the driver's breathing from an image acquired through a camera (110); A step (S220) of monitoring the acquired breathing signal in real time to detect abnormal breathing including at least one of hyperventilation, hypopnea, temporary apnea, and dyspnea; and Step (S230) of displaying a breathing guide UI (user interface) that provides a breathing exercise guide to alleviate the abnormal breathing by reorganizing the driver's breathing when the abnormal breathing is detected; A method comprising:

2. In paragraph 1, The step (S210) of obtaining the above breathing signal is: A step (S1010) of dividing the driver's body part included in the image acquired through the above camera (110) into multiple regions; A step (S1020) of detecting movement of the divided plurality of regions and obtaining a respiratory signal including at least one of a respiratory waveform and a respiratory intensity from each of the plurality of regions; A step (S1030) of selecting an optimal region for monitoring the driver's breathing signal among the plurality of regions based on at least one of the pattern consistency of the breathing waveform and the amount of change in the breathing intensity; and Step of acquiring breathing signals of the selected area in real time (S1040); A method comprising:

3. In either of paragraphs 1 or 2, The step (S210) of obtaining the above breathing signal is: Step (S1210) of obtaining motion information due to vibration of the vehicle (1000) or body movement of the driver; Step (S1220) of identifying a time period in which the motion information is detected among the respiratory waveforms according to the time flow of the respiratory signal; and A step of correcting the respiratory signal of the respiratory waveform in the identified section (S1230); A method further comprising:

4. In any one of clauses 1 to 3, The step (S230) of displaying the above breathing guide UI is as follows: A method for displaying the breathing guide UI on the instrument cluster of the vehicle (1000).

5. In any one of clauses 1 to 4, The step (S230) of displaying the above breathing guide UI is as follows: A method for displaying a respiratory waveform of the respiratory signal acquired in real time together with the respiratory movement guide.

6. In any one of clauses 1 to 5, The step (S230) of displaying the above breathing guide UI is as follows: A step of displaying together the respiratory waveforms of the thoracic respiratory signal obtained from the driver's chest movement and the abdominal respiratory signal obtained from the driver's abdominal movement among the above respiratory signals; and A step of displaying information regarding real-time changes in each of the thoracic breathing signal and the abdominal breathing signal; A method comprising:

7. In any one of clauses 1 to 6, Step (S1910) of outputting a warning message indicating an emergency situation when hyperventilation is detected during the above abnormal breathing; Step (S1920) of controlling the driving of the vehicle (1000) to move the vehicle (1000) to the shoulder and stop it when there is no interaction of the driver corresponding to the warning message outputted above; and Step of sending an emergency call for rescue request (S1930); A method further comprising:

8. In an electronic device (100) in a vehicle (1000) that detects the driver's breathing and provides guidance for driver protection, Display section (140); A camera (110) for capturing an image by photographing the driver; At least one processor (120) comprising a processing circuit; and A memory (130) storing one or more instructions; Including, The electronic device (100) is configured such that the one or more instructions are individually or collectively executed by the at least one processor (120). Obtaining the driver's breathing signal by detecting movement of at least one of the chest and abdomen caused by the driver's breathing from the image acquired through the camera (110), Monitor the acquired breathing signal in real time to detect abnormal breathing including at least one of hyperventilation, hypopnea, transient apnea, and dyspnea, An electronic device (100) that displays a breathing guide UI (user interface) on the display unit (140) to provide a breathing exercise guide to alleviate the abnormal breathing by reorganizing the driver's breathing when the abnormal breathing is detected.

9. In paragraph 8, The electronic device (100) is configured such that the one or more instructions are individually or collectively executed by the at least one processor (120). Divide the driver's body part included in the image acquired through the above camera (110) into multiple regions, By detecting the movement of the above divided plurality of regions, a respiratory signal including at least one of a respiratory waveform and a respiratory intensity is obtained from each of the plurality of regions, Selecting an optimal region for monitoring the driver's breathing signal among the plurality of regions based on at least one of the pattern consistency of the breathing waveform and the amount of change in the breathing intensity; An electronic device (100) that acquires breathing signals of the selected area in real time.

10. In either of paragraphs 8 or 9, The electronic device (100) is configured such that the one or more instructions are individually or collectively executed by the at least one processor (120). Obtaining motion information due to vibration of the vehicle (1000) or body movement of the driver, Identify the time period in which the motion information is detected among the respiratory waveforms according to the time flow of the above respiratory signal, An electronic device (100) for correcting a respiratory signal of the respiratory waveform in the identified section.

11. In any one of clauses 8 to 10, The above display unit (140) is configured as an instrument cluster within the vehicle (1000), The electronic device (100) is configured such that the one or more instructions are individually or collectively executed by the at least one processor (120). An electronic device (100) that displays the breathing guide UI on the instrument panel display.

12. In any one of clauses 8 to 11, The above display unit (140) is An electronic device (100) that displays a respiratory waveform of the respiratory signal acquired in real time together with the respiratory movement guide.

13. In any one of paragraphs 8 to 12, The above display unit (140) is Among the above breathing signals, the respiratory waveforms of the thoracic breathing signal obtained from the driver's chest movement and the abdominal breathing signal obtained from the driver's abdominal movement are displayed together, An electronic device (100) that displays information about real-time changes in each of the thoracic breathing signal and the abdominal breathing signal.

14. In any one of paragraphs 8 to 13, Further comprising a communication interface (160); The electronic device (100) is configured such that the one or more instructions are individually or collectively executed by the at least one processor (120). If hyperventilation or respiratory distress is detected during the above abnormal breathing, a warning message indicating an emergency situation is output through the display unit (140). If there is no interaction of the driver corresponding to the warning message outputted above, the driving of the vehicle (1000) is controlled to move the vehicle (1000) to the shoulder and stop it, An electronic device (100) that controls the above communication interface (160) to send an emergency call for a rescue request.

15. In a computer program product including a computer-readable storage medium, The above storage medium, An operation of acquiring a breathing signal of the driver by detecting movement of at least one of the chest and abdomen caused by the driver's breathing from an image acquired through a camera (110); An operation of monitoring the acquired breathing signal in real time to detect abnormal breathing including at least one of hyperventilation, hypopnea, transient apnea, and dyspnea; and An action of displaying a breathing guide UI (user interface) that provides a breathing exercise guide to alleviate the abnormal breathing by reorganizing the driver's breathing when the abnormal breathing is detected; A computer program product including instructions executed by an electronic device (100) mounted on a vehicle (1000) to perform a task.

Images