Non-contact autonomic nervous system measurement device and method for monitoring state of occupant in next-generation mobility

Abstract

According to a non-contact autonomic nervous system measurement device and method for monitoring the state of an occupant in next-generation mobility proposed in the present invention, the identity of an occupant is verified using a facial image of the occupant captured by a camera and bio-signals are estimated to monitor the occupant's health and emotional states, thereby tracking the occupant's bio-signals, stress, and emotional states to enable accident prevention and protection of occupant safety.

Description

Non-contact autonomic nervous system measurement device and method for monitoring the condition of next-generation mobility passengers

[0001] The present invention relates to a non-contact autonomic nervous system measurement device and method, and more specifically, to a non-contact autonomic nervous system measurement device and method for monitoring the condition of next-generation mobility vehicle passengers. The present invention was conducted with support from the Daegu Digital Innovation Promotion Agency (Research Project Name: Industry-Academia R&BD Collaboration Commercialization Project).

[0002] The content described in this section merely provides background information for one embodiment of the present invention and does not constitute prior art.

[0003]

[0004] Next-generation mobility refers to future modes of transportation that are evolving with technological advancements. Figures 1 and 2 are diagrams illustrating examples of next-generation mobility. Electric vehicles (EVs), autonomous vehicles (AVs), and hydrogen vehicles (HVs), as shown in Figure 1, and drones and urban air mobility (UAM), as shown in Figure 2, are examples of next-generation mobility.

[0005]

[0006] These next-generation mobility vehicles will have passengers, including drivers, inside. Monitoring the health of these passengers is crucial for rapid response to health issues. In particular, with the aging population growing, drivers are also getting older. Furthermore, fully autonomous driving has not yet been achieved, requiring a driver on board is essential. Therefore, monitoring the driver's current cognitive function and other health conditions is crucial for accident prevention.

[0007]

[0008] However, existing biosignal acquisition technologies are unable to monitor in real time, and most are limited to capturing single biosignals. Furthermore, because measured biosignals are collected without user awareness, reliability of the data is questionable. Consequently, personalized predictions of comprehensive health and emotional states are limited. Therefore, to ensure the safe and efficient use of next-generation mobility, the development of technologies capable of monitoring the diverse health status of passengers is essential.

[0009]

[0010] Meanwhile, as a prior art related to the present invention, Patent Publication No. 10-2023-0109748 (Title of invention: System for monitoring biometric data of vehicle occupants, Publication date: July 20, 2023) has been disclosed.

[0011]

[0012] The background technology described above is technical information that the inventor possessed for the purpose of deriving the present invention or acquired in the process of deriving the present invention, and cannot necessarily be said to be publicly known technology disclosed to the general public prior to the application for the present invention.

[0013] The present invention has been proposed to solve the above-mentioned problems of the existing proposed methods, and the purpose of the present invention is to provide a non-contact autonomic nervous system measurement device and method for monitoring the state of a next-generation mobility passenger, which can prevent accidents and ensure passenger safety by tracking the passenger's bio-signals, stress, and emotional state by verifying the passenger's identity using the passenger's facial image captured by a camera and estimating the bio-signals to monitor the passenger's health and emotional state.

[0014]

[0015] However, the technical problem to be achieved by the present invention is not limited to the technical problem described above, and other technical problems may exist, and even if not explicitly mentioned, the purpose or effect that can be understood from the solution or embodiment of the problem is also included.

[0016] A non-contact autonomic nervous system measuring device for monitoring the status of a next-generation mobility passenger according to the features of the present invention to achieve the above-mentioned purpose,

[0017] As a non-contact autonomic nervous system measurement device,

[0018] A camera unit installed in next-generation mobility that captures the faces of passengers;

[0019] An identification unit that recognizes the passenger's face captured by the above camera unit and verifies the passenger's identity;

[0020] A bio-signal estimation unit that estimates the passenger's bio-signal from the image captured by the above camera unit;

[0021] An emotion estimation unit that estimates the emotions of the passenger from the image captured by the camera unit and the bio-signal estimated by the bio-signal estimation unit; and

[0022] The configuration is characterized by including a monitoring unit that monitors the health and emotional state of the passenger using the bio-signal estimated by the bio-signal estimation unit and the emotion estimated by the emotion estimation unit.

[0023]

[0024] Preferably, the biosignal estimation unit,

[0025] Heart rate variability (HRV) can be estimated in real time from the video captured by the above camera unit.

[0026]

[0027] More preferably, the emotion estimation unit,

[0028] An expression feature extraction module that analyzes the expression of a passenger from an image captured by the above camera unit and extracts expression features;

[0029] An autonomic nervous system feature extraction module that estimates an HRV index from the heart rate variability estimated in the biosignal estimation unit and extracts autonomic nervous system features from the estimated HRV index; and

[0030] It may include an emotion prediction module that combines the above facial expression features and autonomic nervous system features to form a final feature and predicts the passenger's emotion using the formed final feature.

[0031]

[0032] More preferably, the biosignal estimation unit,

[0033] Bio-signals including blood pressure, oxygen saturation, and pulse can be estimated in real time from images captured by the above camera unit.

[0034]

[0035] Preferably, the monitoring unit,

[0036] The health and emotional state of the above passengers can be provided to the passengers in real time through a display.

[0037]

[0038] Preferably, the monitoring unit,

[0039] If the health and emotional state of the above passenger deviates from the standard value, a warning signal can be output to indicate the need for rest.

[0040]

[0041] More preferably, the reference value is:

[0042] The passenger's identity confirmed in the above identification section can be set differently according to the passenger's information.

[0043]

[0044] Preferably,

[0045] It may further include a storage unit that links and stores the passenger's identity confirmed in the above identification unit and the passenger's health status and emotional status data.

[0046]

[0047] A non-contact autonomic nervous system measurement method for monitoring the status of a next-generation mobility passenger according to the features of the present invention to achieve the above-mentioned purpose is as follows:

[0048] A non-contact autonomic nervous system measurement method in which each step is performed in a non-contact autonomic nervous system measurement device,

[0049] (1) Video shooting step for capturing the passenger’s face through a camera unit installed in next-generation mobility;

[0050] (2) An identity verification step for recognizing the face of the passenger filmed in the above video filming step and confirming the identity;

[0051] (3) A biosignal estimation step for estimating the passenger's biosignal from the image captured in the above image capture step;

[0052] (4) an emotion estimation step for estimating the passenger's emotions from the video captured in the video capturing step and the bio-signals estimated in the bio-signal estimation step; and

[0053] (5) It is characterized in its configuration by including a monitoring step for monitoring the health and emotional state of the passenger using the biological signal estimated in the biological signal estimation step and the emotion estimated in the emotional estimation step.

[0054]

[0055] Preferably, in the biosignal estimation step,

[0056] Heart rate variability (HRV) can be estimated in real time from the video captured in the above video capture step.

[0057]

[0058] More preferably, the emotion estimation step is:

[0059] (4-1) A facial expression feature extraction step for analyzing the passenger's facial expression from the video captured in the above video capture step and extracting facial expression features;

[0060] (4-2) An autonomic nervous system feature extraction step for estimating the HRV index from the heart rate variability estimated in the biosignal estimation step and extracting autonomic nervous system features from the estimated HRV index; and

[0061] (4-3) It may include an emotion prediction step of combining the above facial expression features and autonomic nervous system features to form a final feature and predicting the passenger's emotion using the formed final feature.

[0062]

[0063] More preferably, the biosignal estimation step comprises:

[0064] In the above image capturing step, bio-signals including blood pressure, oxygen saturation, and pulse can be estimated in real time from the captured images.

[0065]

[0066] Preferably, the monitoring step comprises:

[0067] The health and emotional state of the above passengers can be provided to the passengers in real time through a display.

[0068]

[0069] Preferably, the monitoring step comprises:

[0070] If the health and emotional state of the above passenger deviates from the standard value, a warning signal can be output to indicate the need for rest.

[0071]

[0072] More preferably, the reference value is:

[0073] The passenger's identity confirmed in the above identification step can be set differently according to the passenger's information.

[0074]

[0075] Preferably, after the above step (5),

[0076] (6) It may further include a storage step of linking and storing the passenger's identity and the passenger's health and emotional state data confirmed in the above identification unit.

[0077] According to the non-contact autonomic nervous system measurement device and method for monitoring the condition of a next-generation mobility passenger proposed in the present invention, the passenger's identity is confirmed using a facial image captured by a camera, and the passenger's health and emotional state are monitored by estimating the passenger's bio-signals, thereby preventing accidents and ensuring the passenger's safety by tracking the passenger's bio-signals, stress, and emotional state.

[0078]

[0079] In addition, the various advantageous advantages and effects of the present invention are not limited to the above-described contents, and will be more easily understood in the process of explaining specific embodiments of the present invention.

[0080] Figures 1 and 2 are drawings illustrating next-generation mobility as an example.

[0081] FIG. 3 is a drawing illustrating a detailed configuration of a non-contact autonomic nervous system measurement device for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention.

[0082] FIG. 4 is a drawing showing the overall appearance of a non-contact autonomic nervous system measuring device for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention.

[0083] FIG. 5 is a drawing illustrating a non-contact autonomic nervous system measurement device for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention, installed inside the next-generation mobility.

[0084] FIG. 6 is a drawing illustrating a biosignal estimation unit in a non-contact autonomic nervous system measurement device for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention.

[0085] FIG. 7 is a drawing illustrating an emotion estimation unit in a non-contact autonomic nervous system measurement device for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention.

[0086] FIG. 8 is a drawing showing a detailed configuration of an emotion estimation unit in a non-contact autonomic nervous system measurement device for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention.

[0087] FIG. 9 is a diagram illustrating a flow chart of a non-contact autonomic nervous system measurement method for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention.

[0088] FIG. 10 is a diagram illustrating a detailed flow of an emotion estimation step in a non-contact autonomic nervous system measurement method for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention.

[0089] <Explanation of symbols>

[0090] 100: Non-contact autonomic nervous system measurement device

[0091] 110: Camera section

[0092] 120: Identity Verification Department

[0093] 130: Biosignal estimation unit

[0094] 140: Emotion Estimation Department

[0095] 141: Facial Expression Feature Extraction Module

[0096] 142: Autonomic Nervous System Feature Extraction Module

[0097] 143: Emotion Prediction Module

[0098] 150: Monitoring Department

[0099] 160: Storage

[0100] S110: Video recording stage

[0101] S120: Identity Verification Step

[0102] S130: Biosignal Estimation Step

[0103] S140: Emotion Estimation Step

[0104] S141: Facial expression feature extraction step

[0105] S142: Autonomic Nervous System Feature Extraction Step

[0106] S143: Emotion Prediction Stage

[0107] S150: Monitoring phase

[0108] S160: Save phase

[0109] Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily implement them. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, irrelevant parts have been omitted for clarity of description, and similar reference numerals have been used throughout the specification to indicate similar parts.

[0110]

[0111] Throughout the specification, when a part is said to be "connected" to another part, this includes not only cases where it is "directly connected" but also cases where it is "indirectly connected" with another element in between. In addition, terms such as "comprise," "include," or "have" used below should be interpreted to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, and should be understood to not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. In addition, the singular expression used in the present invention includes the plural expression unless the context clearly indicates otherwise.

[0112]

[0113] In addition, each configuration, process, procedure or method included in each embodiment of the present invention may be shared within a scope that is not technically inconsistent with each other.

[0114]

[0115] Additionally, terms such as “part,” “unit,” and “module” described in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware, software, or a combination of hardware and software.

[0116]

[0117] Additionally, some of the operations or functions described as being performed by a terminal, apparatus, or device in the present invention may instead be performed by a server connected to the terminal, apparatus, or device. Similarly, some of the operations or functions described as being performed by a server may also be performed by a terminal, apparatus, or device connected to the server.

[0118]

[0119] In particular, a means for executing a system according to each embodiment of the present invention may be an application or a web server, and a terminal that is a means for reading a recording medium recording the application or web server may include not only a general PC such as a general desktop or laptop, but also a mobile terminal such as a smart phone or tablet PC.

[0120]

[0121] The following examples are provided as detailed explanations to aid understanding of the present invention and do not limit the scope of the invention. Therefore, inventions with the same functions and scope as the present invention are also within the scope of the present invention.

[0122]

[0123] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0124]

[0125] FIG. 3 is a drawing showing a detailed configuration of a non-contact autonomic nervous system measuring device (100) for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention. As illustrated in FIG. 3, the non-contact autonomic nervous system measuring device (100) for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention is a non-contact autonomic nervous system measuring device (100), which may be configured to include a camera unit (110), an identification unit (120), a biosignal estimation unit (130), an emotion estimation unit (140), and a monitoring unit (150), and may further include a storage unit (160).

[0126]

[0127] According to the non-contact autonomic nervous system measuring device (100) and method for monitoring the condition of a next-generation mobility passenger proposed in the present invention, a camera unit (110) captures an image of the passenger's face, and using the captured image, an identification unit (120) performs identification of the passenger, a bio-signal estimation unit (130) performs bio-signal estimation, and an emotion estimation unit (140) performs emotion estimation of the passenger, respectively. In addition, a monitoring unit (150) can monitor the passenger's health and emotional state using the estimated bio-signals and emotions. Therefore, by tracking the passenger's bio-signals, stress, and emotional state, accidents can be prevented and the passenger's safety can be protected.

[0128]

[0129] FIG. 4 is a drawing illustrating the overall appearance of a non-contact autonomic nervous system measurement device (100) for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention. The non-contact autonomic nervous system measurement device (100) for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention can monitor bio-signals such as blood pressure, oxygen saturation, and pulse by taking a picture of the driver's face in a non-contact manner using a general camera. To this end, as illustrated in FIG. 4, the device may include a webcam-shaped camera and a display screen on the front. As illustrated in FIG. 4, the device is designed in the shape of a spherical jar with flat upper and lower surfaces to match the interior decoration of next-generation mobility, and a touch-enabled display panel can be applied to facilitate operation. According to an embodiment, a detachable case (not illustrated) may be provided to protect the non-contact autonomic nervous system measurement device (100), while allowing selection of the color or shape of the case design to suit the interior of next-generation mobility or the preferences of passengers such as the driver.

[0130]

[0131] FIG. 5 is a drawing illustrating a non-contact autonomic nervous system measurement device (100) for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention, installed inside the next-generation mobility. As illustrated in FIG. 5, it can be installed by placing it facing the passenger to be monitored inside the next-generation mobility. At this time, the detachable case (not shown) is made of silicone material so that it can be easily removed, and at the same time, it can be well fixed inside the next-generation mobility by frictional force, so that the non-contact autonomic nervous system measurement device (100) does not shake even when simply placed inside the mobility. Depending on the embodiment, it can also be installed in a built-in manner inside the next-generation mobility.

[0132]

[0133] Hereinafter, the detailed configuration of a non-contact autonomic nervous system measurement device (100) for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention will be described in detail with reference to FIGS. 3 to 5.

[0134]

[0135] The camera unit (110) can be installed in next-generation mobility to capture images of a passenger's face. The camera unit (110) can be a standard camera, such as a webcam. When a non-contact autonomic nervous system measurement device (100) is installed in the next-generation mobility so that the camera faces the passenger, the camera unit (110) can capture images of the passenger's face.

[0136]

[0137] In addition, the camera unit (110) may include an illuminance sensor to obtain illuminance data at the time the image is captured, so as to take into account that the brightness of the image changes depending on the illuminance. That is, at night or inside a tunnel, the illuminance may be low, resulting in a dark image of the passenger's face, and during the day when sunlight is strong, the passenger's face may be captured with reflected light or shadows depending on the angle of the sun. These problems may make it difficult to confirm identity or estimate bio-signals using the image, and may lower the accuracy of the estimated bio-signals. Therefore, the illuminance data obtained by the camera unit (110) may be used to correct the image, or the illuminance data may be provided to the identification unit (120) and the emotion estimation unit (140) together with the image so that the image can be corrected or the data obtained from the image can be corrected.

[0138]

[0139] The identity verification unit (120) can verify the identity of a passenger by recognizing the passenger's face captured by the camera unit (110). That is, the identity verification unit (120) can verify the passenger's identity by searching the passenger's face recognized in a pre-configured face database. If the identified passenger's identity is not on the passenger list authorized to board, the unit can take measures such as stopping the operation or generating an alarm sound. In addition, by securing the passenger's existing health and emotional state data through identity verification, the biosignal estimation unit (130), the emotion estimation unit (140), and the monitoring unit (150), which will be described in detail below, can integrate the existing data with the current data to analyze and monitor.

[0140]

[0141] The biosignal estimation unit (130) can estimate the passenger's biosignal from the image captured by the camera unit (110). The biosignal estimation unit (130) can estimate heart rate variability (HRV) in real time from the image captured by the camera unit (110), and can estimate biosignals including blood pressure, oxygen saturation, and pulse in real time.

[0142]

[0143] FIG. 6 is a diagram illustrating a biosignal estimation unit (130) of a non-contact autonomic nervous system measurement device (100) for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention. As illustrated in FIG. 6, the biosignal estimation unit (130) of the non-contact autonomic nervous system measurement device (100) for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention can analyze a quantitative signal of heart rate variability using an interpolation method and a re-sampling method. HRV is one of the biomarkers for evaluating the balance of the autonomic nervous system and indicates stress resistance. More specifically, a lower HRV indicates an increased vulnerability to stress, a more sensitive reaction to even minor stressors, and a decreased ability to cope with stressful situations. In addition, a higher HRV indicates a higher adaptability to functionally adapt to new changes. Therefore, HRV can be used to determine a passenger's stress level and ability to cope with unexpected situations while driving.

[0144]

[0145] Meanwhile, the biosignal estimation unit (130) can extract a region of interest from the passenger's face image and input the features extracted from the region of interest into a biosignal estimation model, which is an artificial intelligence model that has completed learning, to estimate the biosignal. If the face image is distorted due to lighting, etc., an accurate biosignal cannot be estimated. Therefore, the biosignal estimation unit (130) can receive illumination data together with the image from the camera unit (110) and use it to estimate the biosignal. For example, the image can be corrected using the illumination data, and the corrected image can be used to estimate the bioinformation. Depending on the embodiment, the illumination data can be input together into the biosignal estimation model to estimate the bioinformation that reflects the difference according to the illumination.

[0146]

[0147] The emotion estimation unit (140) can estimate the emotions of the passenger from the video captured by the camera unit (110) and the bio-signals estimated by the bio-signal estimation unit (130). Facial expressions are the most widely used features for emotion recognition, but since facial expressions can be controlled and manipulated, it is difficult to identify the truly felt emotional state through facial expressions. Therefore, identifying emotions using only facial expression features extracted from facial images is inaccurate because emotions may be hidden or misrecognized. There is a strong correlation between human emotional states and bio-signals, and heart rate variability provides information on the activity of the sympathetic and parasympathetic nervous systems, allowing for inferences about the suppression and excitation processes of emotional regulation. Therefore, the emotion estimation unit (140) of the present invention can estimate the emotions of the passenger by using both facial expressions and heart rate variability.

[0148]

[0149] FIG. 7 is a diagram illustrating an emotion estimation unit (140) in a non-contact autonomic nervous system measurement device (100) for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention. As illustrated in FIG. 7, the emotion estimation unit (140) of the non-contact autonomic nervous system measurement device (100) for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention can estimate the passenger's emotion by using an expression feature obtained from a facial image of the passenger and an HRV index obtained from the HRV estimated by the biosignal estimation unit (130).

[0150]

[0151] FIG. 8 is a drawing showing a detailed configuration of an emotion estimation unit (140) in a non-contact autonomic nervous system measurement device (100) for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention. As illustrated in FIG. 8, the emotion estimation unit (140) of the non-contact autonomic nervous system measurement device (100) for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention may be configured to include an expression feature extraction module (141), an autonomic nervous system feature extraction module (142), and an emotion prediction module (143).

[0152]

[0153] The facial expression feature extraction module (141) can analyze the facial expression of a passenger from an image captured by the camera unit (110) and extract facial expression features. The facial expression feature extraction module (141) inputs the facial image of the passenger into an artificial intelligence algorithm that extracts facial expression features from the image, extracts the features, and can encode them into facial expression features of a predetermined size.

[0154]

[0155] The autonomic nervous system feature extraction module (142) can estimate the HRV index from the heart rate variability estimated by the biosignal estimation unit (130), and extract autonomic nervous system features from the estimated HRV index. The HRV index is an HRV index calculated based on the RR interval sequence extracted from the ECG, and can be configured to include six indices: the standard deviation of RR intervals (SDNN), the square root of the mean square difference (RMSSD) of consecutive RR intervals, normalized low-frequency power (LF), high-frequency power (HF), the ratio of low-frequency and high-frequency power (LF / HF), and a nonlinear index (SampEn) used for the analysis of heart rate variability and time series complexity. Therefore, the autonomic nervous system feature extraction module (142) can calculate six HRV indices from the heart rate variability, and extract autonomic nervous system features by encoding the six calculated HRV indices into features of a predetermined size through a Dense layer.

[0156]

[0157] The emotion prediction module (143) can combine facial expression features and autonomic nervous system features to form a final feature, and can predict the emotions of the passenger using the final feature. That is, the emotion prediction module (143) can concatenate facial expression features and autonomic nervous system features to form a single final feature, and input the final feature into an artificial intelligence-based emotion classification model to predict the emotions of the passenger. At this time, the emotion prediction module (143) can predict four emotions: angry, happy, sad, and neutral. In some embodiments, drowsiness or cognitive impairment can be added to the predicted emotion classes to determine whether the next-generation mobility passenger is driving while drowsy or cognitively impaired.

[0158]

[0159] The monitoring unit (150) can monitor the health and emotional state of the passenger using the bio-signals estimated by the bio-signal estimation unit (130) and the emotions estimated by the emotion estimation unit (140). Here, the health state of the passenger refers to changes in the bio-signals of the passenger, and the monitoring unit (150) can monitor the health state by tracking the changes in the bio-signals of the passenger and determining whether the bio-signals deviate from a reference value. For example, it can monitor whether the HRV deviates from a reference value and thus stress is excessive and rest is necessary. In addition, the monitoring unit (150) can also determine the drowsiness state by combining the bio-signals and emotions of the passenger.

[0160]

[0161] Meanwhile, the monitoring unit (150) can output a warning signal indicating the need for rest if the passenger's health and emotional state deviate from the reference value. That is, as illustrated in FIG. 4, a message requesting rest, "REST RECOMMENDED," can be displayed, and a warning sound or voice can also be output. Furthermore, the monitoring unit (150) can provide the passenger with the passenger's health and emotional state in real time through the display. At this time, as illustrated in FIG. 4, specific biometric information itself can be displayed, a stress index can be displayed, or summarized information such as health status good / normal / caution can be provided.

[0162]

[0163] Meanwhile, the threshold value at which the monitoring unit (150) outputs a warning signal can be set differently depending on the passenger's information, using the passenger's identity confirmed by the identification unit (120). That is, the passenger's existing health and emotional state data can be obtained through identification of the passenger, and the threshold value can be set according to the passenger's characteristics based on the existing data.

[0164]

[0165] The storage unit (160) can store the passenger's identity confirmed by the identification unit (120), the passenger's health status and emotional state data, and bio-signals in conjunction with each other. Therefore, the reliability of the stored data can be ensured.

[0166]

[0167] FIG. 9 is a diagram illustrating a flowchart of a non-contact autonomic nervous system measurement method for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention. As illustrated in FIG. 9, the non-contact autonomic nervous system measurement method for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention is a non-contact autonomic nervous system measurement method in which each step is performed in a non-contact autonomic nervous system measurement device (100), including: an image capturing step (S110) of capturing the face of a passenger through a camera unit (110) mounted on next-generation mobility; an identity verification step (S120) of recognizing the face of the passenger captured in the image capturing step (S110) to verify the identity; a bio-signal estimation step (S130) of estimating the bio-signal of the passenger from the image captured in the image capturing step (S110); an emotion estimation step (S140) of estimating the emotion of the passenger from the image captured in the image capturing step (S110) and the bio-signal estimated in the bio-signal estimation step (S130); And it can be implemented including a monitoring step (S150) for monitoring the health and emotional state of the passenger using the bio-signals estimated in the bio-signal estimation step (S130) and the emotions estimated in the emotion estimation step (S140), and it can be implemented further including a storage step (S160) for linking and storing the identity of the passenger confirmed in the identity verification unit (120) and the health and emotional state data of the passenger.

[0168]

[0169] FIG. 10 is a diagram illustrating a detailed flow of an emotion estimation step (S140) in a non-contact autonomic nervous system measurement method for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention. As illustrated in FIG. 9, the emotion estimation step (S140) of the non-contact autonomic nervous system measurement method for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention may be implemented by including an expression feature extraction step (S141) of analyzing the passenger's expression from an image captured in the image capture step (S110) to extract expression features; an autonomic nervous system feature extraction step (S142) of estimating an HRV index from the heart rate variability estimated in the biosignal estimation step (S130) and extracting autonomic nervous system features from the estimated HRV index; and an emotion prediction step (S143) of combining the expression features and the autonomic nervous system features to configure a final feature and predicting the passenger's emotion using the configured final feature.

[0170]

[0171] Since the details related to each step have been sufficiently described in relation to the non-contact autonomic nervous system measurement device (100) for monitoring the status of a next-generation mobility passenger according to an embodiment of the present invention, a detailed description will be omitted.

[0172]

[0173] As described above, according to the non-contact autonomic nervous system measuring device (100) and method for monitoring the condition of a next-generation mobility passenger proposed in the present invention, the passenger's identity is confirmed using the passenger's facial image captured by the camera, and the passenger's health and emotional state are monitored by estimating the bio-signals, thereby preventing accidents and ensuring the passenger's safety by tracking the passenger's bio-signals, stress, and emotional state.

[0174]

[0175] Meanwhile, the present invention may include a computer-readable medium including program instructions for performing operations implemented by various communication terminals. For example, the computer-readable medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and perform program instructions such as ROMs, RAMs, and flash memories.

[0176]

[0177] Such computer-readable media may contain program commands, data files, data structures, and the like, either singly or in combination. The program commands recorded on the computer-readable media may be specifically designed and configured to implement the present invention, or may be known and available to those skilled in the art of computer software. For example, they may include not only machine language codes, such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

[0178]

[0179] The foregoing description of the present invention is provided for illustrative purposes only, and those skilled in the art will readily appreciate that the present invention can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single entity may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined manner.

[0180]

[0181] The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention.

Claims

1. As a non-contact autonomic nervous system measuring device (100), A camera unit (110) mounted on a next-generation mobility vehicle to photograph the face of the passenger; An identity verification unit (120) that recognizes the face of a passenger captured by the camera unit (110) and verifies their identity; A biosignal estimation unit (130) that estimates the biosignal of a passenger from an image captured by the camera unit (110); An emotion estimation unit (140) that estimates the emotion of a passenger from an image captured by the camera unit (110) and a biosignal estimated by the biosignal estimation unit (130); and A non-contact autonomic nervous system measuring device (100) for monitoring the condition of a next-generation mobility passenger, characterized in that it includes a monitoring unit (150) that monitors the passenger's health and emotional state using the bio-signal estimated by the bio-signal estimation unit (130) and the emotion estimated by the emotion estimation unit (140).

2. In paragraph 1, the biosignal estimation unit (130) is, A non-contact autonomic nervous system measurement device (100) for monitoring the status of a next-generation mobility passenger, characterized by estimating heart rate variability (HRV) in real time from an image captured by the above camera unit (110).

3. In paragraph 2, the above-mentioned appraisal estimation unit (140) is, A facial expression feature extraction module (141) that analyzes the facial expression of a passenger from an image captured by the camera unit (110) and extracts facial expression features; An autonomic nervous system feature extraction module (142) that estimates an HRV index from the heart rate variability estimated by the biosignal estimation unit (130) and extracts autonomic nervous system features from the estimated HRV index; and A non-contact autonomic nervous system measuring device (100) for monitoring the status of a next-generation mobility passenger, characterized in that it includes an emotion prediction module (143) that combines the above facial expression features and autonomic nervous system features to form a final feature and predicts the passenger's emotion using the formed final feature.

4. In paragraph 2, the biosignal estimation unit (130) is, A non-contact autonomic nervous system measurement device (100) for monitoring the condition of a next-generation mobility passenger, characterized by estimating biosignals including blood pressure, oxygen saturation, and pulse in real time from an image captured by the camera unit (110).

5. In paragraph 1, the monitoring unit (150) is, A non-contact autonomic nervous system measuring device (100) for monitoring the condition of a next-generation mobility passenger, characterized by providing the passenger with the health and emotional conditions of the passenger in real time through a display.

6. In paragraph 1, the monitoring unit (150) is, A non-contact autonomic nervous system measuring device (100) for monitoring the condition of a next-generation mobility passenger, characterized by outputting a warning signal indicating that rest is needed when the health and emotional state of the passenger deviate from a reference value.

7. In paragraph 6, the reference value is A non-contact autonomic nervous system measurement device (100) for monitoring the condition of a next-generation mobility passenger, characterized by being set differently according to the passenger's information using the passenger's identity confirmed by the above-mentioned identity verification unit (120).

8. In paragraph 1, A non-contact autonomic nervous system measurement device (100) for monitoring the condition of a next-generation mobility passenger, characterized by further including a storage unit (160) that stores the passenger's identity confirmed by the identity verification unit (120) and the passenger's health and emotional state data in conjunction.

9. A non-contact autonomic nervous system measurement method in which each step is performed in a non-contact autonomic nervous system measurement device (100), (1) A video recording step (S110) of capturing the face of a passenger through a camera unit (110) mounted on a next-generation mobility device; (2) An identity verification step (S120) that recognizes the face of the passenger captured in the above video recording step (S110) and verifies the identity; (3) A biosignal estimation step (S130) for estimating the occupant's biosignal from the image captured in the above image capturing step (S110); (4) An emotion estimation step (S140) for estimating the passenger's emotions from the image captured in the image capturing step (S110) and the bio-signal estimated in the bio-signal estimation step (S130); and (5) A non-contact autonomic nervous system measurement method for monitoring the status of a next-generation mobility passenger, characterized in that it includes a monitoring step (S150) for monitoring the health status and emotional status of the passenger using the bio-signal estimated in the bio-signal estimation step (S130) and the emotion estimated in the emotion estimation step (S140).

10. In claim 9, in the biosignal estimation step (S130), A non-contact autonomic nervous system measurement method for monitoring the status of a next-generation mobility passenger, characterized by estimating heart rate variability (HRV) in real time from an image captured in the above image capture step (S110).

11. In Clause 10, the above-mentioned appraisal estimation step (S140) is, (4-1) A facial expression feature extraction step (S141) that extracts facial expression features by analyzing the facial expression of the passenger from the video captured in the above video capturing step (S110); (4-2) An autonomic nervous system feature extraction step (S142) for estimating the HRV index from the heart rate variability estimated in the biosignal estimation step (S130) and extracting autonomic nervous system features from the estimated HRV index; and (4-3) A non-contact autonomic nervous system measurement method for monitoring the status of a next-generation mobility passenger, characterized in that it includes an emotion prediction step (S143) of combining the above facial expression features and autonomic nervous system features to form a final feature and predicting the passenger's emotion using the formed final feature.

12. In item 10, in the above biosignal estimation step (S130), A non-contact autonomic nervous system measurement method for monitoring the status of a next-generation mobility passenger, characterized in that it estimates bio-signals including blood pressure, oxygen saturation, and pulse in real time from an image captured in the above image capture step (S110).

13. In paragraph 9, in the monitoring step (S150), A non-contact autonomic nervous system measurement method for monitoring the condition of a next-generation mobility passenger, characterized by providing the passenger with the passenger's health and emotional conditions in real time through a display.

14. In paragraph 9, in the monitoring step (S150), A non-contact autonomic nervous system measurement method for monitoring the condition of a next-generation mobility passenger, characterized by outputting a warning signal indicating that rest is needed when the passenger's health and emotional state deviate from reference values.

15. In Paragraph 14, the above reference value is, A non-contact autonomic nervous system measurement method for monitoring the condition of a next-generation mobility passenger, characterized by using the passenger's identity confirmed in the above-mentioned identity verification step (S120) and setting it differently according to the passenger's information.

16. In paragraph 9, after the above step (5), (6) A non-contact autonomic nervous system measurement method for monitoring the condition of a next-generation mobility passenger, characterized by further including a storage step (S160) for storing the passenger's identity confirmed by the identity verification unit (120) and the passenger's health and emotional state data in conjunction.

Images