Apparatus and method for integrating vital data of vehicle

Abstract

An apparatus for integrating vital data of a vehicle according to the present invention comprises: a reception unit for receiving respective pieces of vital data collected by a plurality of cameras installed inside the vehicle; a data determination unit for evaluating the reliability of the received vital data to assign a weight thereto, and selecting or converging the vital data on the basis of the assigned weight; and a data providing unit for providing finally generated vital data so as to estimate a state index of a driver. Accordingly, reliable input data can be provided for high-level vital analysis.

Description

Vital data integration device and method for integrating vital data of a vehicle

[0001] The present invention relates to a vital data integration device and a vital data integration method for a vehicle, and more specifically, to a vital data integration device and a vital data integration method using vital data sensed through a plurality of cameras in a vehicle.

[0002] A vehicle is a device that allows the user to move in the desired direction. A representative example is an automobile.

[0003] Meanwhile, various sensors and electronic devices are being installed in vehicles to enhance the convenience of users. In particular, research is actively underway on Advanced Driver Assistance Systems (ADAS) to enhance user convenience. Furthermore, development of autonomous vehicles (AVs) is also actively underway.

[0004] Furthermore, research into technologies that continuously monitor driver drowsiness is increasing. Furthermore, Euro NCAP, the European car assessment organization, plans to introduce non-contact technology to assess driver and passenger drowsiness. To detect driver drowsiness, multiple in-vehicle cameras are being used to seamlessly track driver expressions and facial expressions.

[0005] According to some embodiments of the present invention, the purpose is to provide a vital data integration device and a vital data integration method for a vehicle capable of seamlessly generating reliable vital data by selecting and fusing a plurality of vital data acquired using a plurality of cameras in the vehicle.

[0006] To this end, the vehicle vital data integration device according to an embodiment of the present invention can estimate the driver's condition index with higher reliability by selecting and fusing vital data collected through a plurality of cameras installed inside the vehicle and providing the data.

[0007] Additionally, the vehicle's vital data integration device can evaluate the reliability of vital data based on various perceptions and assign different weights based on the evaluation to generate highly reliable vital data.

[0008] Specifically, a vital data integration device for a vehicle according to an embodiment of the present invention may include a receiving unit that receives each vital data collected by a plurality of cameras installed inside the vehicle; a data determining unit that evaluates the reliability of the received vital data, assigns a weight to the data, and selects or fuses the vital data based on the assigned weight; and a data providing unit that provides the final generated vital data so as to estimate a driver's condition index.

[0009] In an embodiment, the reliability evaluation for the received vital data may determine different weights for each vital data based on at least one of the reliability of each of the plurality of cameras, landmark information of images sensed through the plurality of cameras, the area occupied by the face in the images, and the amount of change compared to previous vital data.

[0010] In an embodiment, each of the plurality of cameras is installed around the driver's seat in the vehicle to sense the driver's vital data using the rPPG (Remote PPG) method, and the receiving unit can time-synchronize the plurality of sensed vital data and transmit them to the data determination unit.

[0011] In an embodiment, the data determination unit may remove data having outliers from among the plurality of vital data, and then evaluate the reliability of normal vital data and assign a weight to it.

[0012] In an embodiment, the data determination unit may select vital data having a high reliability weight based on the assigned weight or fuse each vital data with a reliability weight and transmit it to the data provision unit.

[0013] In an embodiment, the data determination unit may calculate an average of each vital data based on the assigned weights, select the calculated average vital data, and transmit it to the data provision unit as a single stream.

[0014] In an embodiment, the data determination unit may apply the assigned weight to the probability value of each vital data, and then calculate an average based on the assigned weight, thereby calculating a reliability weight probability for each vital data.

[0015] In an embodiment, the data determination unit may select vital data having the highest value among the reliability weight probabilities or calculate an average of the reliability weight probabilities to generate fused vital data, and transmit the selected vital data or fused vital data to the data provision unit.

[0016] In an embodiment, the data determination unit may adjust the reliability weight for each vital data differently based on the vehicle control information and the passenger's movement information.

[0017] In an embodiment, the data determination unit may determine that the amount of change in each vital data has temporarily changed based on control information of the vehicle, and may reduce the reliability weight for each vital data according to the determination.

[0018] In an embodiment, the control information of the vehicle may be a control result corresponding to any one of brake, acceleration, and steering operation exceeding a set range performed by a driver or an ADAS system.

[0019] In an embodiment, the data determination unit may determine that there is no change in vehicle control information received for a predetermined period of time or longer, and may increase a reliability weight for each vital data or shorten a reliability weight calculation cycle based on the determination.

[0020] In an embodiment, the data determination unit may determine that the amount of change in each vital data has temporarily changed based on the movement information of the passenger, and may reduce the reliability weight for each vital data or lengthen the reliability weight calculation cycle based on the determination.

[0021] In addition, a method for integrating vital data of a vehicle according to an embodiment of the present invention may include the steps of: receiving each vital data sensed by a plurality of cameras installed inside the vehicle; evaluating the reliability of the received vital data and assigning a weight; selecting or merging the vital data based on the assigned weight; and providing the final generated vital data as input data to estimate a driver's condition index.

[0022] In an embodiment, the step of assigning weights may be a step of evaluating the reliability of each vital data and determining different weights based on at least one of the reliability of each of the plurality of cameras, landmark information of images sensed through the plurality of cameras, the area occupied by the face in the images, and the amount of change compared to previous vital data.

[0023] In an embodiment, the step of assigning weights may include a step of additionally adjusting a reliability weight for each vital data based on vehicle control information and passenger movement information.

[0024] The effects of the vital data integration device and vital data integration method of a vehicle according to the present invention are described as follows.

[0025] According to an embodiment of the present invention, when estimating driver / passenger drowsiness, stress, health information, etc. using multiple cameras, the vital data of the driver / passenger can be collected and provided reliably, seamlessly, and with high accuracy. In other words, reliable input data can be provided for high-level vital analysis.

[0026] FIG. 1 is a drawing illustrating an example of a vehicle related to an embodiment of the present invention.

[0027] FIG. 2 is a drawing of a vehicle related to an embodiment of the present invention viewed from various angles.

[0028] FIGS. 3 and 4 are drawings showing the interior of a vehicle related to an embodiment of the present invention.

[0029] FIG. 5 and FIG. 6 are drawings for reference in explaining various objects related to driving of a vehicle related to an embodiment of the present invention.

[0030] FIG. 7 is a block diagram for reference in explaining a vital data integration device of a vehicle related to an embodiment of the present invention.

[0031] FIG. 8 is a block diagram illustrating how a vital data integration device according to an embodiment of the present invention interacts with other components of a vehicle.

[0032] FIG. 9 is a block diagram illustrating a detailed configuration of a vital data integration device related to an embodiment of the present invention.

[0033] Figure 10 is a representative flowchart for explaining a vital data integration method related to an embodiment of the present invention.

[0034] FIG. 11 is a drawing for explaining collecting vital data using multiple cameras in a vehicle according to an embodiment of the present invention.

[0035] FIG. 12 is an exemplary diagram for explaining data selection using a hard voting method related to an embodiment of the present invention.

[0036] FIGS. 13 and 14 are drawings for explaining examples of assigning different weights to vital data according to a facial recognition algorithm related to an embodiment of the present invention.

[0037] FIG. 15 is a flowchart illustrating a method of generating final input data by selecting and fusing vital data related to an embodiment of the present invention using a soft voting method.

[0038] FIG. 16 and FIG. 17 are block diagrams and flowcharts illustrating obtaining, integrating, evaluating, and selecting a plurality of vital data related to an embodiment of the present invention and providing them to a driver state estimation algorithm.

[0039] FIGS. 1 and 2 are views showing the exterior of a vehicle related to an embodiment of the present invention, and FIGS. 3 and 4 are views showing the interior of a vehicle related to an embodiment of the present invention.

[0040] FIGS. 5 and 6 are drawings illustrating various objects related to driving of a vehicle according to an embodiment of the present invention.

[0041] Figure 7 is a block diagram for reference in explaining a vehicle related to an embodiment of the present invention.

[0042] Referring to FIGS. 1 to 7, the vehicle (100) may include wheels that rotate by a power source and a steering input device (510) for controlling the direction of travel of the vehicle (100).

[0043] The vehicle (100) may be an autonomous vehicle. The vehicle (100) may be switched between an autonomous driving mode and a manual driving mode based on user input. For example, the vehicle (100) may be switched from a manual mode to an autonomous driving mode, or from an autonomous driving mode to a manual mode, based on user input received through a user interface device (hereinafter, referred to as a "user terminal") (200).

[0044] The vehicle (100) can be switched to autonomous driving mode or manual driving mode based on driving situation information. The driving situation information can be generated based on object information provided by the object detection device (300). For example, the vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on the driving situation information generated by the object detection device (300). For example, the vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on driving situation information received through the communication device (400).

[0045] The vehicle (100) can be switched from manual mode to autonomous driving mode or from autonomous driving mode to manual mode based on information, data, and signals provided from an external device.

[0046] When the vehicle (100) is operated in autonomous driving mode, the autonomous vehicle (100) may be operated based on the driving system (700). For example, the autonomous vehicle (100) may be operated based on information, data, or signals generated from the driving system (710), the exit system (740), and the parking system (750).

[0047] When the vehicle (100) is driven in manual mode, the autonomous vehicle (100) can receive user input for driving through the driving control device (500). Based on the user input received through the driving control device (500), the vehicle (100) can be driven.

[0048] The overall length refers to the length from the front to the rear of the vehicle (100), the overall width refers to the width of the vehicle (100), and the overall height refers to the length from the bottom of the wheel to the roof. In the following description, the overall length direction (L) may refer to the direction that serves as a reference for measuring the overall length of the vehicle (100), the overall width direction (W) may refer to the direction that serves as a reference for measuring the overall width of the vehicle (100), and the overall height direction (H) may refer to the direction that serves as a reference for measuring the overall height of the vehicle (100).

[0049] As illustrated in FIG. 7, the vehicle (100) may include a user interface device (hereinafter, referred to as a 'user terminal') (200), an object detection device (300), a communication device (400), a driving operation device (500), a vehicle driving device (600), a driving system (700), a navigation system (770), a sensing unit (120), a vehicle interface unit (130), a memory (140), a control unit (170), and a power supply unit (190).

[0050] Depending on the embodiment, the vehicle (100) may include other components in addition to the components described herein, or may not include some of the components described herein.

[0051] The user interface device (200) is a device for communication between a vehicle (100) and a user. The user interface device (200) can receive user input and provide information generated in the vehicle (100) to the user. The vehicle (100) can implement a UI (User Interfaces) or UX (User Experience) through the user interface device (hereinafter, referred to as a 'user terminal') (200).

[0052] The user interface device (200) may include an input unit (210), an internal camera (220), a biometric detection unit (230), an output unit (250), and a processor (270). Depending on the embodiment, the user interface device (200) may include other components in addition to the described components, or may not include some of the described components.

[0053] The input unit (210) is for receiving information from a user, and data collected from the input unit (210) can be analyzed by a processor (270) and processed into a user's control command.

[0054] The input unit (210) may be placed inside the vehicle. For example, the input unit (210) may be placed in an area of ​​a steering wheel, an area of ​​an instrument panel, an area of ​​a seat, an area of ​​each pillar, an area of ​​a door, an area of ​​a center console, an area of ​​a head lining, an area of ​​a sun visor, an area of ​​a windshield, or an area of ​​a window.

[0055] The input unit (210) may include a voice input unit (211), a gesture input unit (212), a touch input unit (213), and a mechanical input unit (214).

[0056] The voice input unit (211) can convert a user's voice input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170). The voice input unit (211) can include one or more microphones.

[0057] The gesture input unit (212) can convert a user's gesture input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170).

[0058] The gesture input unit (212) may include at least one of an infrared sensor and an image sensor for detecting a user's gesture input. According to an embodiment, the gesture input unit (212) may detect a user's three-dimensional gesture input. To this end, the gesture input unit (212) may include a light output unit that outputs a plurality of infrared lights or a plurality of image sensors.

[0059] The gesture input unit (212) can detect a user's 3D gesture input through a TOF (Time of Flight) method, a structured light method, or a disparity method.

[0060] The touch input unit (213) can convert a user's touch input into an electrical signal. The converted electrical signal can be provided to a processor (270) or a control unit (170).

[0061] The touch input unit (213) may include a touch sensor for detecting a user's touch input. In some embodiments, the touch input unit (213) may be formed integrally with the display unit (251), thereby implementing a touch screen. Such a touch screen may provide both an input interface and an output interface between the vehicle (100) and the user.

[0062] The mechanical input unit (214) may include at least one of a button, a dome switch, a jog wheel, and a jog switch. An electrical signal generated by the mechanical input unit (214) may be provided to a processor (270) or a control unit (170). The mechanical input unit (214) may be placed on a steering wheel, a center fascia, a center console, a cockpit module, a door, etc.

[0063] The internal camera (220) can capture images of the vehicle interior. The processor (270) can detect the user's status based on the images of the vehicle interior. The processor (270) can obtain information about the user's gaze from the images of the vehicle interior. The processor (270) can detect the user's gestures from the images of the vehicle interior.

[0064] The biometric detection unit (230) can obtain the user's biometric information. The biometric detection unit (230) includes a sensor capable of obtaining the user's biometric information, and can use the sensor to obtain the user's fingerprint information, heartbeat information, etc. The biometric information can be used for user authentication.

[0065] The output unit (250) is for generating output related to visual, auditory, or tactile sensations. The output unit (250) may include at least one of a display unit (251), an audio output unit (252), and a haptic output unit (253).

[0066] The display unit (251) can display graphic objects corresponding to various pieces of information. The display unit (251) can include at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.

[0067] The display unit (251) can implement a touch screen by forming a mutual layer structure with the touch input unit (213) or forming it as an integral part.

[0068] The display unit (251) may be implemented as a HUD (Head Up Display). When the display unit (251) is implemented as a HUD, the display unit (251) may be equipped with a projection module to output information through an image projected onto a windshield or window.

[0069] The display unit (251) may include a transparent display. The transparent display may be attached to a windshield or a window. The transparent display may have a predetermined transparency and display a predetermined screen. In order to have transparency, the transparent display may include at least one of a transparent TFEL (Thin Film Electroluminescent), a transparent OLED (Organic Light-Emitting Diode), a transparent LCD (Liquid Crystal Display), a transparent display, and a transparent LED (Light Emitting Diode) display. The transparency of the transparent display may be adjusted.

[0070] Meanwhile, the user interface device (200) may include a plurality of display units (251a to 251g).

[0071] The display unit (251) may be arranged in one area of ​​the steering wheel, one area of ​​the instrument panel (521a, 251b, 251e), one area of ​​the seat (251d), one area of ​​each pillar (251f), one area of ​​the door (251g), one area of ​​the center console, one area of ​​the head lining, one area of ​​the sun visor, or may be implemented in one area of ​​the windshield (251c), one area of ​​the window (251h).

[0072] The audio output unit (252) converts an electric signal provided from the processor (270) or the control unit (170) into an audio signal and outputs the converted signal. To this end, the audio output unit (252) may include one or more speakers.

[0073] The haptic output unit (253) generates a tactile output. For example, the haptic output unit (253) can operate by vibrating a steering wheel, a seat belt, or a seat (110FL, 110FR, 110RL, 110RR) so that the user can perceive the output.

[0074] The processor (hereinafter, referred to as a “control unit”) (270) can control the overall operation of each unit of the user interface device (200). Depending on the embodiment, the user interface device (200) may include a plurality of processors (270) or may not include a processor (270).

[0075] If the user interface device (200) does not include a processor (270), the user interface device (200) may be operated under the control of a processor or control unit (170) of another device in the vehicle (100).

[0076] Meanwhile, the user interface device (200) may be referred to as a vehicle display device. The user interface device (200) may be operated under the control of the control unit (170).

[0077] The object detection device (300) is a device for detecting an object located outside a vehicle (100). The object may be various objects related to the operation of the vehicle (100). Referring to FIGS. 5 and 6, the object (O) may include a lane (OB10), another vehicle (OB11), a pedestrian (OB12), a two-wheeled vehicle (OB13), a traffic signal (OB14, OB15), a light, a road, a structure, a speed bump, a terrain, an animal, etc.

[0078] A lane (OB10) may be a driving lane, a lane adjacent to a driving lane, or a lane in which opposing vehicles drive. A lane (OB10) may be a concept that includes lines on the left and right sides that form a lane.

[0079] Another vehicle (OB11) may be a vehicle driving around the vehicle (100). The other vehicle may be a vehicle located within a predetermined distance from the vehicle (100). For example, the other vehicle (OB11) may be a vehicle preceding or following the vehicle (100).

[0080] A pedestrian (OB12) may be a person located around a vehicle (100). A pedestrian (OB12) may be a person located within a predetermined distance from a vehicle (100). For example, a pedestrian (OB12) may be a person located on a sidewalk or roadway.

[0081] A two-wheeled vehicle (OB12) may refer to a vehicle that is positioned around a vehicle (100) and moves using two wheels. The two-wheeled vehicle (OB12) may be a vehicle with two wheels that is positioned within a predetermined distance from the vehicle (100). For example, the two-wheeled vehicle (OB13) may be a motorcycle or bicycle positioned on a sidewalk or roadway.

[0082] Traffic signals may include traffic lights (OB15), traffic signs (OB14), and patterns or text painted on the road surface.

[0083] The light may be generated from a lamp installed in another vehicle. The light may be generated from a streetlight. The light may be sunlight.

[0084] A road may include slopes such as road surfaces, curves, uphill and downhill slopes, etc.

[0085] Structures may be objects located along roads and fixed to the ground. For example, structures may include streetlights, street trees, buildings, utility poles, traffic lights, and bridges.

[0086] Landforms may include mountains, hills, etc.

[0087] Meanwhile, objects can be classified into moving objects and fixed objects. For example, moving objects may include concepts such as other vehicles and pedestrians. For example, fixed objects may include concepts such as traffic signals, roads, and structures.

[0088] The object detection device (300) may include a camera (310), a radar (320), a lidar (330), an ultrasonic sensor (340), an infrared sensor (350), and a processor (370).

[0089] Depending on the embodiment, the object detection device (300) may include other components in addition to the described components, or may not include some of the described components.

[0090] The camera (310) may be positioned at an appropriate location outside the vehicle to capture images of the vehicle's exterior. The camera (310) may be a mono camera, a stereo camera (310a), an AVM (Around View Monitoring) camera (310b), or a 360-degree camera.

[0091] For example, the camera (310) may be positioned inside the vehicle, close to the front windshield, to capture an image of the front of the vehicle. Alternatively, the camera (310) may be positioned around the front bumper or radiator grill.

[0092] For example, the camera (310) may be positioned inside the vehicle, close to the rear glass, to capture images of the rear of the vehicle. Alternatively, the camera (310) may be positioned around the rear bumper, trunk, or tailgate.

[0093] For example, the camera (310) may be positioned close to at least one of the side windows inside the vehicle to obtain an image of the side of the vehicle. Alternatively, the camera (310) may be positioned around a side mirror, fender, or door.

[0094] The camera (310) can provide the acquired image to the processor (370).

[0095] The radar (320) may include an electromagnetic wave transmitter and receiver. The radar (320) may be implemented in a pulse radar or continuous wave radar manner based on the principle of radio wave emission. Among continuous wave radar methods, the radar (320) may be implemented in a frequency modulated continuous wave (FMCW) manner or a frequency shift keying (FSK) manner depending on the signal waveform.

[0096] The radar (320) can detect an object using electromagnetic waves, based on a TOF (Time of Flight) method or a phase-shift method, and can detect the location of the detected object, the distance to the detected object, and the relative speed.

[0097] The radar (320) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0098] The lidar (330) may include a laser transmitter and receiver. The lidar (330) may be implemented using a TOF (Time of Flight) method or a phase-shift method.

[0099] The lidar (330) can be implemented as a driven or non-driven type.

[0100] When implemented as a drive type, the lidar (330) is rotated by a motor and can detect objects around the vehicle (100).

[0101] When implemented in a non-driven manner, the lidar (330) can detect an object located within a predetermined range relative to the vehicle (100) through optical steering. The vehicle (100) can include a plurality of non-driven lidars (330).

[0102] Lidar (330) can detect an object based on a time-of-flight (TOF) method or a phase-shift method using laser light as a parameter, and can detect the position of the detected object, the distance to the detected object, and the relative speed.

[0103] The lidar (330) can be placed at an appropriate location outside the vehicle to detect objects located in front, behind, or to the side of the vehicle.

[0104] The ultrasonic sensor (340) may include an ultrasonic transmitter and a receiver. The ultrasonic sensor (340) may detect an object based on ultrasonic waves, and may detect the location of the detected object, the distance from the detected object, and the relative speed.

[0105] The ultrasonic sensor (340) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0106] The infrared sensor (350) may include an infrared transmitter and a receiver. The infrared sensor (340) may detect an object based on infrared light, and may detect the location of the detected object, the distance to the detected object, and the relative speed.

[0107] The infrared sensor (350) can be placed at an appropriate location outside the vehicle to detect objects located in front, rear, or to the side of the vehicle.

[0108] The processor (370) can control the overall operation of each unit of the object detection device (300).

[0109] The processor (370) can detect and track an object based on the acquired image. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object through an image processing algorithm.

[0110] The processor (370) can detect and track an object based on the reflected electromagnetic waves that are returned when the transmitted electromagnetic waves are reflected by the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the electromagnetic waves.

[0111] The processor (370) can detect and track an object based on the reflected laser light that is reflected back by the transmitted laser beam from the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the laser light.

[0112] The processor (370) can detect and track an object based on the reflected ultrasonic waves that are returned when the transmitted ultrasonic waves are reflected off the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the ultrasonic waves.

[0113] The processor (370) can detect and track an object based on the reflected infrared light that is reflected back by the transmitted infrared light from the object. The processor (370) can perform operations such as calculating the distance to the object and calculating the relative speed with the object based on the infrared light.

[0114] Depending on the embodiment, the object detection device (300) may include multiple processors (370) or may not include a processor (370). For example, each of the camera (310), radar (320), lidar (330), ultrasonic sensor (340), and infrared sensor (350) may individually include a processor.

[0115] If the object detection device (300) does not include a processor (370), the object detection device (300) can be operated under the control of the processor or control unit (170) of the device in the vehicle (100).

[0116] The object detection device (400) can be operated under the control of the control unit (170).

[0117] The communication device (400) is a device for communicating with an external device. Here, the external device may be another vehicle, a mobile terminal, or a server.

[0118] The communication device (400) may include at least one of a transmitting antenna, a receiving antenna, an RF (Radio Frequency) circuit capable of implementing various communication protocols, and an RF element to perform communication.

[0119] The communication device (400) may include a short-range communication unit (410), a location information unit (420), a V2X communication unit (430), an optical communication unit (440), a broadcast transmission / reception unit (450), and a processor (470).

[0120] Depending on the embodiment, the communication device (400) may include additional components other than the described components, or may not include some of the described components.

[0121] The short-range communication unit (410) is a unit for short-range communication. The short-range communication unit (410) can support short-range communication using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

[0122] The short-range communication unit (410) can form a short-range wireless communication network (Wireless Area Network) to perform short-range communication between the vehicle (100) and at least one external device.

[0123] The location information unit (420) is a unit for obtaining location information of a vehicle (100). For example, the location information unit (420) may include a GPS (Global Positioning System) module or a DGPS (Differential Global Positioning System) module.

[0124] The V2X communication unit (430) is a unit for performing wireless communication with a server (V2I: Vehicle to Infrastructure), another vehicle (V2V: Vehicle to Vehicle), or a pedestrian (V2P: Vehicle to Pedestrian). The V2X communication unit (430) may include an RF circuit capable of implementing protocols for communication with infrastructure (V2I), communication between vehicles (V2V), and communication with pedestrians (V2P).

[0125] The optical communication unit (440) is a unit for communicating with an external device via light. The optical communication unit (440) may include an optical transmission unit that converts an electrical signal into an optical signal and transmits it to the outside, and an optical reception unit that converts a received optical signal into an electrical signal.

[0126] According to an embodiment, the light transmitting unit may be formed to be integrated with a lamp included in the vehicle (100).

[0127] The broadcast transmitter / receiver (450) is a unit for receiving broadcast signals from an external broadcast management server via a broadcast channel, or transmitting broadcast signals to the broadcast management server. The broadcast channels may include satellite channels and terrestrial channels. The broadcast signals may include TV broadcast signals, radio broadcast signals, and data broadcast signals.

[0128] The processor (470) can control the overall operation of each unit of the communication device (400).

[0129] Depending on the embodiment, the communication device (400) may include a plurality of processors (470) or may not include a processor (470).

[0130] If the communication device (400) does not include a processor (470), the communication device (400) may be operated under the control of a processor or control unit (170) of another device in the vehicle (100).

[0131] Meanwhile, the communication device (400) may implement a vehicle display device together with the user interface device (200). In this case, the vehicle display device may be referred to as a telematics device or an AVN (Audio Video Navigation) device.

[0132] The communication device (400) can be operated under the control of the control unit (170).

[0133] The driving control device (500) is a device that receives user input for driving.

[0134] When in manual mode, the vehicle (100) can be driven based on signals provided by the driving control device (500).

[0135] The driving control device (500) may include a steering input device (510), an acceleration input device (530), and a brake input device (570).

[0136] The steering input device (510) can receive input for the direction of travel of the vehicle (100) from the user. The steering input device (510) is preferably formed in the form of a wheel so that steering input can be provided by rotation. Depending on the embodiment, the steering input device may be formed in the form of a touch screen, a touch pad, or a button.

[0137] The acceleration input device (530) can receive an input from a user for accelerating the vehicle (100). The brake input device (570) can receive an input from a user for decelerating the vehicle (100). The acceleration input device (530) and the brake input device (570) are preferably formed in the form of a pedal. Depending on the embodiment, the acceleration input device or the brake input device may also be formed in the form of a touch screen, a touch pad, or a button.

[0138] The driving operation device (500) can be operated under the control of the control unit (170).

[0139] The vehicle driving device (600) is a device that electrically controls the driving of various devices in the vehicle (100).

[0140] The vehicle driving device (600) may include a power train driving unit (610), a chassis driving unit (620), a door / window driving unit (630), a safety device driving unit (640), a lamp driving unit (650), and an air conditioning driving unit (660).

[0141] Depending on the embodiment, the vehicle drive device (600) may include additional components other than the described components, or may not include some of the described components.

[0142] Meanwhile, the vehicle driving device (600) may include a processor. Each unit of the vehicle driving device (600) may individually include a processor.

[0143] The power train drive unit (610) can control the operation of the power train device.

[0144] The power train drive unit (610) may include a power source drive unit (611) and a transmission drive unit (612).

[0145] The power source driving unit (611) can perform control over the power source of the vehicle (100).

[0146] For example, if a fossil fuel-based engine is the power source, the power source drive unit (610) can perform electronic control of the engine. This can control the engine output torque, etc. The power source drive unit (611) can adjust the engine output torque according to the control of the control unit (170).

[0147] For example, if an electric energy-based motor is the power source, the power source driving unit (610) can perform control over the motor. The power source driving unit (610) can adjust the rotation speed, torque, etc. of the motor according to the control of the control unit (170).

[0148] The transmission drive unit (612) can perform control over the transmission. The transmission drive unit (612) can adjust the state of the transmission. The transmission drive unit (612) can adjust the state of the transmission to forward (D), reverse (R), neutral (N), or parking (P).

[0149] Meanwhile, when the engine is the power source, the transmission drive unit (612) can adjust the gear engagement state in the forward (D) state.

[0150] The chassis drive unit (620) can control the operation of the chassis device. The chassis drive unit (620) can include a steering drive unit (621), a brake drive unit (622), and a suspension drive unit (623).

[0151] The steering drive unit (621) can perform electronic control of the steering apparatus within the vehicle (100). The steering drive unit (621) can change the direction of travel of the vehicle.

[0152] The brake drive unit (622) can perform electronic control of the brake apparatus within the vehicle (100). For example, the speed of the vehicle (100) can be reduced by controlling the operation of the brakes placed on the wheels.

[0153] Meanwhile, the brake driving unit (622) can individually control each of the plurality of brakes. The brake driving unit (622) can control the braking force applied to the plurality of wheels differently.

[0154] The suspension drive unit (623) can perform electronic control of the suspension apparatus within the vehicle (100). For example, when there is a curve in the road surface, the suspension drive unit (623) can control the suspension apparatus to reduce vibration of the vehicle (100). Meanwhile, the suspension drive unit (623) can individually control each of the plurality of suspensions.

[0155] The door / window actuator (630) can perform electronic control of a door apparatus or window apparatus in a vehicle (100).

[0156] The door / window driving unit (630) may include a door driving unit (631) and a window driving unit (632).

[0157] The door driving unit (631) can control the door device. The door driving unit (631) can control the opening and closing of a plurality of doors included in the vehicle (100). The door driving unit (631) can control the opening or closing of a trunk or tail gate. The door driving unit (631) can control the opening or closing of a sunroof.

[0158] The window driving unit (632) can perform electronic control of a window apparatus. It can control the opening or closing of a plurality of windows included in a vehicle (100).

[0159] The safety device driving unit (640) can perform electronic control of various safety devices in the vehicle (100).

[0160] The safety device drive unit (640) may include an airbag drive unit (641), a seat belt drive unit (642), and a pedestrian protection device drive unit (643).

[0161] The airbag driving unit (641) can perform electronic control of the airbag apparatus within the vehicle (100). For example, the airbag driving unit (641) can control the airbag to deploy when a danger is detected.

[0162] The seat belt drive unit (642) can perform electronic control of the seat belt apparatus within the vehicle (100). For example, the seat belt drive unit (642) can control the passenger to be secured to the seat (110FL, 110FR, 110RL, 110RR) using the seat belt when a danger is detected.

[0163] The pedestrian protection device drive unit (643) can perform electronic control of the hood lift and pedestrian airbag. For example, the pedestrian protection device drive unit (643) can control the hood lift up and the pedestrian airbag to deploy when a collision with a pedestrian is detected.

[0164] The lamp driving unit (650) can perform electronic control of various lamp apparatuses within the vehicle (100).

[0165] The air conditioning drive unit (660) can perform electronic control of the air conditioning device (air cinditioner) within the vehicle (100). For example, the air conditioning drive unit (660) can control the air conditioning device to operate and supply cool air to the vehicle when the temperature inside the vehicle is high.

[0166] The vehicle driving device (600) may include a processor. Each unit of the vehicle driving device (600) may individually include a processor.

[0167] The vehicle driving device (600) can be operated under the control of the control unit (170).

[0168] The driving system (700) is a system that controls various operations of the vehicle (100). The driving system (700) can be operated in autonomous driving mode.

[0169] The driving system (700) may include a driving system (710), an exiting system (740), and a parking system (750).

[0170] Depending on the embodiment, the driving system (700) may include other components in addition to the described components, or may not include some of the described components.

[0171] Meanwhile, the driving system (700) may include a processor. Each unit of the driving system (700) may individually include a processor.

[0172] Meanwhile, depending on the embodiment, if the driving system (700) is implemented in software, it may be a sub-concept of the control unit (170).

[0173] Meanwhile, according to an embodiment, the driving system (700) may be a concept including at least one of a user interface device (200), an object detection device (300), a communication device (400), a vehicle driving device (600), and a control unit (170).

[0174] The driving system (710) can drive the vehicle (100).

[0175] The driving system (710) can receive navigation information from the navigation system (770) and provide a control signal to the vehicle driving device (600) to drive the vehicle (100). The driving system (710) can receive object information from the object detection device (300) and provide a control signal to the vehicle driving device (600) to drive the vehicle (100). The driving system (710) can receive a signal from an external device through the communication device (400) and provide a control signal to the vehicle driving device (600) to drive the vehicle (100).

[0176] The exit system (740) can perform exit of a vehicle (100).

[0177] The exit system (740) can receive navigation information from the navigation system (770) and provide a control signal to the vehicle driving device (600) to perform exit of the vehicle (100). The exit system (740) can receive object information from the object detection device (300) and provide a control signal to the vehicle driving device (600) to perform exit of the vehicle (100). The exit system (740) can receive a signal from an external device through the communication device (400) and provide a control signal to the vehicle driving device (600) to perform exit of the vehicle (100).

[0178] The parking system (750) can perform parking of a vehicle (100).

[0179] The parking system (750) can receive navigation information from the navigation system (770) and provide a control signal to the vehicle driving device (600) to perform parking of the vehicle (100). The parking system (750) can receive object information from the object detection device (300) and provide a control signal to the vehicle driving device (600) to perform parking of the vehicle (100). The parking system (750) can receive a signal from an external device through the communication device (400) and provide a control signal to the vehicle driving device (600) to perform parking of the vehicle (100).

[0180] A navigation system (770) can provide navigation information. The navigation information can include at least one of map information, set destination information, route information based on the set destination, information on various objects along the route, lane information, and current vehicle location information.

[0181] The navigation system (770) may include memory and a processor. The memory may store navigation information. The processor may control the operation of the navigation system (770).

[0182] According to an embodiment, the navigation system (770) may receive information from an external device via the communication device (400) and update previously stored information.

[0183] Depending on the embodiment, the navigation system (770) may be classified as a subcomponent of the user interface device (200).

[0184] The sensing unit (120) can sense the status of the vehicle. The sensing unit (120) can include a posture sensor (e.g., a yaw sensor, a roll sensor, a pitch sensor), a collision sensor, a wheel sensor, a speed sensor, an inclination sensor, a weight detection sensor, a heading sensor, a yaw sensor, a gyro sensor, a position module, a vehicle forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor by steering wheel rotation, a vehicle interior temperature sensor, a vehicle interior humidity sensor, an ultrasonic sensor, an illuminance sensor, an accelerator pedal position sensor, a brake pedal position sensor, etc.

[0185] The sensing unit (120) can obtain sensing signals for vehicle attitude information, vehicle collision information, vehicle direction information, vehicle location information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle inclination information, vehicle forward / backward information, battery information, fuel information, tire information, vehicle lamp information, vehicle internal temperature information, vehicle internal humidity information, steering wheel rotation angle, vehicle external illumination, pressure applied to an accelerator pedal, pressure applied to a brake pedal, etc.

[0186] The sensing unit (120) may further include, in addition, an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), etc.

[0187] The vehicle interface unit (130) can serve as a conduit for various types of external devices connected to the vehicle (100). For example, the vehicle interface unit (130) may be equipped with a port capable of connecting to a mobile terminal, and may be connected to the mobile terminal through the port. In this case, the vehicle interface unit (130) can exchange data with the mobile terminal.

[0188] Meanwhile, the vehicle interface unit (130) may serve as a conduit for supplying electrical energy to a connected mobile terminal. When the mobile terminal is electrically connected to the vehicle interface unit (130), the vehicle interface unit (130) may provide the mobile terminal with electrical energy supplied from the power supply unit (190) under the control of the control unit (170).

[0189] The memory (140) is electrically connected to the control unit (170). The memory (140) can store basic data for the unit, control data for controlling the operation of the unit, and input / output data. The memory (140) can be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc. in terms of hardware. The memory (140) can store various data for the overall operation of the vehicle (100), such as programs for processing or controlling the control unit (170).

[0190] Depending on the embodiment, the memory (140) may be formed integrally with the control unit (170) or implemented as a sub-component of the control unit (170).

[0191] The control unit (170) can control the overall operation of each unit within the vehicle (100). The control unit (170) can be referred to as an ECU (Electronic Control Unit).

[0192] The power supply unit (190) can supply power required for the operation of each component under the control of the control unit (170). In particular, the power supply unit (190) can receive power from a battery or the like inside the vehicle.

[0193] One or more processors and control units (170) included in the vehicle (100) may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0194] Meanwhile, in embodiments of the present invention, "driver," "passenger," and "driver or passenger (driver / passenger)" refer to a driver, passenger, co-passenger, or passenger riding in a vehicle capable of performing personal driving control according to embodiments of the present invention, and may be one or more. In addition, in this specification, "driver" includes a passenger riding in the driver's seat while the vehicle is performing autonomous driving.

[0195] Additionally, in embodiments of the present invention, the term 'vehicle' is used to mean a vehicle that runs according to a driver's operation, a vehicle capable of autonomous driving, or a vehicle that can receive such services.

[0196] Meanwhile, as illustrated in FIG. 7, a vital data integration device (800) according to an embodiment of the present invention may be provided in a vehicle (100).

[0197] In some embodiments, the vital data integration device (800) may be in the form of a service platform that provides vital data integration services. In such cases, the vehicle (100) may receive vital data integration services from the service platform via a network or the like.

[0198] The vital data integration device (800) receives vital data of multiple drivers / passengers collected through each of multiple cameras installed at different locations within the vehicle (100). The vital data integration device (800) can receive multiple time-synchronized vital data in the form of a single stream.

[0199] The vital data integration device (800) can evaluate the reliability of each of the received vital data sets based on preset criteria and assign different weights to each vital data set accordingly. The preset criteria can be determined based on one or more of the performance of the multiple cameras, the results of the facial recognition algorithms of each of the multiple cameras, and the performance of the state estimation algorithm to be applied.

[0200] The vital data integration device (800) can select and fuse vital data based on assigned weights, ultimately generating input data for high-level data analysis. Here, the input data for high-level data analysis may refer to input values ​​of an algorithm that estimates the driver / passenger's condition index.

[0201] The vital data integration device (800) provides the final generated input data to a driver / passenger status estimation algorithm, thereby enabling the acquisition of more reliable status index results. The driver / passenger status estimation algorithm may include, for example, a drowsiness estimation algorithm, a stress assessment algorithm, an emergency status estimation (assessment) algorithm, etc.

[0202] In this way, the vital data integration device (800) receives multiple vital data from various angles for the driver / passenger in the vehicle (100), evaluates the reliability and assigns weights, and selects and fuses the vital data based on the reliability to provide a more reliable input value to the status estimation algorithm.

[0203] FIG. 8 is a block diagram illustrating how a vital data integration device (800) related to an embodiment of the present invention interacts with other components of a vehicle.

[0204] The vital data integration device (800) can communicate with multiple cameras (220), a motion detection module (235), a vehicle control unit (170), and an ADAS system (300) within the vehicle. The vital data integration device (800) communicates with these devices to acquire multiple vital data about the driver / passenger. In addition, the vital data integration device (800) communicates with these devices to perform a reliability evaluation on the acquired multiple vital data and reflects the information in the reliability evaluation.

[0205] The vital data integration device (800) can communicate with other components within the vehicle in addition to the components illustrated in FIG. 8. In addition, the vital data integration device (880) can additionally receive network data and the like from an external source.

[0206] The vital data integration device (800) can receive different vital data from multiple cameras (220) in the vehicle, for example, a first camera (221), a second camera (222), and a third camera (223).

[0207] The plurality of cameras (220) may be installed at different locations within the vehicle. For example, the first camera (221) may be an IR camera installed in the vehicle's rearview mirror, the second camera (222) may be an A-Pillar camera, and the third camera (223) may be an IR camera installed in the cluster. However, the plurality of cameras (220) are not limited to these examples, and may be installed at different locations within the vehicle (100) and may include more than three cameras.

[0208] Each of the plurality of cameras (220) acquires vital data of the driver / passenger using the rPPG (Remote PPG) method. For example, each of the plurality of cameras (220) can acquire a facial image of the driver / passenger. Then, a facial recognition algorithm is applied to each acquired facial image, and based on the measured blood flow change, the heart rate data of the driver / passenger can be collected using the rPPG (Remote PPG) method. At this time, it should be noted that the plurality of vital data acquired by each of the plurality of cameras (220) are heart rate data for the same person.

[0209] The vital data integration device (800) can receive detection results and / or control results from at least one of a motion detection module (235), an ADAS system (300), and a vehicle control unit (170), and can use this to evaluate the reliability of the vital data.

[0210] Reliability assessment of vital data is quantitatively performed by considering the accuracy, recency, consistency, etc. of each vital data. For example, if the first camera (221) provides data with high accuracy and low recency, the second camera (222) provides data with somewhat low accuracy and high recency, and the third camera (223) provides data with average accuracy and recency, the reliability assessment results for each vital data collected by these cameras may be assigned different weights.

[0211] Meanwhile, the vital data integration device (800) can reflect the detection result and / or control result of at least one of the motion detection module (235), ADAS system (300), and vehicle control unit (170) in the vehicle (100) in the reliability evaluation.

[0212] In an embodiment, the vital data integration device (800) may change the weight for the received vital data based on the detection results and / or control results received from the motion detection module (235), the ADAS system (300), and / or the vehicle control unit (170).

[0213] For example, if an event is detected by the ADAS system (300) and sudden braking is applied without driver intervention, the heart rate of the driver / passenger will increase, so a higher weight can be temporarily given to vital data.

[0214] In another embodiment, the vital data integration device (800) can change the weight calculation cycle to be applied to the received vital data based on the detection results and / or control results received from the motion detection module (235), the ADAS system (300), and / or the vehicle control unit (170).

[0215] For example, if the control result of the vehicle control unit (170) remains the same or has only small changes for a long period of time, the driver's heart rate may decrease due to drowsiness, etc., so in this case, the weight calculation cycle for vital data may be changed to be short to perform drowsiness prevention service response.

[0216] In another embodiment, the vital data integration device (800) can adjust the degree of change in weight or algorithm sensitivity for vital data based on detection results and / or control results received from the motion detection module (235), the ADAS system (300), and / or the vehicle control unit (170).

[0217] For example, while the driver / passenger's movement is significantly detected based on the detection result of the motion detection module (235), the heart rate will increase due to the driver / passenger's body movement, so at this time, the weight change degree of the entire vital data or the algorithm sensitivity can be applied by lowering it.

[0218] Meanwhile, in some embodiments, in addition to the configuration within the vehicle (100) illustrated in FIG. 8, the reliability evaluation of vital data by the vital data integration device (800) may vary based on the detection results and / or control results of other configurations within the vehicle (100), network data provided from outside the vehicle (100), etc. For example, the reliability evaluation of vital data for the current driver / passenger may vary based on traffic situation information, weather information, health status information received from the driver / passenger's terminal, etc.

[0219] Once the reliability assessment results are weighted, the weighted vital data is selected and fused, and the resulting data is input into an algorithm for estimating the driver / passenger's condition. Since the resulting data is reliable heart rate data, it can more accurately estimate the driver / passenger's condition.

[0220] Below, FIG. 9 is a block diagram for explaining the detailed configuration of a vital data integration device (800) related to an embodiment of the present invention.

[0221] Referring to FIG. 9, the vital data integration device (800) according to the present invention may include a receiving unit (810), a data determination unit (820), and a data provision unit (830). Here, the data determination unit (820) may have the same meaning as a control unit / processor. In addition, the data provision unit (830) may be a transmission unit / transceiver for transmitting the final generated data to a driver / passenger status estimation algorithm.

[0222] The receiving unit (810) can receive, as vital data, heart rate data based on rPPG (Remote PPG) measured from facial images acquired from multiple cameras installed at different locations within the vehicle (100).

[0223] The received vital data is multiple heart rate data sensed for the same person. Multiple cameras installed in different locations within the vehicle (100) are used as an example to acquire vital data for the same person using the rPPG (Remote PPG) method, but the present invention is not limited to this data acquisition method.

[0224] Each vital data received by the receiver (810) has different values ​​and precisions. The receiver (810) provides each vital data to the data determination unit (820), and the data determination unit (820) performs data selection and integration to provide a highly reliable driver status determination service.

[0225] The data decision unit (820) evaluates the reliability of the received vital data and assigns weights to it. Furthermore, the data decision unit (820) selects highly reliable vital data or fuses multiple vital data based on the weights assigned to each vital data.

[0226] The data determination unit (820) can evaluate the reliability of each vital data based on landmark information included in the driver / passenger's facial image, the reliability of the logic, the amount of change compared to previous data, the reliability of each of the multiple cameras, etc. At this time, the reliability evaluation result is expressed as a weight value.

[0227] The data decision unit (820) can select or fuse one or more of the received vital data based on landmark information included in the driver / passenger's facial image, logic reliability, changes compared to previous data, and the reliability of each of the multiple cameras. For example, vital data with outliers can be removed or missing vital data can be ignored.

[0228] The data provision unit (830) receives vital data selected or fused from the data determination unit (820).

[0229] The data provider (830) provides the final generated vital data to a condition index estimation algorithm (840) for high-level vital analysis, so that the condition index of the driver / passenger can be estimated based on the selected or fused vital data. Here, the condition index of the driver / passenger may mean one of the driver / passenger's stress index, drowsiness index, and fatigue index.

[0230] The condition index estimation (840) uses the final generated vital data as input data for each condition estimation algorithm to evaluate the driver / passenger's fatigue index, stress index, and emergency status. Based on the calculated fatigue index, stress index, and emergency status evaluation, various operations and services related to the driving and safety of the vehicle (100) can be provided to the vehicle (100).

[0231] Figure 10 is a representative flowchart for explaining a vital data integration method related to an embodiment of the present invention.

[0232] Each step illustrated in FIG. 10 may correspond to commands performed by a processor of a computing device, and in this case, the vital data integration method may be implemented in the form of a downloadable computer program or a computer program recording medium including commands.

[0233] Each step of the flowchart of FIG. 10 is performed by the vital data integration device (800) described above, and more specifically, by one of the receiving unit (810), data determining unit (820), and data providing unit (830) described in FIG. 9, unless otherwise described.

[0234] The vital data integration device (800) receives each vital data sensed by multiple cameras installed in the vehicle (S10).

[0235] Multiple cameras installed within the vehicle each capture heart rate vital data for the same driver / passenger using the rPPG (Remote Photoplethysmography) method. The rPPG (Remote Photoplethysmography) method is a non-contact method for acquiring biosignals. Specifically, it extracts skin pixels from facial images captured by the cameras, then analyzes skin color changes caused by blood flow in response to heartbeats to obtain heart rate data.

[0236] When vital data is received in this way, the vital data integration device (800) evaluates the reliability of the received vital data and assigns a weight to it (S20).

[0237] In an embodiment, the step of assigning weights (S20) may include a step of evaluating the reliability of each vital data and determining different weights based on at least one of the reliability of each of the plurality of cameras in the vehicle, landmark information of images sensed through the plurality of cameras, the area occupied by the face in each image, and the amount of change compared to the previous vital data.

[0238] For example, different weights can be assigned to each camera based on the accuracy of the facial recognition algorithm used by each camera. Furthermore, for example, data based on images with many landmarks or a large occupied area can be given a higher weight. Alternatively, for example, data from images where the angle between the face and the camera is close to a right angle can be given a higher weight. Furthermore, for example, data from images with a high accuracy score among the facial recognition algorithm results can be given a higher weight. Furthermore, for example, data with high continuity and similarity can be given a higher weight by comparing the amount of change with previous vital data.

[0239] In an embodiment, the step of assigning weights (S20) may include a step of additionally adjusting the reliability weights for each vital data based on the vehicle control information and the passenger's movement information.

[0240] Here, vehicle control information refers to control information for controlling vehicle braking and steering. Braking and steering include both driver-initiated braking, acceleration, and steering, which occur without driver intervention through the ADAS system.

[0241] Also, here, the passenger movement information refers to the result of driver and passenger movement monitoring based on the operation result of the camera-based motion monitoring module.

[0242] In this way, after different weights are assigned to vital data, the vital data integration device (800) selects or fuses the vital data based on the assigned weights (S30).

[0243] Selection and fusion of vital data can be performed through one of the following methods: 1) data fusion, 2) soft voting, or 3) a combination of soft voting and data fusion.

[0244] 1) Data fusion removes abnormal data from multiple sensed vital data sets, then integrates each vital data value into a single data stream. Here, abnormal data refers to values ​​that differ from all other data sets or exhibit significant variation.

[0245] 2) The soft voting method averages the probability values ​​for each sensed vital data point and calculates a weighted probability for each. Among the multiple values ​​generated, the label with the highest probability is ultimately selected.

[0246] 3) The fusion of soft voting and data fusion adds a data fusion method to the aforementioned soft voting. Specifically, this method averages the probability values ​​for each sensed vital data point, calculates a weighted probability for each, and then performs an additional data fusion method that takes the weighted average based on each value.

[0247] The vital data integration device (800) selects and fuses vital data through the various methods described above to generate reliable final vital data.

[0248] Next, the vital data integration device (800) can estimate the driver's condition index by providing the final generated vital data as input data for condition index estimation (840, Fig. 9) (S40).

[0249] At this time, the final generated vital data may be provided simultaneously or sequentially to multiple different condition index algorithms, thereby providing various condition indices for the same person as result values.

[0250] In some embodiments, the vital data integration device (800) may provide the final generated vital data to a device / server / cloud, etc., including various health index algorithms. In such cases, the health index estimation (840) may indicate that the vital data integration device (800) exists externally. The health index result may then be transmitted to the vital data integration device (800) or the vehicle (100) to track the driver / passenger's condition and / or provide related services.

[0251] As described above, the vital data integration method according to an embodiment of the present invention provides seamless and stable data collection for driver / passenger status analysis and tracking. In other words, reliable input data can be generated and provided for high-level vital data analysis.

[0252] FIG. 11 is a drawing for explaining collecting vital data using multiple cameras in a vehicle according to an embodiment of the present invention.

[0253] Referring to Fig. 11, an example of acquiring a driver's vital data is provided using three cameras around the driver's seat in a vehicle (100). As illustrated, the plurality of cameras may include an IR camera (221, first camera) installed in a room mirror, an A-Pillar camera (222, second camera) on the driver's seat window side, and an IR camera (223, third camera) installed in a cluster.

[0254] However, the number and location of the cameras illustrated in Figure 11 may be variably applied. However, the point of acquiring, selecting, and merging multiple vital data for the same person through multiple cameras remains the same.

[0255] Meanwhile, each of the multiple cameras (221, 222, 223) acquires vital data on the driver / passenger using the rPPG (Remote PPG) method. However, this method is not limited to this method, and camera-based vital data acquisition using other methods is also possible.

[0256] In Fig. 11, each of a plurality of cameras (221, 222, 223) is installed around the driver's seat in the vehicle and senses the driver's vital data using the rPPG (Remote PPG) method. Different vital data are sensed simultaneously (or with a short time difference) by each of the plurality of cameras (221, 222, 223).

[0257] Specifically, each of the plurality of cameras (221, 222, 223) senses heart rate data according to changes in blood flow from the face image of the driver / passenger sitting in the driver's seat of the vehicle and transmits the data to the vehicle's vital data integration device (800). Each of the plurality of sensed vital data has different values ​​and different precisions.

[0258] The sensed plurality of vital data are time-synchronized in the receiving unit (810, FIG. 9) of the vital data integration device (800) and transmitted to the data determination unit (820).

[0259] As mentioned above, multiple vital data sensed at the same time have different values ​​and precisions, so data selection and fusion (integration) are required for a reliable driver status judgment service.

[0260] Figure 12 is an exemplary diagram illustrating data selection using a hard voting method related to an embodiment of the present invention. Hard voting is also called majority voting, as it selects the result with the most votes from each model.

[0261] Referring to Fig. 12, when a new driver state factor (new instance) (1200) is input to each of a plurality of means for sensing a driver's state, for example, a plurality of cameras (1201, 1202, 1203, 1204), a prediction for the new factor is sensed by each of the plurality of cameras (1201, 1202, 1203, 1204). As a result, let's say that the sensing results of three cameras (1201, 1202, 1204) are '1', and the sensing result of a specific camera (1203) is predicted to be '2', which is different from these. Then, '1' is selected as the final data (1210) according to majority vote, that is, ensemble's prediction.

[0262] Meanwhile, the soft voting employed in the present invention produces more accurate and flexible results than the aforementioned hard voting. Soft voting is also called probability voting. Specifically, soft voting does not ignore the sensing results of a specific camera (1203) in the example of FIG. 12, but instead returns results as probability values.

[0263] Specifically, let's say there are three camera sensors measuring the driver's heart rate, each measuring the driver's heart rate every second. In this case, the soft voting method determines the driver's heart rate through the following process.

[0264] Each of the three camera sensors transmits the measured values ​​and time information reliability together. The vital data integration device (800) temporally synchronizes the heart rate data transmitted from each camera and integrates them into a single data stream.

[0265] For example, let's say that camera A measured 80 bpm at 10:00:01, camera B measured 82 bpm at 10:00:02, and camera C measured 78 bpm at 10:00:03. The vital data integration device (800) creates a single data stream from these, and removes any previously missing or outlier values.

[0266] Specifically, the data determination unit (820) of the vital data integration device (800) removes data with outliers from among multiple vital data, and then evaluates the reliability of normal vital data and assigns weights to each. At this time, the reliability evaluation is quantitatively calculated by considering data accuracy, recency, consistency, etc.

[0267] For example, in the example above, if camera A provides data with high accuracy and low recency, camera sensor B provides data with low accuracy and high recency, and camera C provides data with average accuracy and recency, then different weights can be given to each of them.

[0268] After different weights are assigned in this way, the vital data integration device (800) can select the optimal data or average value of each vital data according to reliability (data fusion), combine them (fusion) to calculate a weight probability value, and then select the label with the highest probability value (soft voting), or select the average value thereof (soft voting + data fusion).

[0269] Reliable data generated in this way is provided to the driver / passenger condition index estimation (840, Fig. 9), so that more accurate and reliable condition index results can be obtained.

[0270] FIGS. 13 and 14 are drawings for explaining examples of assigning different weights to vital data according to a facial recognition algorithm related to an embodiment of the present invention.

[0271] In an embodiment of the present invention, reliability evaluation of vital data (e.g., heart rate data) received from a plurality of cameras is performed by determining a weight for each vital data based on at least one of the reliability of each of the plurality of cameras, landmark information of images sensed through the plurality of cameras, area occupied by a face in the images, reliability of logic, and amount of change compared to previous vital data.

[0272] The vital data sensed by each of the multiple cameras are acquired at the same time but have different values ​​and precisions. Therefore, data selection and fusion (integration) are necessary to provide a reliable driver status judgment service.

[0273] As a data processing method for selecting and fusing weighted vital data, the present invention uses a data fusion method and a soft voting method.

[0274] Data Fusion is a data processing method that obtains more accurate and flexible results by applying weights to the results of applying a facial recognition algorithm to facial images sensed by multiple cameras and integrating each value into a single data stream.

[0275] The Soft Voting method is a data processing method that applies weights to the results of applying a facial recognition algorithm to facial images sensed by multiple cameras, then averages the probability values ​​of each, and ultimately selects the label with the highest probability.

[0276] Before data selection and integration, the method for assigning weights to each vital data item is as follows. Weighting each vital data item begins with determining which factors will receive weighting. Below, we will examine specific examples of assigning weights based on various factors.

[0277]

[0278] 1) When the result of applying the facial recognition algorithm is used as a factor:

[0279] As an example, weighting can be given to images in which the angle between each of the multiple cameras and the driver / passenger's face is close to a right angle.

[0280] Deeplearning algorithms can detect the angle of the face when the facial recognition algorithm operates. The angle of the driver / passenger's face affects landmark size and skin color changes due to blood flow changes, which can be used to measure heart rate (HR). For this reason, weight is given to cameras whose faces are measured at close to a right angle to the driver / passenger's face.

[0281] For example, referring to FIG. 13, each of the multiple cameras installed in the vehicle (100) has a different shooting angle at each sensing point in time depending on the head movement of the driver / passenger, etc. Accordingly, as illustrated in FIG. 13, facial images captured at various shooting angles at each sensing point in time are acquired. In FIG. 13, a weight is given to an image (1310) in which the shooting angle of the camera and the face angle of the driver / passenger are close to a right angle.

[0282] In another embodiment, a high weight can be given to an image with the largest and clearest landmark among facial images acquired through multiple cameras.

[0283] When implementing facial recognition algorithms within deep learning, identifying facial landmarks and determining their area are crucial factors. This is because the size of these landmarks significantly impacts the performance of the algorithm after face recognition.

[0284] Facial landmarks refer to facial features (e.g., eyes, nose, mouth, eyebrows, etc.). When images are captured by multiple cameras, for example, a face detection algorithm can be used to detect the driver / passenger's face, and then a facial landmark algorithm can be used to detect landmarks within the face (e.g., eyes, nose, mouth, eyebrows, etc.).

[0285] Referring to Figure 14, it can be confirmed that landmarks such as eyes, eyebrows, and lips are detected in the facial image as a result of applying the facial landmark algorithm. Among the multiple images captured at the same time by multiple cameras, as shown in Figure 14, the image with the largest and clearest facial landmark of the driver / occupant is weighted to be used as the main image.

[0286] As another example, weight can be given to facial landmarks that appear frequently in images acquired through multiple cameras.

[0287] In another embodiment, weight may be given to images acquired through multiple cameras in which the area occupied by the driver / occupant is large.

[0288] Specifically, this is a method that places weight on the large areas of the driver / occupant's face and body before detecting landmarks using the deep learning algorithm. In cases where the driver / occupant's face is obscured by a hat, sunglasses, etc., the landmarks may be narrow, but this method places weight on areas with a large area of ​​the driver / occupant in the actual image. This differs from the facial landmark method in that it utilizes a large area of ​​the driver / occupant's outline.

[0289] In another embodiment, weights may be given to a camera on which a higher accuracy algorithm is running among multiple cameras.

[0290] Performance differences can exist between algorithms running on multiple cameras. Furthermore, even within the same rPPG algorithm, optimizations may vary depending on camera installation location, performance, and type, which can lead to performance differences. For example, if the algorithm for camera A has an accuracy of 95%, while the algorithm for camera B has an accuracy of 75%, a higher weighting may be assigned to camera A.

[0291] As another example, weighting can be given to algorithm results with higher accuracy values ​​within the same algorithm.

[0292] When the vital data measurement algorithm operates, it outputs data accuracy, indicating the accuracy level of the currently measured vital data. Therefore, by assigning weights based on the accuracy of the algorithm's output, it can be utilized for vital data fusion and selection.

[0293] In another embodiment, each of the plurality of vital data may be weighted based on data having high continuity and similarity with previous vital data values.

[0294] Human vital data exhibits continuity. That is, when a specific index in vital data increases or decreases, it tends to increase or decrease linearly. Based on this characteristic, data that changes rapidly compared to previous data points is more likely to be an outlier or a measurement error. Therefore, when continuously measured vital data loses linearity or exhibits excessive variability, data reliability can be assessed by assigning greater weight to data with greater similarity to previous data.

[0295] For example, if the heart rate is measured every second, let's say that for camera A, it is 60 bpm at 0 second, 75 pbm at 1 second, and 61 pbm at 2 seconds. Also, for camera B, let's say it is 60 pbm at 0 second, 62 pbm at 1 second, and 61 pbm at 2 seconds. Then, by giving weight to the data results of camera B, which has secured the continuity of the heart rate data at the 1-second point, data reliability can be secured.

[0296]

[0297] 2) When the performance of the camera itself is a factor:

[0298] As an example, weighting can be given to cameras with higher pixel counts in the captured facial area. This utilizes the fact that cameras with higher pixel counts can capture clearer and larger facial landmarks.

[0299] In another example, weighting can be given to cameras with higher resolution. This takes advantage of the fact that higher resolution cameras can capture clearer and larger facial landmarks.

[0300] As another example, we can weight cameras with higher frame rates.

[0301] The camera's frame rate significantly affects the accuracy of estimating the heart rate (HR) after measuring vital data or calculating the stress index, which is one of the vital post-processing methods. Therefore, a high frame rate is essential for more accurate vital data processing. The frame rates of the multiple cameras installed in a vehicle (100) vary significantly depending on the installation purpose and the performance of the ISP processing chip. Therefore, weighting may be applied based on the size of the frame rate.

[0302] As another example, weighting can be applied based on the overall / partial illumination of the camera image.

[0303] When measuring rPPG using an RGB camera, the algorithm's performance can be significantly affected by shadows or light on the driver / occupant's face. Furthermore, if the surroundings are too bright or dark, the algorithm may not function at all. This applies not only to RGB cameras but also to IR cameras. Specifically, because IR cameras are susceptible to ambient light, image illumination is a critical factor affecting algorithm performance. Therefore, algorithm performance can be improved by assigning specific weight to areas with high overall / partial illumination in the camera image.

[0304] As another example, weighting can be given to elements with less noise in the image. Image noise is a major factor in reducing algorithm accuracy. Therefore, by assigning lower weights to images with greater noise, algorithm performance can be improved.

[0305] Above, we examined how to weight factors when assessing the reliability of vital data acquired through multiple cameras. Below, we will describe in detail how to distribute weights to these factors.

[0306] Meanwhile, the method of measuring vital data non-contactly through multiple cameras in a vehicle (100) requires correction and error removal work on the data as measurement errors occur during measurement.

[0307] Accordingly, the present invention proposes a method for processing data by assigning weights based on reliability assessments to measured vital data. As previously described, the present invention utilizes 1) data fusion, 2) soft voting, and 3) a combination of the two to distribute weights to vital data.

[0308] Data Fusion is a data processing method that acquires vital data in real time from multiple cameras that capture the driver / occupant, synchronizes the vital data acquired through each camera in time, and integrates multiple vital data into a single data stream.

[0309] Soft Voting is a data processing method that generates final vital data by excluding missing or outlier values ​​from vital data acquired from each camera and fusing normal data by averaging or soft voting.

[0310] The final vital data generated through data fusion, soft voting, or a combination of these is passed to the upper layer and used as input data to estimate the driver / passenger condition index.

[0311] FIG. 15 is a flowchart illustrating a method of generating final input data by selecting and fusing vital data related to an embodiment of the present invention using a soft voting method.

[0312] First, each of the multiple cameras in the vehicle (100), Camera 1 (221), Camera 2 (222), and Camera 3 (223), senses vital data for the same driver / passenger at the same time.

[0313] Specifically, vital data sensed through camera 1 (221) is transmitted to the soft voting system (1510). In addition, vital data sensed through camera 2 (222) is transmitted to the soft voting system (1510). In addition, vital data sensed through camera 3 (223) is transmitted to the soft voting system (1510).

[0314] The soft voting system (1510) may be included in the form of an algorithm / processor / program within the vital data integration device (800). Alternatively, the soft voting system (1510) may be the data determination unit (820, FIG. 9) of the vital data integration device (800). Hereinafter, the soft voting system (1510) is disclosed as the data determination unit (820).

[0315] The data decision unit (820) selects vital data with a high reliability weight based on the weight assigned to each vital data, or fuses each vital data with a reliability weight reflected (“soft voting”) and transmits it to the data provision unit (830, FIG. 9).

[0316] The data decision unit (820) applies weights to the probability values ​​of each vital data and then calculates an average based on the assigned weights. Then, the data decision unit (820) calculates a reliability weighted probability for each vital data.

[0317] In another embodiment, the data determination unit (820) calculates an average of each vital data based on the weight assigned to each vital data, selects the calculated average vital data, and transmits it to the data provision unit (830) as a single integrated stream (“data fusion”).

[0318] In another embodiment, the data determination unit (820) may apply weights to the probability values ​​of each vital data and then select the vital data (label with the highest probability) having the highest value among the reliability weight probabilities. In addition, the data determination unit (820) may calculate the average of the reliability weight probabilities to generate fused vital data and transmit the selected vital data or the fused vital data to the data provision unit (830).

[0319] Next, the final generated reliable vital data is provided to the final level (1520) (1504) to more accurately estimate the driver / passenger condition index.

[0320] FIG. 16 and FIG. 17 are block diagrams and flowcharts illustrating obtaining, integrating, evaluating, and selecting a plurality of vital data related to an embodiment of the present invention and providing them to a driver state estimation algorithm.

[0321] Referring to FIG. 16, the plurality of cameras (220) in the vehicle may be a first camera (221) installed in the room mirror, a second camera (222) installed in the driver's seat A-Pillar, and a third camera (223) installed in the cluster.

[0322] A plurality of vital data acquired at the same time through a plurality of cameras (220) are transmitted to a vital data analysis module (1610). Here, the vital data analysis module (1610) may refer to a vital data integration device (800) according to the present invention.

[0323] The vital data analysis module (1610) may include a data synchronization (1601), a detector (1602), an evaluator (1603), a data selector (1604), a generator (1605), and a dispatcher (1606).

[0324] Data synchronization (1601) temporally synchronizes multiple vital data sensed through multiple cameras at the same time (but not perfectly at the same time).

[0325] The detector (1602) can remove missing or outlier data from temporally synchronized vital data and detect normal vital data.

[0326] The evaluator (1603) evaluates the reliability of each vital data according to a preset reliability criterion and assigns a weight.

[0327] For example, based on a preset confidence level, a weighting method can be adopted and applied as the aforementioned face and camera angles are closer to a right angle.

[0328] Specifically, when it is assumed that each of the multiple cameras (220) recognizes the same driver / passenger A, a higher ranking or higher weight may be assigned as the shooting angle approaches a right angle. For example, the first camera (221) closest to a right angle may be assigned a weight of '3', the second camera (222) may be assigned a weight of '2', and the third camera (223) may be assigned a weight of '1'.

[0329] The data selector (1604) and the generator (1605) select and fuse final data based on the weights assigned to the plurality of vital data and provide the data to the dispatcher (1606).

[0330] Specifically, the data selector (1604) and the generator (1605) can generate new final data by fusing the average values ​​of multiple vital data based on the weights assigned to them. For example, in the above example, the final data for the same driver / passenger A can be the output value of {(first camera (221) * 3) + (second camera (222) * 2) + (third camera (223) * 1)} / 3.

[0331] As another example, the data selector (1604) and the generator (1605) may select the label with the highest probability by averaging the probability values ​​based on the weights assigned to the plurality of vital data, or may produce the average of these as new final data.

[0332] Specifically, in the example above, let's say that the recognition probability of the first camera (221) is 0.8, the recognition probability of the second camera (222) is 0.7, the recognition probability of the third camera (223) is 0.6, and the respective weights are 0.6, 0.3, and 0.1 in the order in which the shooting angle is closest to a right angle.

[0333] In this case, the weight probability values ​​for each camera are as follows. Specifically, the weight probability value of the first camera (221) is calculated as (0.8*0.6) / (0.6+0.3+0.1) = 0.48. The weight probability value of the second camera (222) is calculated as (0.7*0.3) / (0.6+0.3+0.1) = 0.21. The weight probability value of the third camera (223) is calculated as (0.6*0.1) / (0.6+0.3+0.1) = 0.06.

[0334] At this time, the vital data sensed by the first camera (221) with the highest weight probability value may be finally selected, or the data fusion method may be further fused to determine the average of these as one fused final value.

[0335] Meanwhile, the dispatcher (1606) extracts the final generated vital data and provides it to the status estimation algorithm (1620).

[0336] The status estimation algorithm (1620) calculates / estimates various status indices using reliable vital data transmitted from the dispatcher (1606) as input data.

[0337] As illustrated in FIG. 16, the fatigue estimation algorithm (1621), stress assessment algorithm (1622), and emergency status assessment algorithm (1623) are each transmitted to estimate the driver / passenger's fatigue, calculate a stress index, and assess whether an emergency state exists. The evaluation results of the algorithms are transmitted to the vehicle (100), so that related additional actions / services can be performed. Furthermore, it goes without saying that more condition index algorithms may be included in addition to the algorithms illustrated in FIG. 16.

[0338] Figure 17 is a flowchart of operations related to the block diagram of Figure 16.

[0339] Referring to FIG. 17, vital data sensed from camera 1 (221) is transmitted to a data integrator (1710) (1701). In addition, vital data sensed from camera 2 (222) is transmitted to a data integrator (1710) (1702). In addition, vital data sensed from camera 3 (223) is transmitted to a data integrator (1710) (1701).

[0340] The data integrator (1710) synchronizes multiple vital data in time, removes data with missing or outlier values, and then transmits the data to the data evaluator (1720) (1704).

[0341] The data evaluator (1720) evaluates the reliability of each vital data according to a preset reliability criterion, calculates each weight, and transmits it to the data selector (1730) (1705).

[0342] The data selector (1730) selects and fuses reliable final data (1706) according to a data processing method based on the aforementioned data fusion method, soft voting method, or a combination of the two. The vital data finally generated by the data selector (1730) is transmitted to various algorithms of the driver status estimation algorithm (1740) (1707, 1708, 1709). The resulting various status index evaluations are transmitted to the associated operating system within the vehicle (100), enabling the provision of various services.

[0343] Meanwhile, in embodiments of the present invention, the vital data integration device (800) can reflect the sensing results and / or control results of other sensors or operating systems of the vehicle in the reliability evaluation of each vital data.

[0344] In the embodiment, the data determination unit (810, FIG. 9) of the vital data integration device (800)

[0345] The reliability weights for each vital data can be adjusted differently based on the vehicle's control information and the passenger's movement information.

[0346] Here, vehicle control information refers to control information for controlling the vehicle's braking and steering. This includes both cases where the vehicle is operated by the driver, and cases where sudden braking / sudden acceleration / sudden steering is triggered without driver intervention due to the ADAS system's operation.

[0347] In addition, the passenger movement information refers to the driver / passenger movement information obtained through a camera-based motion monitoring module, which is sensed by the movement monitoring module installed in the vehicle (100).

[0348] In an embodiment, the data determination unit (820) of the vital data integration device (800) may determine that the amount of change in each vital data has temporarily changed based on vehicle control information. Based on such determination, the reliability weight for each vital data may be reduced. Alternatively, in another embodiment, based on such determination, the weight calculation cycle for each vital data may be lengthened.

[0349] For example, when a sudden vehicle control event occurs, such as a sudden braking event due to a forward collision while driving, or when an internal or external event of the vehicle (100) significantly affects the heart rate of the driver / occupant, the weight calculation cycle for vital data may be temporarily extended, or the weight value based on the reliability assessment may be reduced by a certain value.

[0350] In another embodiment, the data determination unit (820) of the vital data integration device (800) determines that there is no change in the control information of the vehicle received for a predetermined period of time or longer, and based on such determination, increases the reliability weight for each vital data or shortens the reliability weight calculation cycle.

[0351] At this time, the predetermined time may be a time sufficient to determine that the driver is not operating due to drowsiness or an emergency situation. However, the predetermined time may vary depending on the driving status of the vehicle (100), the driving time zone, the driver's health status, etc.

[0352] For example, when driving with the same or no significant changes in steering and braking for a long period of time, the driver may experience changes in vital data, such as a decreased heart rate, due to drowsy driving in that situation. In such a situation, the weights of all vital data sensed from multiple cameras within the vehicle (100) can be increased or the weight calculation cycle can be shortened to utilize the service to prevent driver drowsiness.

[0353] In another embodiment, the data determination unit (820) of the vital data integration device (800) determines that the amount of change in each vital data has temporarily changed based on the movement information of the passenger, and may reduce the reliability weight for each vital data or lengthen the reliability weight calculation cycle based on such determination.

[0354] For example, if a driver engages in a lot of movement within a vehicle, such as eating, talking, or making large steering movements while driving, the driver's heart rate may temporarily increase due to the physical movement. These temporary changes in heart rate caused by these temporary movements may cause confusion in services related to assessing the driver's stress index or fatigue level and assessing their condition. Therefore, if a temporary large movement of the driver / occupant is detected, the weights of all vital data sensed from multiple cameras within the vehicle (100) may be lowered and / or the weight calculation cycle may be lengthened.

[0355] As described above, the vehicle vital data integration device according to an embodiment of the present invention can stably, seamlessly, and accurately collect and provide vital data of the driver / passenger when estimating driver / passenger drowsiness, stress, health information, etc. using multiple cameras. In other words, reliable input data can be provided for high-level vital data analysis.

[0356] The present invention described above can be implemented as computer-readable code on a medium having a program recorded thereon. Computer-readable media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid state disks (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc., and also include media implemented in the form of carrier waves (e.g., transmission via the Internet). In addition, the computer may include a controller / processor of a vital data integration device (800). Therefore, the above detailed description should not be construed as limiting in all respects, but should be considered as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are intended to be included in the scope of the present invention.

Claims

1. A receiving unit that receives each vital data collected by multiple cameras installed inside the vehicle; A data determination unit that evaluates the reliability of the received vital data, assigns weights to it, and selects or fuses the vital data based on the assigned weights; and Including a data providing unit that provides the final generated vital data to estimate the driver's condition index. A vital data integration device for vehicles.

2. In paragraph 1, The reliability assessment of the vital data received above is as follows: A method characterized in that different weights for each vital data are determined based on at least one of the reliability of each of the plurality of cameras, landmark information of images sensed through the plurality of cameras, area occupied by a face in the images, and change amount compared to previous vital data. A vital data integration device for vehicles.

3. In paragraph 1, Each of the above multiple cameras is installed around the driver's seat in the vehicle and senses the driver's vital data using the rPPG (Remote PPG) method. The above-mentioned receiving unit is characterized in that it transmits the plurality of sensed vital data to the data determination unit in a time-synchronized manner. A vital data integration device for vehicles.

4. In paragraph 3, The above data determination unit, After removing data with outliers from the above plurality of vital data, the reliability of normal vital data is evaluated and weights are given. A vital data integration device for vehicles.

5. In paragraph 1, The above data determination unit, Based on the weights given above, the data is selected based on vital data having a high reliability weight or each vital data reflecting the reliability weight is fused and transmitted to the data provider. A vital data integration device for vehicles.

6. In paragraph 5, The above data determination unit, A method characterized in that the average of each vital data is calculated based on the weights assigned above, and the calculated average vital data is selected and transmitted to the data provider as a single stream. A vital data integration device for vehicles.

7. In paragraph 5, The above data determination unit, After applying the weight assigned above to the probability value of each vital data, an average is calculated based on the weight assigned above, thereby calculating the reliability weight probability for each vital data. A vital data integration device for vehicles.

8. In paragraph 7, The above data determination unit, A method characterized in that vital data having the highest value among the reliability weight probabilities is selected or an average of the reliability weight probabilities is calculated to generate fused vital data, and the selected vital data or fused vital data is transmitted to the data provider. A vital data integration device for vehicles.

9. In paragraph 1, The above data determination unit, It is characterized by adjusting the reliability weight for each vital data differently based on the vehicle control information and the passenger's movement information. A vital data integration device for vehicles.

10. In paragraph 9, The above data determination unit, A method characterized in that it determines that the amount of change in each vital data has temporarily changed based on the vehicle control information and reduces the reliability weight for each vital data according to the determination. A vital data integration device for vehicles.

11. In paragraph 10, The control information of the above vehicle is: Characterized in that the control result corresponds to any one of the brake, start, or steering operation that exceeds the set range performed by the driver or the ADAS system. A vital data integration device for vehicles.

12. In paragraph 9, The above data determination unit, It is characterized by determining that there is no change in the control information of a vehicle received for a predetermined period of time or longer, and increasing the reliability weight for each vital data or shortening the reliability weight calculation cycle according to the determination. A vital data integration device for vehicles.

13. In paragraph 9, The above data determination unit, A method characterized in that it determines that the amount of change in each vital data has temporarily changed based on the movement information of the passenger, and reduces the reliability weight for each vital data or lengthens the reliability weight calculation cycle according to the determination. A vital data integration device for vehicles.

14. A step of receiving each vital data sensed by multiple cameras installed inside the vehicle; A step of evaluating the reliability of the received vital data and assigning weights to it; A step of selecting or fusing vital data based on the assigned weights; and It includes a step of estimating the driver's condition index by providing the final generated vital data as input data. Method for integrating vital data of a vehicle.

15. In paragraph 14, The step of assigning the above weights is: A step of evaluating the reliability of each vital data and determining different weights based on at least one of the reliability of each of the plurality of cameras, landmark information of images sensed through the plurality of cameras, the area occupied by the face in the images, and the amount of change compared to the previous vital data, characterized in that the step of determining different weights is performed. Method for integrating vital data of a vehicle.

16. In paragraph 14, The step of assigning the above weights is: Characterized in that it includes a step of additionally adjusting the reliability weight for each vital data based on the vehicle control information and the passenger's movement information. Method for integrating vital data of a vehicle.

Images